Skip to content

[WIP] docs: consolidate setup guides and fix outdated instructions #1944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mentonin wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[WIP] docs: consolidate setup guides and fix outdated instructions #1944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,18 @@ KernelCI Dashboard is an open-source project and contributions of all kinds are

## Before you start

- Read the README for an overview of the monorepo.
- We recommend following the [Onboarding guide](./docs/Onboarding.md) to set up your environment and learn the project workflow. **Start here** if this is your first setup.
- New to the project? Pick an issue labeled ["good first issue"](https://github.com/kernelci/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
- There are a couple of extensions that may help you with linting and formatting your code. Consider installing [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/) for your preferred code editor.
- For Python development, this repository uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting, and [pre-commit](https://pre-commit.com/) for Git hooks.
1. Read the [README](./README.md) for a project overview.
2. Set up locally:
- **Docker with live reload:** [docs/dev-environment.md](./docs/dev-environment.md) — or run `make setup` then `make dev`
- **Guided walkthrough:** [Onboarding guide](./docs/Onboarding.md)
- **Manual backend + frontend on the host:** [backend/README.md](./backend/README.md) and [dashboard/README.md](./dashboard/README.md)
3. New to the project? Pick an issue labeled ["good first issue"](https://github.com/kernelci/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
4. For Python development, this repository uses [Ruff](https://docs.astral.sh/ruff/) and [pre-commit](https://pre-commit.com/). For the frontend, consider [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/) in your editor.

## Useful links

- README: [./README.md](./README.md)
- Dev environment: [./docs/dev-environment.md](./docs/dev-environment.md)
- Onboarding guide: [./docs/Onboarding.md](./docs/Onboarding.md)
- Open issues on [this page](https://github.com/kernelci/dashboard/issues)
- Good first issues on [this page](https://github.com/kernelci/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
Expand Down Expand Up @@ -41,7 +44,7 @@ KernelCI Dashboard is an open-source project and contributions of all kinds are
1. Fork the repo and create a feature branch from main
2. Keep PRs focused and small; one change per PR
3. Follow Conventional Commits for commit messages (see Conventional Commits below)
4. Run services locally (backend, frontend, or Docker) as described in the Onboarding guide; ensure builds and checks pass
4. Run services locally as described above; ensure builds and checks pass
5. Update documentation and add tests when applicable
6. Ensure the PR passes automated checks; see the suggested workflow in the "Make your code pass automated code checks" section

Expand Down
69 changes: 14 additions & 55 deletions DEPLOYMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
# KernelCI Dashboard — Deployment Guide

This guide covers three deployment scenarios: [development](#1-development), [production](#2-production), and [staging](#3-staging).
This guide covers three deployment scenarios: [containerized staging-like local run](#1-containerized-staging-like-local-run), [production](#2-production), and [staging](#3-staging).

For contributor live reload, see [docs/dev-environment.md](docs/dev-environment.md) instead.

## Quick Reference

| Scenario | Compose File | Database | Profiles |
|----------|-------------|----------|----------|
| Development | `docker-compose.yml` | Local (always on) | `with_commands` (for ingester) |
| Contributor live reload | `docker-compose.dev.yml` | Local | — |
| Staging-like local run | `docker-compose.yml` | Local (`dashboard_db`) | `with_commands` (for ingester) |
| Production | `docker-compose-next.yml` | External PostgreSQL | none (or `with_commands`) |
| Staging | `docker-compose.yml` | External PostgreSQL (shared with production) | none (or `with_commands`) |

Expand All @@ -15,7 +18,7 @@ This guide covers three deployment scenarios: [development](#1-development), [pr
## Table of Contents

- [Prerequisites](#prerequisites)
- [1. Development](#1-development)
- [1. Containerized staging-like local run](#1-containerized-staging-like-local-run)
- [2. Production](#2-production)
- [3. Staging](#3-staging)
- [Profile Reference](#profile-reference)
Expand Down Expand Up @@ -50,24 +53,13 @@ This guide covers three deployment scenarios: [development](#1-development), [pr

### Accessing production database

If direct access to the production database is required,
whether for local debugging or validating critical feature development,
you must request permissions for the SSH connection and database user.
See [Onboarding Task 0](docs/Onboarding.md#task-0-check-your-ssh-and-database-access) for SSH tunnel and credential setup.

1. Connect to the Azure database SSH bridge:
- Create a new SSH key and add it to your SSH agent.
- Share the public SSH key to the database maintainer, to be granted access
to the SSH tunnel.
- Connect to the database via SSH tunnel with the provided URL.
2. Request credentials: Obtain a new username and password for the database access.
3. Connect: Once you have your credentials, connect to the database via `psql`, `pgAdmin`,
`DBeaver`, or any other PostgreSQL manager.
## 1. Containerized staging-like local run

## 1. Development
> For contributor live reload (Django + Vite HMR), use [docs/dev-environment.md](docs/dev-environment.md) (`docker-compose.dev.yml`) instead.

This is the minimal guide for development using
the fully containerized (Docker Compose) alternative.
Images are built locally and use a global `.env` file.
This section covers a staging-like stack built from source with Gunicorn and a production-style frontend image. Images are built locally and use root `.env`. The proxy defaults to port **9000**.

### Setup

Expand Down Expand Up @@ -95,40 +87,9 @@ docker compose up --build -d backend
docker compose up --build -d
```

### Local Frontend development

When implementing frontend features, a fast "hot reload" workflow
is usually preferred. In these scenarios, it is recommended to
run a local frontend server on the host machine.

For active frontend work, we can run the Vite dev server directly:

```bash
cd dashboard
pnpm install
# Copy the example env file and verify VITE_API_BASE_URL
cp .env.example .env
pnpm dev
```

The frontend connects to the backend API via the `VITE_API_BASE_URL`
defined in `dashboard/.env` (defaults to `http://localhost:8000`).

### Local Backend development
To implement backend features with hot reloading,
you can also start a local Django instance.

```bash
cd backend
poetry install

# Copy the env file and edit the DB_* to match your local database instance.
# Also set DEBUG=True to allow CORS connections and stack traces.
poetry run python3 manage.py runserver
```
### Host-based development

The backend connects to a PostgreSQL instance using the environment variables:
`DB_NAME`, `DB_PASSWORD`, `DB_HOST`, `DB_PORT`, `DB_ENGINE`, `DB_OPTIONS_CONNECT_TIMEOUT`.
For hot reload on the host machine, see [dashboard/README.md](dashboard/README.md) (frontend) and [backend/README.md](backend/README.md) (backend). Point `VITE_API_BASE_URL` in `dashboard/.env` at your backend (default `http://localhost:8000`).

---

Expand All @@ -138,9 +99,7 @@ Unlike the development deployment, our production environment connects to a
pre-existing external PostgreSQL instance and use pre-built docker images
stored in the GitHub Container Registry (GHCR).

Production images are automatically built via GitHub Workflow,
to every new commit in the main branch, or when
the `Publish GHCR Images` workflow is triggered manually.
Production images are built on every push to main and when [deploy-containers.yaml](.github/workflows/deploy-containers.yaml) is triggered manually (`workflow_dispatch`).

The GitHub workflow for production is defined at: [deploy-production](.github/workflows/deploy-production.yaml)

Expand Down Expand Up @@ -266,7 +225,7 @@ Previous versions used `DB_DEFAULT_*` prefixed variables (e.g., `DB_DEFAULT_PASS

## Related Documentation

- [README](./README.md) - Project overview and build instructions
- [README](./README.md) - Project overview and local development
- [CONTRIBUTING](./CONTRIBUTING.md) - How to contribute to the project
- [Monitoring Setup](./docs/monitoring.md) — Prometheus metrics configuration
- [Notifications](./docs/notifications.md) — Email and Discord notification setup
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ BACKEND_DIR ?= backend
help: ## Show this help
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_.-]+:.*##/ {printf " \033[36m%-10s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

setup: ## Copy env files for Docker dev and manual backend workflows, then install dependencies
setup: ## Copy env files (.env for Docker; .env.backend for manual backend reference), then install dependencies
@test -f .env || (cp .env.example .env && echo "Created .env from .env.example")
@test -f .env.backend || (cp .env.backend.example .env.backend && echo "Created .env.backend from .env.backend.example")
@test -f $(DASHBOARD_DIR)/.env || (cp $(DASHBOARD_DIR)/.env.example $(DASHBOARD_DIR)/.env && echo "Created dashboard/.env from dashboard/.env.example")
Expand Down
108 changes: 14 additions & 94 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,104 +20,26 @@ A Python http server built with [Django](https://www.djangoproject.com/) + [DRF]

## Quick run

If you want to just run the project, you can try out pre-built images with the [docker-compose-next.yml](./docker-compose-next.yml) file. This pulls images from GHCR and runs them locally without needing to rebuild them. You may still need to set up environment variables, so read the docs.
To run pre-built images without rebuilding, use [docker-compose-next.yml](./docker-compose-next.yml). Copy [`.env.example`](.env.example) to `.env`, set required values, then start the stack. The proxy defaults to port **80**. See [DEPLOYMENT.md](./DEPLOYMENT.md) for details.

## Build
## Local development

### Frontend
Pick the workflow that fits how you want to work:

Create a .env file in /dashboard, check and set the variables and their values
```sh
cp ./dashboard/.env.example ./dashboard/.env
```
| Workflow | Guide |
|---|---|
| Docker with live reload (recommended) | [docs/dev-environment.md](docs/dev-environment.md) — `make dev` or `docker compose -f docker-compose.dev.yml up -d` |
| Manual backend + frontend on the host | [backend/README.md](backend/README.md) and [dashboard/README.md](dashboard/README.md) |
| Staging-like container run (no live reload) | [DEPLOYMENT.md](./DEPLOYMENT.md) §1 |
| Guided first-time setup | [docs/Onboarding.md](docs/Onboarding.md) |

With docker, you can start just the frontend with `docker compose up --build proxy`. It is also possible to run the dashboard outside of it for development purposes.

@alanpeixinho alanpeixinho Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldnt we keep a guide on how to run the project without docker? even if we recommend the docker path?


We use `pnpm` to help with the package management. Install the dependencies with
```sh
pnpm install
```

Then start the dev server with
```sh
pnpm dev
```

If you want to test the production state of the dashboard, use
```sh
pnpm build
pnpm preview
```

### Backend

Create a .env file in the base directory,
```sh
cp .env.backend.example .env.backend
```

Create a secret key for Django:
```sh
export DJANGO_SECRET_KEY=$(openssl rand -base64 22)
```
We are not using sessions or anything like that right now, so changing the secret key won't be a big deal.

Since the production *database* is not open for the public, we use ssh tunneling with a whitelist to access it. This means that the docker setup currently can't access it, but we have a local database that is connected automatically if you don't change the env vars.

If you do use docker, you should create a secret file with the database password:
```sh
mkdir -p backend/runtime/secrets
echo <password> > backend/runtime/secrets/postgres_password_secret
```

If you are going to use a database user other than `kernelci`, set it to `DB_USER`:
```sh
export DB_USER=<user>
```

If you are setting up instance different than production KernelCI dashboard, you need to define CORS_ALLOWED_ORIGINS. On .env.backend:
```
CORS_ALLOWED_ORIGINS=["https://d.kernelci.org","https://dashboard.kernelci.org"]
```

It is also possible to run the backend outside of docker for development purposes. Simply run the ssh tunnel with the instructions sent to you by the database manager, then export the variables seen in [.env.backend.example](/.env.backend.example).

> For other optional envs, check the [backend README](backend/README.md).

### Common

Startup the services:
```sh
docker compose up --build -d
```
Docker exposes port 80 (that you don't need to enter in the URL) instead of 5173 and 8000 that is used when running the dashboard project outside of docker.
So you can hit the frontend with `http://localhost` and the backend with `http://localhost/api` when running locally.

Make sure that docker has the right permissions and has access to the environment variables. One way to do that is to set up a docker permission group.

If you are running the commands for exporting the environment variables and running docker separately, you can run docker with admin privileges and allowing environment variables with:
```sh
sudo -E docker compose up --build -d
```
Or you can also run the env exports and docker compose within the root user by running `sudo su`.

> Tip: you can create a quick script to set all the necessary envs and start the services. This will also allow docker to see the environment variables correctly. Example:

```sh
export DB_USER=email@email.com
export DJANGO_SECRET_KEY=$(openssl rand -base64 22)
export DB_NAME=kcidb
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."

docker compose up --build
```

> [Note] If you are going to run using only the local database, the DB_NAME should be `dashboard` and the `DB_USER` and `DB_PASSWORD` should be `admin` (for now). This simply follows what is going to be setup by the `dashboard_db` service on docker compose.
**Env files:** Docker Compose reads root `.env` plus `dashboard/.env`. For manual backend runs, export variables from [`.env.backend.example`](.env.backend.example) (Django does not load that file automatically).

**Ports:** `docker-compose.dev.yml` and `docker-compose.yml` expose the app at `http://localhost:9000` by default. Running services on the host uses `5173` (frontend) and `8000` (backend). `docker-compose-next.yml` defaults to port **80**.

## Deploying to production

See [DEPLOYMENT.md](./DEPLOYMENT.md) for full deployment instructions covering development, staging, and production scenarios.
See [DEPLOYMENT.md](./DEPLOYMENT.md) for staging, production, and deployment scenarios.

To deploy to prod you need to push a tag in the `release/YYYYMMDD.N` format
like: `release/20240910.0`
Expand Down Expand Up @@ -149,8 +71,6 @@ If you want to verify container/deployment environment settings before running s
- [docs/verify_env.md](docs/verify_env.md) for detailed examples, including test email sending to a specific destination
- Destination is required with `--send-test-email` and `--to-email`.

## Contributing

Check out our [CONTRIBUTING.md](/CONTRIBUTING.md), and there is an [onboarding guide](docs/Onboarding.md) to help get acquainted with the project. Contributions are welcome!
## Contributing

For a local development environment with live reload (backend + frontend), see [docs/dev-environment.md](docs/dev-environment.md). That Docker-based workflow uses a root `.env` file plus `dashboard/.env`; the backend-specific manual setup above still uses `.env.backend`. Use the env files required by the workflow you choose.
Check out [CONTRIBUTING.md](./CONTRIBUTING.md) and the [onboarding guide](docs/Onboarding.md). Contributions are welcome!
Loading