diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e00189a01..3405181b1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,15 +4,18 @@ KernelCI Dashboard is an open-source project and contributions of all kinds are ## Before you start -- Read the README for an overview of the monorepo. -- We recommend following the [Onboarding guide](./docs/Onboarding.md) to set up your environment and learn the project workflow. **Start here** if this is your first setup. -- New to the project? Pick an issue labeled ["good first issue"](https://github.com/kernelci/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). -- There are a couple of extensions that may help you with linting and formatting your code. Consider installing [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/) for your preferred code editor. -- For Python development, this repository uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting, and [pre-commit](https://pre-commit.com/) for Git hooks. +1. Read the [README](./README.md) for a project overview. +2. Set up locally: + - **Docker with live reload:** [docs/dev-environment.md](./docs/dev-environment.md) — or run `make setup` then `make dev` + - **Guided walkthrough:** [Onboarding guide](./docs/Onboarding.md) + - **Manual backend + frontend on the host:** [backend/README.md](./backend/README.md) and [dashboard/README.md](./dashboard/README.md) +3. New to the project? Pick an issue labeled ["good first issue"](https://github.com/kernelci/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). +4. For Python development, this repository uses [Ruff](https://docs.astral.sh/ruff/) and [pre-commit](https://pre-commit.com/). For the frontend, consider [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/) in your editor. ## Useful links - README: [./README.md](./README.md) +- Dev environment: [./docs/dev-environment.md](./docs/dev-environment.md) - Onboarding guide: [./docs/Onboarding.md](./docs/Onboarding.md) - Open issues on [this page](https://github.com/kernelci/dashboard/issues) - Good first issues on [this page](https://github.com/kernelci/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) @@ -41,7 +44,7 @@ KernelCI Dashboard is an open-source project and contributions of all kinds are 1. Fork the repo and create a feature branch from main 2. Keep PRs focused and small; one change per PR 3. Follow Conventional Commits for commit messages (see Conventional Commits below) -4. Run services locally (backend, frontend, or Docker) as described in the Onboarding guide; ensure builds and checks pass +4. Run services locally as described above; ensure builds and checks pass 5. Update documentation and add tests when applicable 6. Ensure the PR passes automated checks; see the suggested workflow in the "Make your code pass automated code checks" section diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index ce05be056..d258d77a3 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -1,12 +1,15 @@ # KernelCI Dashboard — Deployment Guide -This guide covers three deployment scenarios: [development](#1-development), [production](#2-production), and [staging](#3-staging). +This guide covers three deployment scenarios: [containerized staging-like local run](#1-containerized-staging-like-local-run), [production](#2-production), and [staging](#3-staging). + +For contributor live reload, see [docs/dev-environment.md](docs/dev-environment.md) instead. ## Quick Reference | Scenario | Compose File | Database | Profiles | |----------|-------------|----------|----------| -| Development | `docker-compose.yml` | Local (always on) | `with_commands` (for ingester) | +| Contributor live reload | `docker-compose.dev.yml` | Local | — | +| Staging-like local run | `docker-compose.yml` | Local (`dashboard_db`) | `with_commands` (for ingester) | | Production | `docker-compose-next.yml` | External PostgreSQL | none (or `with_commands`) | | Staging | `docker-compose.yml` | External PostgreSQL (shared with production) | none (or `with_commands`) | @@ -15,7 +18,7 @@ This guide covers three deployment scenarios: [development](#1-development), [pr ## Table of Contents - [Prerequisites](#prerequisites) -- [1. Development](#1-development) +- [1. Containerized staging-like local run](#1-containerized-staging-like-local-run) - [2. Production](#2-production) - [3. Staging](#3-staging) - [Profile Reference](#profile-reference) @@ -50,24 +53,13 @@ This guide covers three deployment scenarios: [development](#1-development), [pr ### Accessing production database -If direct access to the production database is required, -whether for local debugging or validating critical feature development, -you must request permissions for the SSH connection and database user. +See [Onboarding Task 0](docs/Onboarding.md#task-0-check-your-ssh-and-database-access) for SSH tunnel and credential setup. -1. Connect to the Azure database SSH bridge: - - Create a new SSH key and add it to your SSH agent. - - Share the public SSH key to the database maintainer, to be granted access - to the SSH tunnel. - - Connect to the database via SSH tunnel with the provided URL. -2. Request credentials: Obtain a new username and password for the database access. -3. Connect: Once you have your credentials, connect to the database via `psql`, `pgAdmin`, -`DBeaver`, or any other PostgreSQL manager. +## 1. Containerized staging-like local run -## 1. Development +> For contributor live reload (Django + Vite HMR), use [docs/dev-environment.md](docs/dev-environment.md) (`docker-compose.dev.yml`) instead. -This is the minimal guide for development using -the fully containerized (Docker Compose) alternative. -Images are built locally and use a global `.env` file. +This section covers a staging-like stack built from source with Gunicorn and a production-style frontend image. Images are built locally and use root `.env`. The proxy defaults to port **9000**. ### Setup @@ -95,40 +87,9 @@ docker compose up --build -d backend docker compose up --build -d ``` -### Local Frontend development - -When implementing frontend features, a fast "hot reload" workflow -is usually preferred. In these scenarios, it is recommended to -run a local frontend server on the host machine. - -For active frontend work, we can run the Vite dev server directly: - -```bash -cd dashboard -pnpm install -# Copy the example env file and verify VITE_API_BASE_URL -cp .env.example .env -pnpm dev -``` - -The frontend connects to the backend API via the `VITE_API_BASE_URL` -defined in `dashboard/.env` (defaults to `http://localhost:8000`). - -### Local Backend development -To implement backend features with hot reloading, -you can also start a local Django instance. - -```bash -cd backend -poetry install - -# Copy the env file and edit the DB_* to match your local database instance. -# Also set DEBUG=True to allow CORS connections and stack traces. -poetry run python3 manage.py runserver -``` +### Host-based development -The backend connects to a PostgreSQL instance using the environment variables: -`DB_NAME`, `DB_PASSWORD`, `DB_HOST`, `DB_PORT`, `DB_ENGINE`, `DB_OPTIONS_CONNECT_TIMEOUT`. +For hot reload on the host machine, see [dashboard/README.md](dashboard/README.md) (frontend) and [backend/README.md](backend/README.md) (backend). Point `VITE_API_BASE_URL` in `dashboard/.env` at your backend (default `http://localhost:8000`). --- @@ -138,9 +99,7 @@ Unlike the development deployment, our production environment connects to a pre-existing external PostgreSQL instance and use pre-built docker images stored in the GitHub Container Registry (GHCR). -Production images are automatically built via GitHub Workflow, -to every new commit in the main branch, or when -the `Publish GHCR Images` workflow is triggered manually. +Production images are built on every push to main and when [deploy-containers.yaml](.github/workflows/deploy-containers.yaml) is triggered manually (`workflow_dispatch`). The GitHub workflow for production is defined at: [deploy-production](.github/workflows/deploy-production.yaml) @@ -266,7 +225,7 @@ Previous versions used `DB_DEFAULT_*` prefixed variables (e.g., `DB_DEFAULT_PASS ## Related Documentation -- [README](./README.md) - Project overview and build instructions +- [README](./README.md) - Project overview and local development - [CONTRIBUTING](./CONTRIBUTING.md) - How to contribute to the project - [Monitoring Setup](./docs/monitoring.md) — Prometheus metrics configuration - [Notifications](./docs/notifications.md) — Email and Discord notification setup diff --git a/Makefile b/Makefile index d5c4b5e36..e0627a5d7 100644 --- a/Makefile +++ b/Makefile @@ -7,7 +7,7 @@ BACKEND_DIR ?= backend help: ## Show this help @awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_.-]+:.*##/ {printf " \033[36m%-10s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) -setup: ## Copy env files for Docker dev and manual backend workflows, then install dependencies +setup: ## Copy env files (.env for Docker; .env.backend for manual backend reference), then install dependencies @test -f .env || (cp .env.example .env && echo "Created .env from .env.example") @test -f .env.backend || (cp .env.backend.example .env.backend && echo "Created .env.backend from .env.backend.example") @test -f $(DASHBOARD_DIR)/.env || (cp $(DASHBOARD_DIR)/.env.example $(DASHBOARD_DIR)/.env && echo "Created dashboard/.env from dashboard/.env.example") diff --git a/README.md b/README.md index 423f40574..71fe99940 100644 --- a/README.md +++ b/README.md @@ -20,104 +20,26 @@ A Python http server built with [Django](https://www.djangoproject.com/) + [DRF] ## Quick run -If you want to just run the project, you can try out pre-built images with the [docker-compose-next.yml](./docker-compose-next.yml) file. This pulls images from GHCR and runs them locally without needing to rebuild them. You may still need to set up environment variables, so read the docs. +To run pre-built images without rebuilding, use [docker-compose-next.yml](./docker-compose-next.yml). Copy [`.env.example`](.env.example) to `.env`, set required values, then start the stack. The proxy defaults to port **80**. See [DEPLOYMENT.md](./DEPLOYMENT.md) for details. -## Build +## Local development -### Frontend +Pick the workflow that fits how you want to work: -Create a .env file in /dashboard, check and set the variables and their values -```sh - cp ./dashboard/.env.example ./dashboard/.env -``` +| Workflow | Guide | +|---|---| +| Docker with live reload (recommended) | [docs/dev-environment.md](docs/dev-environment.md) — `make dev` or `docker compose -f docker-compose.dev.yml up -d` | +| Manual backend + frontend on the host | [backend/README.md](backend/README.md) and [dashboard/README.md](dashboard/README.md) | +| Staging-like container run (no live reload) | [DEPLOYMENT.md](./DEPLOYMENT.md) §1 | +| Guided first-time setup | [docs/Onboarding.md](docs/Onboarding.md) | -With docker, you can start just the frontend with `docker compose up --build proxy`. It is also possible to run the dashboard outside of it for development purposes. - -We use `pnpm` to help with the package management. Install the dependencies with -```sh -pnpm install -``` - -Then start the dev server with -```sh -pnpm dev -``` - -If you want to test the production state of the dashboard, use -```sh -pnpm build -pnpm preview -``` - -### Backend - -Create a .env file in the base directory, -```sh - cp .env.backend.example .env.backend -``` - -Create a secret key for Django: -```sh -export DJANGO_SECRET_KEY=$(openssl rand -base64 22) -``` -We are not using sessions or anything like that right now, so changing the secret key won't be a big deal. - -Since the production *database* is not open for the public, we use ssh tunneling with a whitelist to access it. This means that the docker setup currently can't access it, but we have a local database that is connected automatically if you don't change the env vars. - -If you do use docker, you should create a secret file with the database password: -```sh -mkdir -p backend/runtime/secrets -echo > backend/runtime/secrets/postgres_password_secret -``` - -If you are going to use a database user other than `kernelci`, set it to `DB_USER`: -```sh -export DB_USER= -``` - -If you are setting up instance different than production KernelCI dashboard, you need to define CORS_ALLOWED_ORIGINS. On .env.backend: -``` -CORS_ALLOWED_ORIGINS=["https://d.kernelci.org","https://dashboard.kernelci.org"] -``` - -It is also possible to run the backend outside of docker for development purposes. Simply run the ssh tunnel with the instructions sent to you by the database manager, then export the variables seen in [.env.backend.example](/.env.backend.example). - -> For other optional envs, check the [backend README](backend/README.md). - -### Common - -Startup the services: - ```sh - docker compose up --build -d - ``` - Docker exposes port 80 (that you don't need to enter in the URL) instead of 5173 and 8000 that is used when running the dashboard project outside of docker. - So you can hit the frontend with `http://localhost` and the backend with `http://localhost/api` when running locally. - -Make sure that docker has the right permissions and has access to the environment variables. One way to do that is to set up a docker permission group. - -If you are running the commands for exporting the environment variables and running docker separately, you can run docker with admin privileges and allowing environment variables with: -```sh -sudo -E docker compose up --build -d -``` -Or you can also run the env exports and docker compose within the root user by running `sudo su`. - -> Tip: you can create a quick script to set all the necessary envs and start the services. This will also allow docker to see the environment variables correctly. Example: - -```sh -export DB_USER=email@email.com -export DJANGO_SECRET_KEY=$(openssl rand -base64 22) -export DB_NAME=kcidb -export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..." - -docker compose up --build -``` - -> [Note] If you are going to run using only the local database, the DB_NAME should be `dashboard` and the `DB_USER` and `DB_PASSWORD` should be `admin` (for now). This simply follows what is going to be setup by the `dashboard_db` service on docker compose. +**Env files:** Docker Compose reads root `.env` plus `dashboard/.env`. For manual backend runs, export variables from [`.env.backend.example`](.env.backend.example) (Django does not load that file automatically). +**Ports:** `docker-compose.dev.yml` and `docker-compose.yml` expose the app at `http://localhost:9000` by default. Running services on the host uses `5173` (frontend) and `8000` (backend). `docker-compose-next.yml` defaults to port **80**. ## Deploying to production -See [DEPLOYMENT.md](./DEPLOYMENT.md) for full deployment instructions covering development, staging, and production scenarios. +See [DEPLOYMENT.md](./DEPLOYMENT.md) for staging, production, and deployment scenarios. To deploy to prod you need to push a tag in the `release/YYYYMMDD.N` format like: `release/20240910.0` @@ -149,8 +71,6 @@ If you want to verify container/deployment environment settings before running s - [docs/verify_env.md](docs/verify_env.md) for detailed examples, including test email sending to a specific destination - Destination is required with `--send-test-email` and `--to-email`. -## Contributing - -Check out our [CONTRIBUTING.md](/CONTRIBUTING.md), and there is an [onboarding guide](docs/Onboarding.md) to help get acquainted with the project. Contributions are welcome! +## Contributing -For a local development environment with live reload (backend + frontend), see [docs/dev-environment.md](docs/dev-environment.md). That Docker-based workflow uses a root `.env` file plus `dashboard/.env`; the backend-specific manual setup above still uses `.env.backend`. Use the env files required by the workflow you choose. +Check out [CONTRIBUTING.md](./CONTRIBUTING.md) and the [onboarding guide](docs/Onboarding.md). Contributions are welcome! diff --git a/backend/README.md b/backend/README.md index e6652ba28..5b8205ce5 100644 --- a/backend/README.md +++ b/backend/README.md @@ -41,7 +41,7 @@ It's possible to export `DEBUG_SQL_QUERY=True` if you want to see which SQL quer ### Databases -For the main database, the backend uses a series of `DB_` environment variables that have to be set such as: +Export variables from [`.env.backend.example`](../.env.backend.example) (Django does not load that file automatically). Example for the **production database via SSH tunnel**: ```sh export DB_NAME=kcidb @@ -53,6 +53,8 @@ export DB_ENGINE=django_prometheus.db.backends.postgresql export DB_OPTIONS_CONNECT_TIMEOUT=16 ``` +For the **local Docker database**, use `DB_NAME=dashboard`, `DB_USER=admin`, and the password from root `.env`. + #### Connection Pooling (Production) For production deployments, you can enable persistent database connections to reduce connection overhead: @@ -92,35 +94,17 @@ poetry run python3 manage.py runserver We have a couple of useful scripts: -* [migrate-cache-db.sh](./migrate-cache-db.sh) will create and apply migrations for the cache SQLite database. This runs automatically when running on docker, but you have to run it mannually otherwise. +* [migrate-cache-db.sh](./migrate-cache-db.sh) will create and apply migrations for the cache SQLite database. This runs automatically when running on docker, but you have to run it manually otherwise. * [migrate-app-db.sh](./migrate-app-db.sh) will create and apply migrations for the main app, generally used for the local database. This does not run automatically in order to avoid potential problems. Be aware that *if* you have write permissions to the production database it will be changed, so double check that you are connected to the right one (but usually developers won't have write access to the production database anyway). It is important to note that Django automatically creates migrations based on changes to the models when running the first command of the scripts above. You can edit the migrations manually, and you can also run the commands by hand if you want more control over it. -* [copy_db_data.sh](./scripts/copy_db_data.sh) will copy 7 days of data from you `DB_DEFAULT` to your `DASH_DB`. You can also modify the script for a custom interval, and you can check the [update_db](./kernelCI_app/management/commands/update_db.py) command for other arguments such as `--table` and `--origins`. +* [update_db command](./docs/update_db%20command.md) copies data from the production database (`kcidb`) to the local app database (`dashboard`) over a time interval. * [generate-schema.sh](./generate-schema.sh) will automatically generate the OpenAPI schema for the endpoints. Please use it whenever the ins and outs of endpoints change. ## Linting, formatting, and hooks -The backend uses Ruff for linting and formatting. - -- Check lint issues: `poetry run ruff check .` -- Auto-fix lint issues: `poetry run ruff check . --fix` -- Check formatting: `poetry run ruff format --check .` -- Auto-fix formatting: `poetry run ruff format .` - -This repository uses pre-commit for Git hooks (`pre-commit` and `pre-push`). Install hooks once per clone from the repository root: - -```sh -poetry -C backend run pre-commit install --hook-type pre-commit --hook-type pre-push --hook-type commit-msg --install-hooks -``` - -To run all configured hooks manually: - -```sh -poetry -C backend run pre-commit run --all-files -``` - +See [CONTRIBUTING.md](../CONTRIBUTING.md) for lint, format, test, and pre-commit commands. ## Running tests @@ -178,7 +162,7 @@ For detailed information on both types of performance testing, including setup i ## Cron jobs -We have support for cron jobs using django-crontab. To set up cron jobs, edit the `CRONJOBS` variable in /backend/kernelCI/settings.py +We have support for cron jobs using django-crontab. To set up cron jobs, edit the `CRONJOBS` variable in `kernelCI/settings.py`. To run said cron jobs locally, execute ```poetry run ./manage.py crontab add``` @@ -187,9 +171,9 @@ You can also use other args such as `show` to show the cron jobs and `remove` to These cron jobs will also be automatically executed from the backend container if you are running with docker. You can check that the cron jobs are listed inside the docker container with -```docker exec -it dashboard-backend-1 crontab -l``` +```docker compose exec backend crontab -l``` or -```docker exec -it dashboard-backend-1 poetry run ./manage.py crontab show``` +```docker compose exec backend poetry run ./manage.py crontab show``` If you are a developer and want to skip the cron job setup while you're testing, you can also export `SKIP_CRONJOBS=True` to skip cron jobs entirely. @@ -252,14 +236,13 @@ For more detailed developer resources, visit https://discord.com/developers/docs The email notification system is used with cron jobs to be able to send regular updates about specific actions to the relevant recipients. You can check more information about it on [notifications.md](../docs/notifications.md) -## IDE Specific: -You are free to use whichever tool you would like, but here are tips for specific IDEs - ## Monitoring and Metrics -The project includes monitoring using Prometheus and Grafana. See the [monitoring documentation](docs/monitoring.md) for detailed information about. +The project includes monitoring using Prometheus and Grafana. See the [monitoring documentation](../docs/monitoring.md) for details. + +## IDE setup -### (Optional): Running in PyCharm +### PyCharm (optional) In order to debug backend in PyCharm, just follow these steps: @@ -270,7 +253,7 @@ In order to debug backend in PyCharm, just follow these steps: - in `Run` session of the dialog, select `script`, then find the script `manage.py` at the `backend` folder - at `script` name input, just enter `runserver` - at `Environment Variables`, enter the following values: - - `DB_ENGINE`: `django.db.backends.postgresql`, `DB_NAME`: `dashboard`, `DB_USER`: `@profusion.mobi`, `DB_PASSWORD`: ``, `DB_HOST`: `127.0.0.1`, `DB_PORT`: `5432`, `DB_OPTIONS_CONNECT_TIMEOUT`: `16`, `DB_CONN_MAX_AGE`: `0`, `DB_CONN_HEALTH_CHECKS`: `False` + - `DB_ENGINE`: `django.db.backends.postgresql`, `DB_NAME`: `dashboard`, `DB_USER`: ``, `DB_PASSWORD`: ``, `DB_HOST`: `127.0.0.1`, `DB_PORT`: `5432`, `DB_OPTIONS_CONNECT_TIMEOUT`: `16`, `DB_CONN_MAX_AGE`: `0`, `DB_CONN_HEALTH_CHECKS`: `False` - `DEBUG`: `True` Quote character in password field is escaped normally with `\"` . diff --git a/dashboard/.env.example b/dashboard/.env.example index 25f0a8dfa..f57667e78 100644 --- a/dashboard/.env.example +++ b/dashboard/.env.example @@ -1,5 +1,5 @@ -# Use port 8000 to go directly to the backend/gunicorn. -# Use port 80 to go through Nginx (proxy service on Docker) +# Use port 8000 to go directly to the backend (host or Docker). +# Use port 9000 for the Nginx proxy with docker-compose.dev.yml / docker-compose.yml. VITE_API_BASE_URL=http://localhost:8000 # Feature Flags diff --git a/dashboard/README.md b/dashboard/README.md index 36d513bcf..a871472ab 100644 --- a/dashboard/README.md +++ b/dashboard/README.md @@ -1,28 +1,16 @@ # Running in development -- Install the packager manager [pnpm](https://pnpm.io/installation) - -- Install all dependencies with: +Install [pnpm](https://pnpm.io/installation), then from the `dashboard` directory: ```sh +cp .env.example .env # set VITE_API_BASE_URL and other variables pnpm install -``` - -Create a .env.development file (Do not forget to check and set the variables and their values) - -```sh - cp .env.example .env.development -``` - -After establishing the connection with Google Cloud and running `backend`, run the `dashboard` dev server with - -```sh pnpm dev ``` -## Running unit tests +Set `VITE_API_BASE_URL=http://localhost:8000` when the backend runs on the host. With [dev-environment.md](../docs/dev-environment.md) (`docker-compose.dev.yml`), use the proxy at `http://localhost:9000` or Vite directly at `http://localhost:5173`. -The frontend includes unit tests covering some parts of the source code. To run the tests, use the following command: +## Running unit tests ```sh pnpm test @@ -30,31 +18,14 @@ pnpm test ## Running end-to-end (e2e) tests -The project includes Playwright-based end-to-end tests. To run the tests, first set the test environment URL in your .env file: +Set `PLAYWRIGHT_TEST_BASE_URL` in `.env` (defaults to staging). Install browsers once with `pnpm exec playwright install`, then: ```sh -# Copy the example file -cp .env.example .env - -# Edit the .env file to set PLAYWRIGHT_TEST_BASE_URL to your desired environment -# Available environments: -# - Staging: https://staging.dashboard.kernelci.org (default) -# - Production: https://dashboard.kernelci.org -# - Local: http://localhost:5173 - -# Install Playwright browsers if you don't have them yet -pnpm exec playwright install +pnpm run e2e # all tests +pnpm run e2e-ui # UI mode for debugging ``` -Then run the e2e tests: - -```sh -# Run all e2e tests -pnpm run e2e - -# Run e2e tests with UI mode for debugging -pnpm run e2e-ui -``` +Available base URLs: staging (`https://staging.dashboard.kernelci.org`), production (`https://dashboard.kernelci.org`), local Vite (`http://localhost:5173`), local proxy (`http://localhost:9000`). ## E2E Test Selectors diff --git a/docs/IntegrationTests.md b/docs/IntegrationTests.md index 9f2047435..3b76f6c18 100644 --- a/docs/IntegrationTests.md +++ b/docs/IntegrationTests.md @@ -242,12 +242,12 @@ docker compose -f docker-compose.test.yml up test_db redis -d **Step 2: Execute migrations** ```bash -docker compose -f docker-compose.test.yml run --rm test_backend python manage.py migrate +docker compose -f docker-compose.test.yml run --rm test_backend poetry run python manage.py migrate ``` **Step 3: Populate database with test data** ```bash -docker compose -f docker-compose.test.yml run --rm test_backend python manage.py seed_test_data --clear +docker compose -f docker-compose.test.yml run --rm test_backend poetry run python manage.py seed_test_data --clear ``` **Step 4: Start test backend** @@ -308,7 +308,7 @@ TEST_BASE_URL=http://localhost:8001 poetry run pytest -m integration --use-local - Verify backend is running on port 8001: `curl http://localhost:8001/` **Test data issues:** -- Clear and repopulate database: `docker compose -f docker-compose.test.yml run --rm test_backend python manage.py seed_test_data --clear` +- Clear and repopulate database: `docker compose -f docker-compose.test.yml run --rm test_backend poetry run python manage.py seed_test_data --clear` **Clean up everything:** ```bash diff --git a/docs/Onboarding.md b/docs/Onboarding.md index 882494713..c8a1d6f44 100644 --- a/docs/Onboarding.md +++ b/docs/Onboarding.md @@ -1,6 +1,6 @@ # Getting Onboarded on the KernelCI dashboard project. -This onboarding is written in the form of Tasks that you you can complete to get acquainted with the KernelCI Dashboard project. +This onboarding is written in the form of Tasks that you can complete to get acquainted with the KernelCI Dashboard project. ## Introduction The KernelCI Dashboard is composed by two main parts @@ -16,7 +16,7 @@ Our backend also houses an email notification system and a submissions monitorin 2. The KernelCI Dashboard Frontend This is the user interface that will be used to interact with the KernelCI Dashboard API. -Here the user can see the data returned by the API in a more user-friendly way and request diferents forms of visualization. +Here the user can see the data returned by the API in a more user-friendly way and request different forms of visualization. This frontend is written in TypeScript and uses the React library. > Note: @@ -24,9 +24,7 @@ This frontend is written in TypeScript and uses the React library. ## Tasks > Note: -> In case you don't have access to the backend, feel free to use the staging api to run the Frontend Code and send PRs. -> https://staging.dashboard.kernelci.org -> You can also ask for access in the #webdashboard channel in the KernelCI Discord. +> If you don't have backend access, see the [staging environment section](../CONTRIBUTING.md#staging-environment) in CONTRIBUTING.md. > Remember: > Always try to look to the production dashboard between tasks to see if you can assimilate the code to the project @@ -41,9 +39,9 @@ In order to access the production database, you must be granted access to it fir * Connect to the database via SSH tunnel with the provided URL. 2. You should ask for the creation of a new user/password for the database access. Once you have your credentials, connect to the database via `psql`, pgAdmin, DBeaver or any other postgresql manager. -3. Start up the local dashboard_db by starting its docker container and running the [migration script](../backend/migrate-app-db.sh). This secondary database is very useful for local development and you can modify it as much as you like. +3. Start the local `dashboard_db` via [dev-environment.md](dev-environment.md) (`make dev` runs migrations automatically), or start the database container yourself and run [migrate-app-db.sh](../backend/migrate-app-db.sh). -You can populate the local db with data dumps provided by colleagues, or run the the `monitor_submissions` command (aka "ingester") and use provided json files to insert data into the db. If you use the ingester, check the [monitor_submissions docs](../backend/docs/monitor_submissions.md) on its description. +You can populate the local db with data dumps provided by colleagues, or run the `monitor_submissions` command (aka "ingester") and use provided json files to insert data into the db. If you use the ingester, check the [monitor_submissions docs](../backend/docs/monitor_submissions.md) on its description. Definition of Done: You have access to kcidb and created the local database. @@ -90,70 +88,25 @@ Definition of Done: You are able to query database data with no problem. ### Task 5: Run the Frontend locally 1. Go to the `dashboard` directory, see the [README.md](../dashboard/README.md) from the frontend and try running the project locally. -2. Look at the folder `api` and see how the requests are made, search for where those functions are being used (where the requests are made) and see if you can relate with the production dashboard. +2. Look at the folder `dashboard/src/api` and see how the requests are made, search for where those functions are being used and see if you can relate with the production dashboard. 3. Try to mess with the code, change some colors, add some logs, try to understand the code structure, if there is a library that you don't know, read the documentation on that. Definition of Done: You have the KernelCI Dashboard frontend running locally. -### Task 6: Run the project in docker +### Task 6: Run the project in Docker > [!TIP] -> Running the project with Docker is especially useful for testing, as the production instance also runs in containers. This setup provides a more similar environment to production and helps ensure consistency between development and deployment. +> Running the project with Docker is especially useful for testing, as the production instance also runs in containers. -> [!IMPORTANT] -> The current docker compose has settings meant for the staging deployment; for local development you'll need to open the `dashboard_db` service's port (set `5434:5432` so the external port doesn't conflict with the backend) and set `STAGING_EXTERNAL_HTTP_PORT` as `80` in `.env`. +Follow [dev-environment.md](dev-environment.md) (`make dev` or `docker compose -f docker-compose.dev.yml up -d`). Stop any local backend, frontend, SSH tunnel, and Redis first — if Redis was installed via snap, run `sudo snap stop redis`. -1. Make sure your backend, frontend, db ssh and Redis are **not** running locally. - - If Redis is running and you installed it with snap, stop it with: - ```bash - sudo snap stop redis - ``` +For notification setup (optional during onboarding), see [notifications.md](notifications.md). -2. Set up the root `.env` file by copying `.env.example` and removing the `.example` suffix. For development you'll need to change the following variables in the `.env` file: -``` -DEBUG_SQL_QUERY=False -DEBUG=True - -DB_NAME=dashboard -DB_USER=admin -DB_PASSWORD=admin -DB_HOST=dashboard_db # Docker can't connect to the ssh tunnel host directly. -DJANGO_SECRET_KEY=$(openssl rand -base64 22) -``` - -If running the notification commands, add these variables to `.env`: -``` -EMAIL_HOST_USER= -EMAIL_HOST_PASSWORD= -``` -For the onboarding you can skip those, but do check out the [notifications](./notifications.md) part of the dashboard. - -Other environment variables can be set as needed. - -3. Start up the services with the command: - -```bash -docker compose up --build -d -``` - -After starting the services, you can check if your Docker containers are running with: - -```bash -docker ps -``` - -If any container fails or exits unexpectedly, you can inspect the logs with: - -```bash -docker logs -``` - -Definition of Done: The KernelCI Dashboard (backend and frontend) is running via Docker and accessible locally. +Definition of Done: The KernelCI Dashboard (backend and frontend) is running via Docker and accessible at `http://localhost:9000`. > [!NOTE] -> The Task 6 teaches you how to build and run the project locally during development. Note that you can also run the project using pre-built images generated by the [deploy-containers](../.github/workflows/deploy-containers.yaml) workflow, using the [docker-compose-next](../docker-compose-next.yml) file to pull and run these pre-built images without rebuilding them. +> Pre-built images from [deploy-containers](../.github/workflows/deploy-containers.yaml) can be run with [docker-compose-next.yml](../docker-compose-next.yml) instead of building locally. See [DEPLOYMENT.md](../DEPLOYMENT.md). ### Task 7: Complete a real Task 1. Go to https://github.com/kernelci/dashboard/issues and search for issues with the label `good first issue`. diff --git a/docs/dev-environment.md b/docs/dev-environment.md index 87a88f48a..5b66f15b4 100644 --- a/docs/dev-environment.md +++ b/docs/dev-environment.md @@ -1,6 +1,8 @@ # Local Development Environment -This document explains how to run the KernelCI Dashboard locally with live reload for both the backend and the frontend. +Recommended Docker workflow for contributors. This document explains how to run the KernelCI Dashboard locally with live reload for both the backend and the frontend. + +Shorthand from the repo root: `make setup` (copy env files, install deps) then `make dev` (start the stack). ## Overview @@ -247,7 +249,7 @@ The running server process has the module in `sys.modules` and keeps serving fro The crash surfaces on the **next reload** (when another watched file changes), because Django then tries to re-import everything from scratch and finds the file missing. At that point the container exits. -**How to handle:** restore the file before making any other change, or immediately run `docker compose up -d backend` after restoring it. +**How to handle:** restore the file before making any other change, or immediately run `docker compose -f docker-compose.dev.yml up -d backend` after restoring it. If a `.pyc` file for the deleted module still exists in `__pycache__/`, Python will silently load from it even after the source is gone. Remove the stale `.pyc` to force the error to surface earlier: diff --git a/docs/monitoring.md b/docs/monitoring.md index d134ea1f0..81f6ac166 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -13,12 +13,12 @@ docker compose -f docker-compose.monitoring.yml up -d #### Option A: Using Docker Compose (Recommended) ```bash -# Add monitoring configuration to .env.backend -echo "PROMETHEUS_METRICS_ENABLED=true" >> .env.backend -echo "PROMETHEUS_METRICS_PORT=8001" >> .env.backend -echo "PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_multiproc_dir" >> .env.backend +# Add monitoring configuration to root .env +echo "PROMETHEUS_METRICS_ENABLED=true" >> .env +echo "PROMETHEUS_METRICS_PORT=8001" >> .env +echo "PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_multiproc_dir" >> .env -# Start the backend with monitoring enabled +# Start the backend with monitoring enabled (uses root .env) docker compose up -d backend # The backend will automatically expose port 8001 for metrics when PROMETHEUS_METRICS_ENABLED=true @@ -105,7 +105,7 @@ The backend can ping healthcheck.io for cronjobs that run Django management comm - Private monitor tokens stay in environment variables and are mapped in Django settings. - Each monitored cron run sends pings to `/start`, `/success`, and `/fail`. -Configure these variables in `.env.backend`: +Configure these variables in root `.env` (or export them for manual backend runs): - `HEALTHCHECK_ID_DELETE_UNUSED_HARDWARE_STATUS` - `HEALTHCHECK_ID_NOTIFICATIONS_HARDWARE_SUMMARY` diff --git a/docs/notifications.md b/docs/notifications.md index 9dd741196..d875ce3da 100644 --- a/docs/notifications.md +++ b/docs/notifications.md @@ -16,7 +16,7 @@ If you are running the project in a docker container, you should add a `docker c ## Actions -The command supports four primary actions: +The command supports these actions: 1. `new_issues` * Generates a summary of new issues. @@ -53,7 +53,7 @@ These options are available for all actions and are always optional: | `--id` | `issue_report`, `test_report` | **Required** for `test_report`; **Required** for `issue_report` unless `--all` is used | Issue ID or Test ID in Dashboard/KCIDB | | `--all` | `issue_report` | **Alternative** to `--id` | Create reports for all issues not sent or not ignored | | `--update-storage` / `-u` | `issue_report` | Optional | Update JSON storage while generating/sending reports | -| `--summary-signup-folder` | `summary` | Optional | Alternative signup folder under `/backend/data` | +| `--summary-signup-folder` | `summary` | Optional | Alternative signup folder under `backend/data` | | `--summary-origins` | `summary` | Optional | Comma-separated list to limit to specific origins | | `--skip-sent-reports` | `summary` | Optional | Skip reports that have already been sent | | `--hardware-origins` | `hardware_summary` | Optional | Comma-separated list to limit to specific origins | diff --git a/docs/performanceTests.md b/docs/performanceTests.md index 88675c979..c83b51dac 100644 --- a/docs/performanceTests.md +++ b/docs/performanceTests.md @@ -16,7 +16,7 @@ K6 is used for performance and load testing of API endpoints. Run tests using Docker Compose: ```bash -docker-compose -f docker-compose.k6.yml up +docker compose -f docker-compose.k6.yml up ``` The Docker setup handles all K6 dependencies automatically. diff --git a/docs/verify_env.md b/docs/verify_env.md index 6ea723ecf..d28e91f32 100644 --- a/docs/verify_env.md +++ b/docs/verify_env.md @@ -63,6 +63,6 @@ Optional email overrides: ## Notes -- The command uses settings from `kernelCI/settings.py` plus the environment values in `.env.backend`. +- The command uses settings from `kernelCI/settings.py` plus environment values from root `.env` when run via Docker Compose. For manual backend runs, export variables from [`.env.backend.example`](../.env.backend.example). - Non-sending checks are read-only and safe for routine manual execution (storage checks are read-only unless `--create-dirs` is used). - Use send mode with a mailbox you control, so you can confirm deliverability and final headers. diff --git a/k6/data/README.md b/k6/data/README.md index 216d1c983..9a5a9250b 100644 --- a/k6/data/README.md +++ b/k6/data/README.md @@ -20,7 +20,7 @@ it will then ask for the user password and start dumping the database. This will 1. Place your SQL dump file as `dump.sql` in this directory 2. The `init-db.sh` script will automatically restore it when the container starts -3. Run: `docker-compose -f docker-compose.k6.yml up` +3. Run: `docker compose -f docker-compose.k6.yml up` **Smart Restore**: The script will only restore the database if it's empty (no user tables in the 'public' schema). @@ -51,6 +51,6 @@ psql -h localhost -p 5436 -U admin -d dashboard < your_dump_file.sql To start with a fresh database: ```bash -docker-compose -f docker-compose.k6.yml down -v -docker-compose -f docker-compose.k6.yml up -``` \ No newline at end of file +docker compose -f docker-compose.k6.yml down -v +docker compose -f docker-compose.k6.yml up +```