diff --git a/Changelog.md b/Changelog.md index 9f15c3431..1ffba2ddd 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,5 +1,9 @@ # Change Log +## 1.11.x (upcoming) + +- MONITORING_AUTH_RAW, is no longer used to configure monitoring authentication. Instead password entries must be entered directly in `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd`. See: [Docker-Metrics](https://github.com/internetstandards/Internet.nl/blob/main/documentation/Docker-metrics.md#monitoring-user/allowlist-management). If you had configured monitoring auth previously you need to move this into the new file. + ## 1.11.0 (in progress) _Compared to the latest 1.10 release._ @@ -11,7 +15,7 @@ All tests were updated to match the [2025-05 version of the NCSC TLS guidelines](https://www.ncsc.nl/en/transport-layer-security-tls/security-guidelines-for-transport-layer-security-2025-05). Most significant changes: -- The list of good/sufficient/phase out/insufficient TLS versions, TLS authentication, curves, hashes, +- The list of good/sufficient/phase out/insufficient TLS versions, TLS authentication, curves, hashes, key exchange algorithms, FFDHE groups, RSA key lengths, and bulk encryption algorithms were updated to match the new guidelines. - A test for Extended Master Secret (RFC7627) was added. @@ -29,13 +33,12 @@ Most significant changes: including some where servers preferred RSA over ECDHE, or CBC over POLY1305. - CCM_8 ciphers are now detected when enabled on a server. - OLD ciphers are no longer detected. -- The cipher order test no longer separates between "the server cipher order preference is wrong" +- The cipher order test no longer separates between "the server cipher order preference is wrong" and "the server has no preference". ### Significant internal changes - ... -### Possibly required changes to deployments ... @@ -107,7 +110,7 @@ The API version is updated to 2.6.0 due to the new CAA fields. - Fixed handling for [CAA with non-ascii characters](https://github.com/internetstandards/Internet.nl/pull/1788). - Fixed possible exception in [mail test prechecks](https://github.com/internetstandards/Internet.nl/pull/1787). - Fixed an [issue with rate limiting](https://github.com/internetstandards/Internet.nl/pull/1792). -- Update [Django to 4.2.22](https://github.com/internetstandards/Internet.nl/pull/1795) to fix +- Update [Django to 4.2.22](https://github.com/internetstandards/Internet.nl/pull/1795) to fix [CVE-2025-48432](https://www.djangoproject.com/weblog/2025/jun/04/security-releases/). ## 1.10.0 @@ -128,7 +131,7 @@ _Compared to the latest 1.9 release._ ### Significant internal changes -- The test code no longer interfaces with libunbound, but +- The test code no longer interfaces with libunbound, but [uses dnspython as a stub resolver](https://github.com/internetstandards/Internet.nl/pull/1578). - Periodic tests [are no longer enabled by default](https://github.com/internetstandards/Internet.nl/pull/1628). - UWSGI [cheaper](https://uwsgi-docs.readthedocs.io/en/latest/Cheaper.html) options are used to reduce idle processes and reduce memory consumption. @@ -159,7 +162,7 @@ docker network rm internetnl-prod_public-internet ## 1.9.3 - Updated the [expired PGP key](https://github.com/internetstandards/Internet.nl_content/pull/57). - + ## 1.9.2 - Fixed an issue where static files incorrectly required authentication (#1676) @@ -214,7 +217,7 @@ jobs to generate the same report over and over. 1.8.7 mainly contains various important fixes to support batch deployment. -* Updated sectxt to use a patched version of PGPy with a fix for a +* Updated sectxt to use a patched version of PGPy with a fix for a [catastrophic regex backtracking issue](https://github.com/SecurityInnovation/PGPy/pull/467) * Updated nassl to fix memory leak in OCSP check. * Connection test zones are now re-signed every week instead of every month. @@ -301,7 +304,7 @@ This release has API version 2.4.0: ## 1.7.1 -- Fixed the new [display of TLS versions](https://github.com/internetstandards/Internet.nl/issues/944) for mail tests. +- Fixed the new [display of TLS versions](https://github.com/internetstandards/Internet.nl/issues/944) for mail tests. - Fixed a [language mix-up](https://github.com/internetstandards/Internet.nl/issues/941) in the security.txt labels. - Fixed an [issue with the connection test and CSP form-action](https://github.com/internetstandards/Internet.nl/issues/945) @@ -411,7 +414,7 @@ Bugfixes - Fix some minor typos and broken link [(#574)] [(#575)] - Add a missing ' in the frame-ancestors explanation [(#578)] - An empty part of Content Security Policy gives an error [(#583)] -- Recursion error when stripping nonces in IPv4 and IPv6 comparison [(#587)] +- Recursion error when stripping nonces in IPv4 and IPv6 comparison [(#587)] - Remove certificate from the certificate chain in the shipped cert chain file [(#614)] Dependencies @@ -718,19 +721,19 @@ Initial public release. --- Brief description for next version --- New -- +- Changes -- +- Bug Fixes - Dependencies -- +- Migrations -- +- Settings -- +- diff --git a/README.md b/README.md index e87b76e92..c2ed0b039 100644 --- a/README.md +++ b/README.md @@ -18,8 +18,8 @@ Platform which is a collaboration of partners from the internet community and the Dutch government. The platform's mission is to jointly promote the use of modern internet standards keeping the internet reliable and accessible for everybody. [ECP](https://ecp.nl/) provides for the administrative home of the -platform. [NLnet Labs](https://nlnetlabs.nl/) laid the foundation for -Internet.nl and the underlying tooling. +platform. [NLnet Labs](https://nlnetlabs.nl/) laid the foundation for +Internet.nl and the underlying tooling. From 1 April 2021 onwards, maintenance and further development will be carried out by the project team of the Internet Standards Platform. @@ -52,6 +52,7 @@ intended as an internet standards compliance test and not as a security test. To develop or run your own instance, see the [documentation overview](https://github.com/internetstandards/Internet.nl/blob/main/documentation/README.md). +For deployment instructions please refer to the documentation for the current release version: https://github.com/internetstandards/Internet.nl/tree/release/1.10.x/documentation ## Building blocks diff --git a/docker/batch-test.env b/docker/batch-test.env index ba3cb8604..ab23e9320 100644 --- a/docker/batch-test.env +++ b/docker/batch-test.env @@ -38,9 +38,6 @@ IPV4_IP_TEST_TARGET_MAIL_PUBLIC=172.16.43.52 IPV6_IP_TEST_TARGET_PUBLIC=fd00:43:1::51 IPV6_IP_TEST_TARGET_MAIL_PUBLIC=fd00:43:1::52 -# use easy test/test user/passwords for authenticated endpoints -MONITORING_AUTH_RAW='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1,test_raw:$apr1$6YuDyduL$706z.FPTe5c09R767N3W90' - LETSENCRYPT_STAGING=1 LETSENCRYPT_EMAIL=letsencrypt@example.com diff --git a/docker/compose.development.yaml b/docker/compose.development.yaml index bdf123a30..c18a1901f 100644 --- a/docker/compose.development.yaml +++ b/docker/compose.development.yaml @@ -26,6 +26,9 @@ services: # auto rebuild/reload when config files change - path: ./webserver/ action: rebuild + volumes: + # mount monitoring credentials for testing/development + - ./webserver/dev.htpasswd:/etc/nginx/htpasswd/monitoring.htpasswd app: develop: diff --git a/docker/compose.integration-tests.yaml b/docker/compose.integration-tests.yaml index f16736699..17e030dfa 100644 --- a/docker/compose.integration-tests.yaml +++ b/docker/compose.integration-tests.yaml @@ -192,6 +192,9 @@ services: public-internet: ipv6_address: $IPV6_IP_PUBLIC ipv4_address: $IPV4_WEBSERVER_IP_PUBLIC + volumes: + # mount monitoring credentials for testing/development + - ./webserver/dev.htpasswd:/etc/nginx/htpasswd/monitoring.htpasswd unbound: networks: diff --git a/docker/compose.yaml b/docker/compose.yaml index 6c77c4ac2..7685b4109 100644 --- a/docker/compose.yaml +++ b/docker/compose.yaml @@ -31,7 +31,6 @@ services: environment: - INTERNETNL_DOMAINNAME - IPV6_TEST_ADDR - - MONITORING_AUTH_RAW - AUTH_ALL_URLS - ALLOW_LIST - ROUTINATOR_ALLOW_LIST @@ -56,7 +55,11 @@ services: volumes: # persist certbot configuration between restarts - certbot-config:/etc/letsencrypt - - htpasswd-files:/etc/nginx/htpasswd/external + # include configured password for http basic auth (if enabled) + - $INTERNETNL_INSTALL_BASE/volumes/webserver/htpasswd:/etc/nginx/htpasswd + # mount old static password configuration for migration + - htpasswd-files:/etc/nginx/htpasswd-old + # share logs with logs exporter - nginx-logs-exporter:/var/log/nginx/prometheus-nginxlog-exporter/ healthcheck: @@ -798,6 +801,7 @@ services: profiles: - monitoring + - monitoring-exporters redis-exporter: image: ${DOCKER_IMAGE_REDIS_EXPORTER} @@ -818,6 +822,7 @@ services: profiles: - monitoring + - monitoring-exporters statsd-exporter: image: ${DOCKER_IMAGE_STATSD_EXPORTER} @@ -841,6 +846,7 @@ services: profiles: - monitoring + - monitoring-exporters healthcheck: # container image includes the test command, setting interval and start_interval here @@ -906,6 +912,7 @@ services: profiles: - monitoring + - monitoring-exporters docker_stats_exporter: # https://github.com/jan4843/docker_stats_exporter @@ -929,6 +936,7 @@ services: profiles: - monitoring + - monitoring-exporters nginx_logs_exporter: platform: linux/amd64 @@ -953,6 +961,7 @@ services: profiles: - monitoring + - monitoring-exporters query-exporter: image: ${DOCKER_IMAGE_QUERY_EXPORTER} diff --git a/docker/defaults.env b/docker/defaults.env index 19c0c5548..c755918ee 100644 --- a/docker/defaults.env +++ b/docker/defaults.env @@ -92,12 +92,6 @@ ALLOW_LIST= # comma separated of IP(v6) addresses/subnets that are allowed to access the /routinator endpoint (used for multi instance deployements ROUTINATOR_ALLOW_LIST= -# comma separated user:htpasswd_encrypted pairs for /grafana and /prometheus, and side wide -# password must already be encrypted -# please not that the value needs to be enclosed by single quotes to prevent interpolation of the dollar signs -# eg: MONITORING_AUTH_RAW='test1:$apr1$wGM8gxBe$DxGwifTGWZJ7nftK7LzFt/,user2:$apr1$BoZzsbb/$2NgfYCfF9lxmGrfSqsZKc/' -MONITORING_AUTH_RAW= - # Django debug mode, on test run without debug, same as production DEBUG=False diff --git a/docker/deploy.sh b/docker/deploy.sh index b53bea6fc..b6782ab4d 100755 --- a/docker/deploy.sh +++ b/docker/deploy.sh @@ -11,6 +11,7 @@ cp -v /dist/docker/* docker # put $RELEASE into the compose.sh file envsubst '$RELEASE' < docker/compose-dist.sh > docker/compose.sh chmod a+x docker/compose.sh +chmod a+x docker/user_manage.sh # set release version in local.env config echo "RELEASE='$RELEASE' # deploy $(date)" >> docker/local.env diff --git a/docker/develop.env b/docker/develop.env index 1dfe84f3f..b5bb66f40 100644 --- a/docker/develop.env +++ b/docker/develop.env @@ -11,9 +11,6 @@ COMPOSE_PROJECT_NAME=internetnl-develop # enable for testing batch api ENABLE_BATCH=True -# use easy test/test user/passwords for authenticated endpoints -MONITORING_AUTH='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1' - LETSENCRYPT_STAGING=1 LETSENCRYPT_EMAIL=letsencrypt@example.com diff --git a/docker/test.env b/docker/test.env index 13c2cbf11..fe9155e25 100644 --- a/docker/test.env +++ b/docker/test.env @@ -37,9 +37,6 @@ IPV4_IP_TEST_TARGET_MAIL_PUBLIC=172.16.43.52 IPV6_IP_TEST_TARGET_PUBLIC=fd00:43:1::51 IPV6_IP_TEST_TARGET_MAIL_PUBLIC=fd00:43:1::52 -# use easy test/test user/passwords for authenticated endpoints -MONITORING_AUTH_RAW='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1,test_raw:$apr1$6YuDyduL$706z.FPTe5c09R767N3W90' - LETSENCRYPT_STAGING=1 LETSENCRYPT_EMAIL=letsencrypt@example.com diff --git a/docker/user_manage.sh b/docker/user_manage.sh index 41a5ae172..d6838516d 100644 --- a/docker/user_manage.sh +++ b/docker/user_manage.sh @@ -1,4 +1,11 @@ #!/usr/bin/env sh + # Small wrapper around user mgmt script shipped in webserver image # For both convenience, and to have a suitable command to put in sudo -/usr/bin/docker compose --env-file=docker/defaults.env --env-file=docker/host.env --env-file=docker/local.env exec -ti webserver /user_manage_inner.sh "$1" "$2" + +set -e # fail on error + +# determine install base (parent of directory containing this file) +INTERNETNL_INSTALL_BASE=$(dirname "$(dirname "$(readlink -f "$0")")") + +"$INTERNETNL_INSTALL_BASE/docker/compose.sh" exec -ti webserver /user_manage_inner.sh "$1" "$2" diff --git a/docker/webserver/authentication.sh b/docker/webserver/authentication.sh index e3d132855..88953ae7e 100755 --- a/docker/webserver/authentication.sh +++ b/docker/webserver/authentication.sh @@ -1,10 +1,29 @@ #!/bin/sh -echo $MONITORING_AUTH_RAW|tr ',' '\n' >> /etc/nginx/htpasswd/monitoring.htpasswd -# enable basic auth when user/password is configured +# this script sets up nginx configuration files for user and IP authentication + +set -e # exit on error + +# migrate htpasswd file from old way to new storage location +if test -f /etc/nginx/htpasswd/users.htpasswd; then + echo "Existing user password configuration found, not migrating old configuration." +else + if ! test -f /etc/nginx/htpasswd-old/users.htpasswd; then + echo "No old user password configuration found, not migrating." + else + echo "Migrating old user password configuration" + cp /etc/nginx/htpasswd-old/users.htpasswd /etc/nginx/htpasswd/users.htpasswd + fi +fi + +# create empty password files if they don't exist touch /etc/nginx/conf.d/basic_auth.include +touch /etc/nginx/htpasswd/users.htpasswd +touch /etc/nginx/htpasswd/monitoring.htpasswd + +# enable basic auth when user/password is configured if [ "$AUTH_ALL_URLS" != "False" ] || [ "$ENABLE_BATCH" != "False" ]; then - echo 'auth_basic "Please enter your access username and password";auth_basic_user_file /etc/nginx/htpasswd/external/users.htpasswd;' > /etc/nginx/conf.d/basic_auth.include + echo 'auth_basic "Please enter your access username and password";auth_basic_user_file /etc/nginx/htpasswd/users.htpasswd;' > /etc/nginx/conf.d/basic_auth.include fi # create IP allow list diff --git a/docker/webserver/dev.htpasswd b/docker/webserver/dev.htpasswd new file mode 100644 index 000000000..2526016ed --- /dev/null +++ b/docker/webserver/dev.htpasswd @@ -0,0 +1 @@ +test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1 diff --git a/docker/webserver/nginx_templates/default.conf.template b/docker/webserver/nginx_templates/default.conf.template index c9dc5112f..69c1e689a 100644 --- a/docker/webserver/nginx_templates/default.conf.template +++ b/docker/webserver/nginx_templates/default.conf.template @@ -278,7 +278,7 @@ server { # batch API, requires authentication and passes basic auth user to Django App via headers location /api/batch/v2 { auth_basic "Please enter your batch username and password"; - auth_basic_user_file /etc/nginx/htpasswd/external/users.htpasswd; + auth_basic_user_file /etc/nginx/htpasswd/users.htpasswd; # pass logged in user to Django proxy_set_header REMOTE-USER $remote_user; @@ -321,6 +321,80 @@ server { proxy_pass $alertmanager; } + # expose all metrics exporters for external monitoring + location /node-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $node_exporter http://node-exporter:9100; + proxy_pass $node_exporter; + } + location /postgresql-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $postgresql_exporter http://postgresql-exporter:9187/metrics; + proxy_pass $postgresql_exporter; + } + location /redis-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $redis_exporter http://redis-exporter:9121/metrics; + proxy_pass $redis_exporter; + } + location /rabbitmq-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $rabbitmq_exporter http://rabbitmq:15692/metrics; + proxy_pass $rabbitmq_exporter; + } + location /statsd-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $statsd_exporter http://statsd-exporter:9102/metrics; + proxy_pass $statsd_exporter; + } + location /celery-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $celery_exporter http://celery-exporter:9808/metrics; + proxy_pass $celery_exporter; + } + location /docker-stats-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $docker_stats_exporter http://docker_stats_exporter:9338/metrics; + proxy_pass $docker_stats_exporter; + } + location /nginx-logs-exporter { + include http.headers; + include hsts_h3.headers; + auth_basic "Please enter your monitoring username and password"; + auth_basic_user_file /etc/nginx/htpasswd/monitoring.htpasswd; + # set proxy_pass argument as variable, this makes sure nginx will still start even if this hostname is unresolvable due to monitoring profile being disabled + set $nginx_logs_exporter http://nginx_logs_exporter:4040/metrics; + proxy_pass $nginx_logs_exporter; + } + # routinator proxy for internal use on multi instance setups location /routinator/ { include http.headers; diff --git a/docker/webserver/user_manage_inner.sh b/docker/webserver/user_manage_inner.sh index 0c310d0d6..85ac8dce9 100755 --- a/docker/webserver/user_manage_inner.sh +++ b/docker/webserver/user_manage_inner.sh @@ -1,6 +1,6 @@ #!/usr/bin/env sh -HTPASSWD_FILE="/etc/nginx/htpasswd/external/users.htpasswd" +HTPASSWD_FILE="/etc/nginx/htpasswd/users.htpasswd" if [ ! -f "$HTPASSWD_FILE" ]; then touch "$HTPASSWD_FILE" diff --git a/documentation/Docker-deployment-batch.md b/documentation/Docker-deployment-batch.md index 6017024d9..ec882c3ea 100644 --- a/documentation/Docker-deployment-batch.md +++ b/documentation/Docker-deployment-batch.md @@ -87,19 +87,13 @@ Batch installations require the following settings: - `ENABLE_BATCH`: Must be set to `True`, to enable batch API - `ENABLE_HOF`: Must be set to `False`, to disable Hall of Fame processing - -And optionally: - -- `MONITORING_AUTH_RAW`: May be a comma separated list of `user:hashed-password` pairs which are allowed to access the metrics at `https://example.com/grafana/`. -- `AUTH_ALL_URLS` and `ALLOW_LIST`: Can be set to restrict access to the single scan webpage. See [Restricting Access](Docker-deployment.md#restricting-access) for more information. +- `ALLOW_LIST` : Can be set to allow visitors to the 'frontpage' which allows single tests to be performed. This is normally not needed but can be useful for manually testing. Batch test result pages are always accessible. For example: cat >> docker/local.env < # allowed IP's to visit web interface without password ALLOW_LIST=198.51.100.1,2001:db8:2::1 EOF diff --git a/documentation/Docker-deployment.md b/documentation/Docker-deployment.md index afc767943..dbd04e5ab 100644 --- a/documentation/Docker-deployment.md +++ b/documentation/Docker-deployment.md @@ -114,7 +114,7 @@ Spin up instance: ghcr.io/internetstandards/util:latest \ /deploy.sh -This command will take a long time (up to 30 minutes) due to RPKI data that needs to be synced initially. After that it should complete without an error, indicating the application stack is up and running healthy. You can already prepare continue with the DNS setup below in the meantime. +This command will take a a few minutes. After that it should complete without an error, indicating the application stack is up and running healthy. You can already prepare continue with the DNS setup below in the meantime. ## DNS setup @@ -130,6 +130,10 @@ After deployment is complete, all services are healthy and DNS is setup you can For more information see: [documentation/Docker-live-tests.md](Docker-live-tests.md) +It might be that RPKI tests are not giving proper results yet. This might be because the RPKI database needs to be synced. This happens automatically and should be finished ~30 minutes after deployment. + + + ## Compose command To reduce issues with different versions a Compose command is included in the installation and can be accessed using `/opt/Internet.nl/docker/compose.sh`. Use this command for everything where you would normaly use `docker compose` to manage the Compose project. @@ -286,9 +290,7 @@ Besides the single scan webpage, the Internet.nl application also contains a Bat ## Metrics (grafana/prometheus) -The default deployment includes a metrics collection system. It consists of a Prometheus metrics server with various exporters and a Grafana frontend. To view metrics and graphs visit: `https://example.com/grafana/`. Authentication is configured using the `MONITORING_AUTH_RAW` variable. - -Also see: [Metrics](Docker-metrics.md) +The default deployment includes a metrics collection system. It consists of a Prometheus metrics server with various exporters and a Grafana frontend. To view metrics and graphs visit: `https://example.com/grafana/`. For authentication and other information see: [Metrics](Docker-metrics.md) ## Monitoring/alerting @@ -376,8 +378,6 @@ To manage users, call the `/opt/Internet.nl/docker/user_manage.sh` script. This and a username. The operation can be `add_update` to add or update a user's password, `delete` to delete a user, and `verify` to verify a user's existence and password. Passwords are entered interactively. -If you would like users on the host to manage batch users, set sudo access for this script. - ### IP allow/deny lists Site wide IP(v6) allow lists can be configured by specifying the `ALLOW_LIST` variable. It should contain a comma separated list of IP(v6) addresses or subnets. diff --git a/documentation/Docker-metrics.md b/documentation/Docker-metrics.md index b66a278e7..2c9156f66 100644 --- a/documentation/Docker-metrics.md +++ b/documentation/Docker-metrics.md @@ -2,9 +2,7 @@ The Docker deployment includes a metrics collection system which is available on production as well as development/test environments. It consists of a Prometheus metrics server which scrapes metrics from various exporters. Grafana is provided as frontend to visualise metrics and create graphs/dashboards. -To view metrics and graphs visit the `/grafana/` endpoint. Eg: `http://localhost:8080/grafana/` for development and `https://example.com/grafana/` for production. For development the user/password is set to `test/test`, for production users can be configured using the `MONITORING_AUTH_RAW` variable in `docker/local.env` (see `docker/defaults.env` for information). - -Metrics collection is defined in the `docker/compose.monitoring.yaml` file. +To view metrics and graphs visit the `/grafana/` endpoint. Eg: `http://localhost:8080/grafana/` for development and `https://example.com/grafana/` for production. For development the user/password is set to `test/test`, for production users can be configured using the `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd` file, see below for information. ## Overview @@ -31,3 +29,17 @@ Dashboard can be manually created through the Grafana web interface and can be s ## Retention time By default metrics are retained for 5 years. This can be changed by settings the `PROMETHEUS_RETENTION_TIME` in the `docker/local.env` file. Optionally you can also set the `PROMETHEUS_RETENTION_SIZE` to limit the disk space used. + +## Monitoring user/allowlist management + +Monitoring endpoints are IP/password restricted. To allow users to visit these pages credentials or allowlists must be configured. + +Set `ALLOW_LIST` in `/opt/Internet.nl/docker/local.env` to allow certain IP's to access the monitoring without authentication, eg: `ALLOW_LIST=198.51.100.1,2001:db8:2::1` + +Add `user:passwordhash` entries to `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd`. These should be Apache2 htpassword hashes. To add or update a user password run: + + /opt/Internet.nl/docker/compose.sh exec webserver htpasswd /etc/nginx/htpasswd/monitoring.htpasswd username + +You will be prompted to enter the password. + +After updating `ALLOW_LIST` or changing users update the configuration by running: `/opt/Internet.nl/docker/compose.sh restart webserver`. diff --git a/documentation/github_release_steps.md b/documentation/github_release_steps.md index 11716e824..4720b1e53 100644 --- a/documentation/github_release_steps.md +++ b/documentation/github_release_steps.md @@ -16,10 +16,11 @@ code changes for the next release are already in the main branch. make translate_content_to_main ``` 4. Make a release branch for the x.y version if not already present (e.g., release/1.8.x). -5. Make a release on GitHub: +5. When updating major/minor versions, make sure to update the documentation link in `README.md` to point to the latest release documentation. +6. Make a release on GitHub: 1. Use tag 'vx.x.x' e.g., v1.2.3 for the release 2. Use release title: x.x.x 3. For the description use the contents of the Changelog.md for this release -6. Update the Changelog.md for the next release and commit with something like: +7. Update the Changelog.md for the next release and commit with something like: "- Bump for next version.". -7. Done. +8. Done. diff --git a/integration_tests/common/test_basic.py b/integration_tests/common/test_basic.py index 69f7fda33..47c939569 100644 --- a/integration_tests/common/test_basic.py +++ b/integration_tests/common/test_basic.py @@ -114,27 +114,13 @@ def test_monitoring_auth(page, app_url, path): response = requests.get(app_url + path, verify=False) assert response.status_code == 401 - # MONITORING_AUTH provided in docker/test.env + # credentials provided in docker/test.env auth = ("test", "test") response = requests.get(app_url + path, auth=auth, verify=False) response.raise_for_status() -def test_monitoring_auth_raw(page, app_url): - """Monitoring endpoints must be behind basic auth.""" - path = "/grafana" - - response = requests.get(app_url + path, verify=False) - assert response.status_code == 401 - - # MONITORING_AUTH_RAW provided in docker/test.env - auth = ("test_raw", "test_raw") - - response = requests.get(app_url + path, auth=auth, verify=False) - response.raise_for_status() - - def test_no_server_banner(page, app_url_subdomain): response = requests.get(app_url_subdomain, verify=False) assert response.headers["server"] == "nginx" diff --git a/integration_tests/conftest.py b/integration_tests/conftest.py index b18f98719..c84e4e0f2 100644 --- a/integration_tests/conftest.py +++ b/integration_tests/conftest.py @@ -162,7 +162,7 @@ def register_test_user(unique_id): # create test used in Apache2 password file command = ( f'docker compose --ansi=never --project-name "{COMPOSE_PROJECT_NAME}"' - f" exec webserver htpasswd -c -b /etc/nginx/htpasswd/external/users.htpasswd {username} {username}" + f" exec webserver htpasswd -c -b /etc/nginx/htpasswd/users.htpasswd {username} {username}" ) subprocess.check_call(command, shell=True, universal_newlines=True) diff --git a/pyproject.toml b/pyproject.toml index fb59ea0c5..9e017733a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -98,7 +98,7 @@ norecursedirs = [ filterwarnings = [ "ignore::DeprecationWarning:aiofiles.*:", "ignore::DeprecationWarning:.*", -# ignore:.*Django now detects this configuration.*:django.utils.deprecation.RemovedInDjango41Warning + "ignore::SyntaxWarning: invalid escape sequence.*", ] # flake8 does not support pyproject.toml, config moved to bin/lint.sh