Configure prometheus exporters to be externally accessible

Changelog.md

@@ -1,5 +1,9 @@
 # Change Log
 
+## 1.11.x (upcoming)
+
+- MONITORING_AUTH_RAW, is no longer used to configure monitoring authentication. Instead password entries must be entered directly in `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd`. See: [Docker-Metrics](https://github.com/internetstandards/Internet.nl/blob/main/documentation/Docker-metrics.md#monitoring-user/allowlist-management). If you had configured monitoring auth previously you need to move this into the new file.
+
 ## 1.11.0 (in progress)
 
 _Compared to the latest 1.10 release._
@@ -11,7 +15,7 @@ All tests were updated to match the
 [2025-05 version of the NCSC TLS guidelines](https://www.ncsc.nl/en/transport-layer-security-tls/security-guidelines-for-transport-layer-security-2025-05).
 Most significant changes:
 
-- The list of good/sufficient/phase out/insufficient TLS versions, TLS authentication, curves, hashes, 
+- The list of good/sufficient/phase out/insufficient TLS versions, TLS authentication, curves, hashes,
   key exchange algorithms, FFDHE groups, RSA key lengths, and bulk encryption algorithms were updated
   to match the new guidelines.
 - A test for Extended Master Secret (RFC7627) was added.
@@ -29,13 +33,12 @@ Most significant changes:
   including some where servers preferred RSA over ECDHE, or CBC over POLY1305.
 - CCM_8 ciphers are now detected when enabled on a server.
 - OLD ciphers are no longer detected.
-- The cipher order test no longer separates between "the server cipher order preference is wrong" 
+- The cipher order test no longer separates between "the server cipher order preference is wrong"
   and "the server has no preference".
 
 ### Significant internal changes
 
 - ...
-### Possibly required changes to deployments
 
 ...
 
@@ -107,7 +110,7 @@ The API version is updated to 2.6.0 due to the new CAA fields.
 - Fixed handling for [CAA with non-ascii characters](https://github.com/internetstandards/Internet.nl/pull/1788).
 - Fixed possible exception in [mail test prechecks](https://github.com/internetstandards/Internet.nl/pull/1787).
 - Fixed an [issue with rate limiting](https://github.com/internetstandards/Internet.nl/pull/1792).
-- Update [Django to 4.2.22](https://github.com/internetstandards/Internet.nl/pull/1795) to fix 
+- Update [Django to 4.2.22](https://github.com/internetstandards/Internet.nl/pull/1795) to fix
   [CVE-2025-48432](https://www.djangoproject.com/weblog/2025/jun/04/security-releases/).
 
 ## 1.10.0
@@ -128,7 +131,7 @@ _Compared to the latest 1.9 release._
 
 ### Significant internal changes
 
-- The test code no longer interfaces with libunbound, but 
+- The test code no longer interfaces with libunbound, but
  [uses dnspython as a stub resolver](https://github.com/internetstandards/Internet.nl/pull/1578).
 - Periodic tests [are no longer enabled by default](https://github.com/internetstandards/Internet.nl/pull/1628).
 - UWSGI [cheaper](https://uwsgi-docs.readthedocs.io/en/latest/Cheaper.html) options are used to reduce idle processes and reduce memory consumption.
@@ -159,7 +162,7 @@ docker network rm internetnl-prod_public-internet
 ## 1.9.3
 
 - Updated the [expired PGP key](https://github.com/internetstandards/Internet.nl_content/pull/57).
-  
+
 ## 1.9.2
 
 - Fixed an issue where static files incorrectly required authentication (#1676)
@@ -214,7 +217,7 @@ jobs to generate the same report over and over.
 
 1.8.7 mainly contains various important fixes to support batch deployment.
 
-* Updated sectxt to use a patched version of PGPy with a fix for a 
+* Updated sectxt to use a patched version of PGPy with a fix for a
   [catastrophic regex backtracking issue](https://github.com/SecurityInnovation/PGPy/pull/467)
 * Updated nassl to fix memory leak in OCSP check.
 * Connection test zones are now re-signed every week instead of every month.
@@ -301,7 +304,7 @@ This release has API version 2.4.0:
 
 ## 1.7.1
 
-- Fixed the new [display of TLS versions](https://github.com/internetstandards/Internet.nl/issues/944) for mail tests. 
+- Fixed the new [display of TLS versions](https://github.com/internetstandards/Internet.nl/issues/944) for mail tests.
 - Fixed a [language mix-up](https://github.com/internetstandards/Internet.nl/issues/941) in the security.txt labels.
 - Fixed an [issue with the connection test and CSP form-action](https://github.com/internetstandards/Internet.nl/issues/945)
 
@@ -411,7 +414,7 @@ Bugfixes
 - Fix some minor typos and broken link [(#574)] [(#575)]
 - Add a missing ' in the frame-ancestors explanation [(#578)]
 - An empty part of Content Security Policy gives an error [(#583)]
-- Recursion error when stripping nonces in IPv4 and IPv6 comparison [(#587)] 
+- Recursion error when stripping nonces in IPv4 and IPv6 comparison [(#587)]
 - Remove certificate from the certificate chain in the shipped cert chain file [(#614)]
 
 Dependencies
@@ -718,19 +721,19 @@ Initial public release.
 --- Brief description for next version ---
 
 New
-- 
+-
 
 Changes
-- 
+-
 
 Bug Fixes
 -
 
 Dependencies
-- 
+-
 
 Migrations
-- 
+-
 
 Settings
-- 
+-

README.md

@@ -18,8 +18,8 @@ Platform which is a collaboration of partners from the internet community and
 the Dutch government. The platform's mission is to jointly promote the use of
 modern internet standards keeping the internet reliable and accessible for
 everybody. [ECP](https://ecp.nl/) provides for the administrative home of the
-platform. [NLnet Labs](https://nlnetlabs.nl/) laid the foundation for 
-Internet.nl and the underlying tooling. 
+platform. [NLnet Labs](https://nlnetlabs.nl/) laid the foundation for
+Internet.nl and the underlying tooling.
 
 From 1 April 2021 onwards, maintenance and further development will be carried
 out by the project team of the Internet Standards Platform.
@@ -52,6 +52,7 @@ intended as an internet standards compliance test and not as a security test.
 To develop or run your own instance, see the
 [documentation overview](https://github.com/internetstandards/Internet.nl/blob/main/documentation/README.md).
 
+For deployment instructions please refer to the documentation for the current release version: https://github.com/internetstandards/Internet.nl/tree/release/1.10.x/documentation
 
 ## Building blocks
 

docker/batch-test.env

@@ -38,9 +38,6 @@ IPV4_IP_TEST_TARGET_MAIL_PUBLIC=172.16.43.52
 IPV6_IP_TEST_TARGET_PUBLIC=fd00:43:1::51
 IPV6_IP_TEST_TARGET_MAIL_PUBLIC=fd00:43:1::52
 
-# use easy test/test user/passwords for authenticated endpoints
-MONITORING_AUTH_RAW='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1,test_raw:$apr1$6YuDyduL$706z.FPTe5c09R767N3W90'
-
 LETSENCRYPT_STAGING=1
 LETSENCRYPT_EMAIL=letsencrypt@example.com
 

docker/compose.development.yaml

@@ -26,6 +26,9 @@ services:
         # auto rebuild/reload when config files change
         - path: ./webserver/
           action: rebuild
+    volumes:
+      # mount monitoring credentials for testing/development
+      - ./webserver/dev.htpasswd:/etc/nginx/htpasswd/monitoring.htpasswd
 
   app:
     develop:

docker/compose.integration-tests.yaml

@@ -192,6 +192,9 @@ services:
       public-internet:
         ipv6_address: $IPV6_IP_PUBLIC
         ipv4_address: $IPV4_WEBSERVER_IP_PUBLIC
+    volumes:
+      # mount monitoring credentials for testing/development
+      - ./webserver/dev.htpasswd:/etc/nginx/htpasswd/monitoring.htpasswd
 
   unbound:
     networks:

docker/compose.yaml

@@ -31,7 +31,6 @@ services:
     environment:
       - INTERNETNL_DOMAINNAME
       - IPV6_TEST_ADDR
-      - MONITORING_AUTH_RAW
       - AUTH_ALL_URLS
       - ALLOW_LIST
       - ROUTINATOR_ALLOW_LIST
@@ -56,7 +55,11 @@ services:
     volumes:
       # persist certbot configuration between restarts
       - certbot-config:/etc/letsencrypt
-      - htpasswd-files:/etc/nginx/htpasswd/external
+      # include configured password for http basic auth (if enabled)
+      - $INTERNETNL_INSTALL_BASE/volumes/webserver/htpasswd:/etc/nginx/htpasswd
+      # mount old static password configuration for migration
+      - htpasswd-files:/etc/nginx/htpasswd-old
+      # share logs with logs exporter
       - nginx-logs-exporter:/var/log/nginx/prometheus-nginxlog-exporter/
 
     healthcheck:
@@ -798,6 +801,7 @@ services:
 
     profiles:
       - monitoring
+      - monitoring-exporters
 
   redis-exporter:
     image: ${DOCKER_IMAGE_REDIS_EXPORTER}
@@ -818,6 +822,7 @@ services:
 
     profiles:
       - monitoring
+      - monitoring-exporters
 
   statsd-exporter:
     image: ${DOCKER_IMAGE_STATSD_EXPORTER}
@@ -841,6 +846,7 @@ services:
 
     profiles:
       - monitoring
+      - monitoring-exporters
 
     healthcheck:
       # container image includes the test command, setting interval and start_interval here
@@ -906,6 +912,7 @@ services:
 
     profiles:
       - monitoring
+      - monitoring-exporters
 
   docker_stats_exporter:
     # https://github.com/jan4843/docker_stats_exporter
@@ -929,6 +936,7 @@ services:
 
     profiles:
       - monitoring
+      - monitoring-exporters
 
   nginx_logs_exporter:
     platform: linux/amd64
@@ -953,6 +961,7 @@ services:
 
     profiles:
       - monitoring
+      - monitoring-exporters
 
   query-exporter:
     image: ${DOCKER_IMAGE_QUERY_EXPORTER}

docker/defaults.env

@@ -92,12 +92,6 @@ ALLOW_LIST=
 # comma separated of IP(v6) addresses/subnets that are allowed to access the /routinator endpoint (used for multi instance deployements
 ROUTINATOR_ALLOW_LIST=
 
-# comma separated user:htpasswd_encrypted pairs for /grafana and /prometheus, and side wide
-# password must already be encrypted
-# please not that the value needs to be enclosed by single quotes to prevent interpolation of the dollar signs
-# eg: MONITORING_AUTH_RAW='test1:$apr1$wGM8gxBe$DxGwifTGWZJ7nftK7LzFt/,user2:$apr1$BoZzsbb/$2NgfYCfF9lxmGrfSqsZKc/'
-MONITORING_AUTH_RAW=
-
 # Django debug mode, on test run without debug, same as production
 DEBUG=False
 

docker/deploy.sh

@@ -11,6 +11,7 @@ cp -v /dist/docker/* docker
 # put $RELEASE into the compose.sh file
 envsubst '$RELEASE' < docker/compose-dist.sh > docker/compose.sh
 chmod a+x docker/compose.sh
+chmod a+x docker/user_manage.sh
 
 # set release version in local.env config
 echo "RELEASE='$RELEASE' # deploy $(date)" >> docker/local.env

docker/develop.env

@@ -11,9 +11,6 @@ COMPOSE_PROJECT_NAME=internetnl-develop
 # enable for testing batch api
 ENABLE_BATCH=True
 
-# use easy test/test user/passwords for authenticated endpoints
-MONITORING_AUTH='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1'
-
 LETSENCRYPT_STAGING=1
 LETSENCRYPT_EMAIL=letsencrypt@example.com
 

docker/test.env

@@ -37,9 +37,6 @@ IPV4_IP_TEST_TARGET_MAIL_PUBLIC=172.16.43.52
 IPV6_IP_TEST_TARGET_PUBLIC=fd00:43:1::51
 IPV6_IP_TEST_TARGET_MAIL_PUBLIC=fd00:43:1::52
 
-# use easy test/test user/passwords for authenticated endpoints
-MONITORING_AUTH_RAW='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1,test_raw:$apr1$6YuDyduL$706z.FPTe5c09R767N3W90'
-
 LETSENCRYPT_STAGING=1
 LETSENCRYPT_EMAIL=letsencrypt@example.com
 

docker/user_manage.sh

@@ -1,4 +1,11 @@
 #!/usr/bin/env sh
+
 # Small wrapper around user mgmt script shipped in webserver image
 # For both convenience, and to have a suitable command to put in sudo
-/usr/bin/docker compose --env-file=docker/defaults.env --env-file=docker/host.env --env-file=docker/local.env exec -ti webserver /user_manage_inner.sh "$1" "$2"
+
+set -e # fail on error
+
+# determine install base (parent of directory containing this file)
+INTERNETNL_INSTALL_BASE=$(dirname "$(dirname "$(readlink -f "$0")")")
+
+"$INTERNETNL_INSTALL_BASE/docker/compose.sh" exec -ti webserver /user_manage_inner.sh "$1" "$2"

docker/webserver/authentication.sh

@@ -1,10 +1,29 @@
 #!/bin/sh
-echo $MONITORING_AUTH_RAW|tr ',' '\n' >> /etc/nginx/htpasswd/monitoring.htpasswd
 
-# enable basic auth when user/password is configured
+# this script sets up nginx configuration files for user and IP authentication
+
+set -e # exit on error
+
+# migrate htpasswd file from old way to new storage location
+if test -f /etc/nginx/htpasswd/users.htpasswd; then
+  echo "Existing user password configuration found, not migrating old configuration."
+else
+  if ! test -f /etc/nginx/htpasswd-old/users.htpasswd; then
+    echo "No old user password configuration found, not migrating."
+  else
+    echo "Migrating old user password configuration"
+    cp /etc/nginx/htpasswd-old/users.htpasswd /etc/nginx/htpasswd/users.htpasswd
+  fi
+fi
+
+# create empty password files if they don't exist
 touch /etc/nginx/conf.d/basic_auth.include
+touch /etc/nginx/htpasswd/users.htpasswd
+touch /etc/nginx/htpasswd/monitoring.htpasswd
+
+# enable basic auth when user/password is configured
 if [ "$AUTH_ALL_URLS" != "False" ] || [ "$ENABLE_BATCH" != "False" ]; then
-  echo 'auth_basic "Please enter your access username and password";auth_basic_user_file /etc/nginx/htpasswd/external/users.htpasswd;' > /etc/nginx/conf.d/basic_auth.include
+  echo 'auth_basic "Please enter your access username and password";auth_basic_user_file /etc/nginx/htpasswd/users.htpasswd;' > /etc/nginx/conf.d/basic_auth.include
 fi
 
 # create IP allow list

docker/webserver/dev.htpasswd

@@ -0,0 +1 @@
+test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1