docs: update HAProxy configuration using conf.d architecture

Author: tazlin

.github/instructions/ansible-role-testing.instructions.md

@@ -21,7 +21,7 @@ applyTo: "tests/**/*.yml,tests/**/*.sh,roles/**/tasks/*.yml,roles/**/defaults/*.
 - `tests/ai_horde/test_ai_horde_render.yml`: Validates Docker mode rendering, security-sensitive compose/env contracts, and fail-fast password/key validation.
 - `tests/ai_horde/test_ai_horde_docker_build_toggle.yml`: Canonical owner of the build-on-start toggle contract (both enabled and disabled cases).
 - `tests/ai_horde/test_ai_horde_native_render.yml`: Validates native systemd template rendering, multi-instance output, and expected file permissions.
-- `tests/ai_horde/test_ai_horde_native_logic.yml`: Validates native HAProxy and instance management logic (stale unit handling, safe edits, ownership normalization gates).
+- `tests/ai_horde/test_ai_horde_native_logic.yml`: Validates native HAProxy conf.d architecture and instance management logic (stale unit handling, ownership normalization gates).
 - `tests/ai_horde/test_ai_horde_policy_contracts.yml`: Enforces reproducibility and policy contracts (pinned refs/digests, installer verification, readiness semantics).
 - `tests/artbot/test_artbot_render.yml`: Validates Artbot rendering and HAProxy/certbot contract output.
 - `tests/frontpage/test_frontpage_render.yml`: Validates Frontpage Docker-mode render contracts, env keys, healthcheck wiring, ports, and log rotation.

.github/instructions/repository-navigability.instructions.md

@@ -46,6 +46,7 @@ find the right folder before making changes.
 | Role path | Purpose |
 | --------- | ------- |
 | `roles/_uv_bootstrap/` | Shared micro-role: download, verify, and install a pinned uv version |
+| `roles/_haproxy_confd_bootstrap/` | Shared micro-role: install HAProxy, create conf.d directory, write systemd override for conf.d loading |
 | `roles/geerlingguy.swap/` | Third-party swap configuration role used as an infrastructure dependency |
 
 ## Meta folders to know first

docs/ai-horde-haproxy-topology.md

@@ -1,17 +1,33 @@
 # HAProxy Topology Guidance
 
-This document covers HAProxy mode selection and topology decisions for the
-`ai_horde` role. For variable reference and basic usage, see
+This document covers HAProxy topology decisions for the `ai_horde` role.
+For variable reference and basic usage, see
 [roles/ai_horde/README.md](../roles/ai_horde/README.md).
 
-## Mode Selection
+## conf.d Architecture
 
-- **Shared hosts** with existing HAProxy config: use `safe_edit` and a
-  non-privileged ingress port (for example `8080`).
-- **Dedicated ingress hosts**: `standalone` can be appropriate, including
-  privileged ingress ports.
+HAProxy configuration is managed via **conf.d drop-in fragments**. When
+`ai_horde_install_haproxy: true`, the role:
+
+1. Includes the shared `_haproxy_confd_bootstrap` micro-role to install
+   HAProxy, create `/etc/haproxy/conf.d/`, and write a systemd override
+   that loads all fragments from `conf.d/` alongside the main config.
+2. Templates the AI-Horde frontend/backend block to
+   `/etc/haproxy/conf.d/ai_horde.cfg`.
+3. Restarts HAProxy when the fragment changes.
+
+This approach is safe on shared hosts — each role drops its own fragment
+into `conf.d/` without touching unrelated configuration.
+
+## Topology Guidance
+
+- **Shared hosts** with existing HAProxy config: use
+  `ai_horde_install_haproxy: true` with a non-privileged ingress port
+  (for example `8080`). The conf.d fragment coexists with other services.
+- **Dedicated ingress hosts**: the same conf.d approach works; use
+  privileged ports if appropriate.
 - **Managed reverse proxy environments**: prefer upstream ingress and keep
-  `ai_horde_install_haproxy=false` unless local HAProxy is explicitly needed.
+  `ai_horde_install_haproxy: false` unless local HAProxy is explicitly needed.
 
 ## Privileged Binding
 
@@ -25,7 +41,6 @@ Copy/paste baseline for running AI-Horde behind an upstream reverse proxy:
 
 ```yaml
 ai_horde_install_haproxy: true
-ai_horde_haproxy_mode: safe_edit
 ai_horde_haproxy_port: 8080
 ai_horde_listen: "127.0.0.1"
 ```
@@ -35,9 +50,9 @@ traffic to AI-Horde on `:8080`.
 
 ## Operator Decision Matrix
 
-| Topology | Recommended HAProxy mode | Baseline ingress port | Notes |
+| Topology | HAProxy setup | Baseline ingress port | Notes |
 | ---- | ---- | ---- | ---- |
-| Single-host lab | `safe_edit` | `8080` | Lowest-conflict local baseline; easy to place another proxy in front later. |
-| Shared host with existing HAProxy | `safe_edit` | `8080` | Preserves unrelated HAProxy configuration via marker-bounded edits. |
-| Dedicated ingress host | `standalone` | `80` (and/or `443` via external TLS strategy) | Full config ownership is acceptable when no other stack owns HAProxy. |
-| Managed reverse proxy upstream | `disabled` (`ai_horde_install_haproxy=false`) | Upstream-owned | Keep AI-Horde behind upstream ingress; avoid duplicate proxy ownership. |
+| Single-host lab | `install_haproxy: true` | `8080` | Lowest-conflict local baseline; easy to place another proxy in front later. |
+| Shared host with existing HAProxy | `install_haproxy: true` | `8080` | conf.d fragment coexists with other service configs. |
+| Dedicated ingress host | `install_haproxy: true` | `80` (and/or `443` via external TLS strategy) | Full config ownership is acceptable when no other stack owns HAProxy. |
+| Managed reverse proxy upstream | `install_haproxy: false` | Upstream-owned | Keep AI-Horde behind upstream ingress; avoid duplicate proxy ownership. |

docs/monitoring/BACKUP.md

@@ -139,8 +139,9 @@ journalctl -u mimir-backup --since "24 hours ago" --no-pager
 
    (1500 minutes ≈ 25 hours, appropriate for a daily schedule)
 
-3. **node_exporter systemd collector:** If node_exporter is configured
-   with `--collector.systemd`, it exposes
+3. **Host metrics systemd collector:** If host metrics collection is configured
+   with a systemd collector (Alloy `prometheus.exporter.unix` or legacy
+   node_exporter with `--collector.systemd`), it exposes
    `node_systemd_unit_state{name="mimir-backup.service",state="failed"}`.
    An alert on this metric provides Prometheus-native backup monitoring.
 

docs/monitoring/OBSERVABILITY.md

@@ -49,7 +49,7 @@ For deployment order and baseline setup, see [MONITORING.md](../../MONITORING.md
 | `horde_monitoring` | Mimir, S3-compatible storage, Memcached, Grafana, Loki by default, optional Tempo/Pyroscope, Grafana datasource provisioning, monitoring alert rules, optional HAProxy backend insertion |
 | `horde_stats_exporter` | `horde-exporter` systemd service and optional downsampling timer |
 | `horde_alloy` | App-host telemetry collection and forwarding (metrics/logs/traces) |
-| `prometheus.prometheus.*` (playbook) | Prometheus, Alertmanager, and optional node_exporter (not managed by `horde_monitoring`) |
+| `prometheus.prometheus.*` (playbook) | Prometheus, Alertmanager (not managed by `horde_monitoring`; configured at the playbook level) |
 
 ## Full Stack Components
 
@@ -192,15 +192,19 @@ Complete example: [examples/alloy_app_host.yml](../../examples/alloy_app_host.ym
 
 ### Interaction With node_exporter Deployment
 
-In [examples/horde_monitoring_stack.yml](../../examples/horde_monitoring_stack.yml),
-node_exporter installation is decided per host using
-`horde_host_metrics_source`:
+**Grafana Alloy is the preferred host metrics source.** It collects host
+metrics natively via `prometheus.exporter.unix` and pushes them to Mimir,
+eliminating the need for a separate node_exporter binary and HAProxy
+basic-auth fronting.
+
+For legacy deployments that still run node_exporter, the monitoring stack
+example supports per-host selection via `horde_host_metrics_source`:
 
 - `auto` (recommended): skip node_exporter when the host has
   `horde_alloy_enabled: true` or `horde_alloy_collect_metrics: true`;
   otherwise install node_exporter.
-- `node_exporter`: force install on all hosts.
-- `alloy`: force skip on all hosts.
+- `node_exporter`: force install on all hosts (legacy).
+- `alloy`: force skip on all hosts (new deployments).
 
 ### Alloy Pipeline Toggles
 
@@ -217,7 +221,7 @@ Role validation fails fast if enabled pipelines do not have endpoints, or if
 ## End-To-End Data Flows
 
 - Exporter metrics: `horde-exporter` -> Prometheus scrape -> Mimir
-- Host metrics: node_exporter -> Prometheus scrape or Alloy `prometheus.exporter.unix` -> Mimir remote_write
+- Host metrics: Alloy `prometheus.exporter.unix` -> Mimir remote_write (preferred), or node_exporter -> Prometheus scrape (legacy)
 - Logs: Alloy journal/file/docker sources -> Loki push API
 - Traces: app OTLP -> Alloy OTLP receiver -> Tempo OTLP HTTP exporter
 - Trace-derived metrics: Tempo metrics generator -> Mimir remote_write

examples/ai_horde.yml

@@ -52,7 +52,6 @@
 #         ai_horde_listen: "127.0.0.1"
 #         ai_horde_admins: ["admin_user#1"]
 #         ai_horde_install_haproxy: true
-#         ai_horde_haproxy_mode: safe_edit
 #         ai_horde_haproxy_port: 8080
 
 # Production ingress variant (80/443) for dedicated ingress hosts only:

examples/alloy_app_host.yml

@@ -31,12 +31,13 @@
     horde_alloy_tenant_id: "infrastructure"
 
     # --- Host metrics source ---
-    # "node_exporter" keeps host metrics in Prometheus scrape path.
-    # "alloy" enables host metrics from Alloy prometheus.exporter.unix.
+    # "alloy" enables host metrics from Alloy prometheus.exporter.unix
+    # (preferred — no separate node_exporter binary needed).
+    # "node_exporter" keeps host metrics in Prometheus scrape path (legacy).
     # When using examples/horde_monitoring_stack.yml with
     # horde_host_metrics_source: auto, set horde_alloy_enabled: true
     # in inventory for these hosts so node_exporter is skipped implicitly.
-    horde_host_metrics_source: "node_exporter"
+    horde_host_metrics_source: "alloy"
 
     # --- Authentication ---
     horde_alloy_basic_auth_username: "alloy"

examples/full_horde_stack.yml

@@ -52,10 +52,8 @@
         ai_horde_backend_host: "127.0.0.1"
         ai_horde_admins: ["admin_user#1"]
         ai_horde_install_haproxy: true
-        ai_horde_haproxy_mode: safe_edit
         ai_horde_haproxy_port: "{{ horde_api_ingress_port }}"
         # Production ingress override for dedicated ingress hosts:
-        # ai_horde_haproxy_mode: standalone
         # ai_horde_haproxy_port: 80
         # Preconditions:
         #   - No conflicting listeners on 80/443.

examples/horde_monitoring_stack.yml

@@ -3,16 +3,17 @@
 # Usage: ansible-playbook -i inventory.yml horde_monitoring_stack.yml
 #
 # Architecture:
-#   Play 1: Deploy node_exporter only on hosts that resolve to node_exporter source
-#   Play 2: Deploy Mimir + Grafana + Prometheus + stats exporter on the monitoring host
+#   Deploys Mimir + Grafana + Prometheus + stats exporter on the monitoring host.
+#   Host metrics are collected by Grafana Alloy (preferred) or legacy
+#   node_exporter. See examples/alloy_app_host.yml for Alloy deployment.
 #
 #   Prometheus (native) --remote_write--> Mimir (Docker) <-- Grafana (Docker)
 #   All services bind to 127.0.0.1. HAProxy fronts Mimir externally with basic_auth.
 #
 # Prerequisites:
 #   ansible-galaxy collection install -r requirements.yml
-#   HAProxy is installed on each monitored host to front node_exporter
-#   with basic auth. node_exporter listens on localhost only.
+#   For legacy node_exporter hosts: HAProxy is installed on each monitored host
+#   to front node_exporter with basic auth. node_exporter listens on localhost only.
 
 # ---------------------------------------------------------------------------
 # Play 1: node_exporter on hosts using node_exporter as host metrics source

examples/inventory_monitoring.yml

@@ -51,11 +51,11 @@ all:
           ansible_user: admin
           ansible_become: true
           # Optional: override the address Prometheus should use for
-          # node_exporter scraping if ansible_host is not routable from
-          # the monitoring host.
+          # node_exporter scraping (legacy) if ansible_host is not routable
+          # from the monitoring host.
           # horde_node_exporter_scrape_address: "10.0.0.101"
-          # If this host is managed by examples/alloy_app_host.yml for host metrics,
-          # set horde_alloy_enabled: true so monitoring stack auto mode skips
+          # If this host uses Alloy for host metrics (preferred), set
+          # horde_alloy_enabled: true so monitoring stack auto mode skips
           # node_exporter on this host.
           # horde_alloy_enabled: true