Skip to content

Add a reusable debug-level diagnostic for the proxy chain reaching the API container #586

Description

@nivek0o0

Problem statement

#571 needed to know what X-Forwarded-For actually reaches the API container, because the correct TRUST_PROXY value is a property of the real request path and cannot be read off the repo. Nothing in the application could answer it: getLoggerOptions() in apps/api/src/app.ts installs a req serializer emitting only id, method, url and params, deliberately dropping Fastify's default remoteAddress, and no route echoes headers.

The measurement was ultimately taken with a throwaway plugin — a /debug/proxy-trust route plus per-request logging — deployed uncommitted and removed afterwards. That is the header-echo approach #571 argued against; it was acceptable only because it was temporary. So the reusable debug-level diagnostic #571 actually preferred was never built, and the next deployment or topology change has to repeat the throwaway.

Who this affects: anyone bringing up a new country deployment, or changing the proxy topology of an existing one — adding Front Door, moving behind a WAF, or self-hosting behind their own nginx. Each of those changes the hop count, and getting it wrong is silent in both directions (too few hops → one shared rate-limit bucket; too many → a caller can forge the header and pick its own bucket).

What this is not about

The operational question — "does this deployment bucket per client?" — is already answerable with no code at all, by reading x-ratelimit-remaining across two requests. That method is documented in docs/security/hardening.md ("Proxy Trust" → Verify afterwards) and works on any environment, including ones you cannot redeploy.

This issue is only about the narrower diagnostic question: what exactly is the chain, for when the answer is wrong or unexpected and you need to see the raw values. That is why this is priority: low rather than inheriting #571's high.

Proposed solution

A single debug-level log line in apps/api recording the proxy chain as received: x-forwarded-for, x-forwarded-proto, the resolved request.ip, request.ips, and the socket peer address.

infra/modules/appService.bicep:141 hardcodes LOG_LEVEL: 'info', so the line is silent in production by default. To measure: raise the app setting to debug, tail the log stream, restore.

Worth folding in: LOG_LEVEL is currently a hardcoded Bicep value, so a portal/CLI override is reset by the next main.bicep deployment. Making it a Bicep parameter would remove that trap and is a small change in the same area.

Constraints

These are the load-bearing part of this issue — they are why the throwaway was deleted rather than promoted.

1. debug level, not info. The throwaway logged at info, which is visible in production. A permanent line must be silent by default.

2. No route. #571's rejection stands: a public endpoint reflecting request headers is an information-disclosure surface that would have to be removed again. The throwaway got a pass because it was temporary and echoed only the caller's own request metadata.

3. It changes a documented security property — this needs a deliberate decision. docs/security/hardening.md:170 states:

No IP is written to any log line and no table in the schema stores a client IP — so there is no IP-based audit trail to corrupt with a forged header, and equally none to investigate an incident with.

A debug-gated IP log makes that conditional rather than absolute. Two consequences:

  • That sentence must be qualified in the same PR, or the doc becomes false.
  • Anyone raising LOG_LEVEL=debug to investigate an unrelated problem would begin writing client IPs to logs, without having decided to. For a public good handling EU-adjacent personal data, that is a privacy consequence worth choosing explicitly rather than inheriting.

Worth considering: the topology question may not need the addresses at all. Logging the shape — chain entry count, whether an entry carries a :port, whether the peer is link-local — answers "how many hops, and is the port suffix present" with no personal data. The tradeoff is that it loses the raw string #571 wanted to eyeball. Naming the tradeoff here rather than deciding it.

4. Nothing permanent touches the rate limiter. The throwaway lowered max to 5 on its own route to make a 429 reachable in a handful of requests. That was test scaffolding; autoConfig in apps/api/src/plugins/external/rate-limit.ts must not be modified for diagnostics.

Alternatives considered

  • The x-ratelimit-remaining check (already documented, no code). Answers verification, not diagnosis. This is the reason this issue is low priority rather than a prerequisite for anything.
  • App Service HTTP logs / Application Insights. docs/operations/risks-and-limitations.md records that App Insights is provisioned but not integrated into application code, so this shows the platform's view of the request rather than the container's — the wrong side of the question. (Raised on Measure the real X-Forwarded-For reaching the API container on the demo deployment #571.)
  • Repeat the throwaway each time. Works, but costs a build + deploy + revert per investigation, and puts a public debug route and IP logging on a live instance in the meantime.

Country-agnosticism

This is precisely the reuse case. Each adopting country provisions its own topology — App Service with or without Front Door, or self-hosted behind the operator's own reverse proxy — so the correct TRUST_PROXY differs per deployment and is not derivable from this repo. A diagnostic that answers the question in place, on the deployment itself is country-agnostic by construction: one code path, no country-specific branches, no per-country configuration beyond the existing LOG_LEVEL and TRUST_PROXY variables.

Primary SDG alignment

Other — internal observability with no direct user-facing SDG mapping. It supports the integrity of the rate limiter, which protects availability of the platform generally.

Additional context

  • #571 (closed) — the original measurement, the evidence gathered, and the alternatives it rejected. Its "Suggested approach" section is the direct source of the proposed shape here.
  • #569 — the proxy trust + rate-limit key change this diagnostic supports.
  • docs/security/hardening.md → "Proxy Trust" — the measured chain for plain App Service, and the no-code verification method.
  • apps/api/src/app.tsgetLoggerOptions() — the req serializer that drops remoteAddress, i.e. the reason nothing logs this today.
  • infra/modules/appService.bicep:141LOG_LEVEL hardcoded to info.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions