Skip to content

Add Vault Proxy as an optional standalone component - #1207

Open
khateeb15 wants to merge 3 commits into
hashicorp:mainfrom
khateeb15:feat/vault-proxy
Open

Add Vault Proxy as an optional standalone component#1207
khateeb15 wants to merge 3 commits into
hashicorp:mainfrom
khateeb15:feat/vault-proxy

Conversation

@khateeb15

Copy link
Copy Markdown

Summary

Adds Vault Proxy as an optional, opt-in component of the chart: a standalone Deployment that provides a cluster-local, caching Vault API endpoint in front of an in-cluster or external Vault server.

Closes #756. Also answers #929, which was closed in 2023 as "nothing to implement (yet)" while the Vault Proxy roadmap was still forming.

Problem

Today the chart offers two ways to put an Agent/Proxy in the request path, and both are per-pod: the Agent Injector's sidecar, and the Agent sidecar inside the CSI provider added in #749. Neither gives operators a shared, independently scalable proxy.

Running the proxy as its own Deployment instead of a sidecar means:

  • Centralised caching and renewal. Token and lease caching and background renewal happen in one place rather than once per pod.
  • Independent scaling and lifecycle. Scale, restart or roll the proxy without touching workloads, with topology-aware scheduling for zonal resilience.
  • One place to monitor. Aggregate Vault traffic is visible at a single point instead of spread across every injected sidecar.
  • Outage buffering. Clients keep reading cached responses while the Vault server is unreachable.

Vault 1.14 moved this functionality into the dedicated vault proxy subcommand and deprecated the Agent's API-proxy mode, so the component runs vault proxy rather than an agent in proxy mode.

What this adds

A new proxy.* values section rendering a Deployment, a ConfigMap holding the tpl-rendered HCL config, a Service, a ServiceAccount, and an optional PodDisruptionBudget. It follows the injector's conventions throughout: replicas, image, resources, affinity, topologySpreadConstraints, tolerations, nodeSelector, priorityClassName, strategy, annotations, extraLabels, extraArgs, extraEnvironmentVars, extraSecretEnvironmentVars, volumes, volumeMounts, securityContext.pod/.container, and serviceAccount.create/.name.

A checksum of the rendered config is set as a pod annotation, so a config change rolls the pods. The Vault address resolves to global.externalVaultAddr when set and to the chart's own Service otherwise, and values.openshift.yaml gains the matching UBI image.

Opt-in, and inert when disabled

proxy.enabled defaults to false and is deliberately not governed by global.enabled, so enabling the chart's other components never silently starts a proxy. With the proxy disabled, helm template output is byte-identical to current main (verified: 615 lines, zero diff).

Health probes default to TCP, not HTTP

Probes default to TCP socket checks on the listener, with HTTP paths available via proxy.livenessProbe.path / readinessProbe.path.

This is deliberate. An HTTP health check forwards through to Vault, so during a Vault outage the kubelet would restart the proxy and destroy the cache that is still serving requests, turning a degraded state into a total one. We have verified cache-served responses through a full Vault outage with zero pod restarts, and that a fresh pod cold-starts cleanly while Vault is unreachable: listener up, TCP-ready, auth handler retrying with backoff, self-recovering when Vault returns.

Optional external Service

proxy.service.externalService.enabled renders a second Service (<fullname>-proxy-external) for exposing the proxy outside the cluster. It is its own template and is independent of proxy.service.enabled, with its own type, port, targetPort, annotations (YAML-map or templated-string form), externalTrafficPolicy, loadBalancerIP and loadBalancerSourceRanges.

This required one fix to an existing helper: service.loadBalancer only resolved serviceType, so loadBalancerIP and loadBalancerSourceRanges were silently dropped for any service configured with a type key. It now falls back to type. The only other caller, ui-service.yaml, renders identically to before (verified).

On the caching caveat raised in #756

@tomhjp's point on #756 is correct and we reproduced it: the lease-cache index includes the client token, so with per-pod projected service account tokens, pass-through requests essentially never get cross-pod cache hits.

This PR does not attempt to relax cache-hit matching, which would need an Agent/Proxy-side feature. It makes the two patterns that exist today deployable and documents their trade-offs:

  1. Pass-through (the shipped default). Clients authenticate as themselves and send their own tokens, so Vault enforces each client's identity and policies exactly as it would without the proxy. Caching and renewal are per client token, so there is no cross-pod dedup. The value is per-client caching, centralised renewal and outage buffering.

  2. Shared identity (auto_auth plus api_proxy { use_auto_auth_token = true }, shipped as a commented example in values.yaml). Tokenless clients ride on the proxy's own Kubernetes service account, so cross-pod cache hits work and dynamic credentials become service-account-scoped, which is the model this issue originally asked for.

    The caveat is spelled out in the values documentation: authorization collapses to the proxy's role, so this is only sound where the proxy serves a single trust domain (replicas of one workload, or one proxy per app/namespace with a minimal bound role and network-restricted access to the Service). It is not a fit for one central proxy in front of heterogeneous workloads.

For clarity on semantics: with use_auto_auth_token = true the proxy's token is applied only to requests that carry no token; client-supplied tokens pass through unchanged. "force" is the variant that overrides them.

Backwards compatibility

Additive. proxy.enabled defaults to false, and the default rendered output is byte-identical to main. The single change to existing behaviour is the service.loadBalancer helper fix described above, which only adds fields that were previously dropped.

Testing

Unit tests follow the repo's existing conventions and cover the ConfigMap, Deployment, Service, external Service, ServiceAccount and PodDisruptionBudget:

bats test/unit/proxy-configmap.bats
bats test/unit/proxy-deployment.bats
bats test/unit/proxy-disruptionbudget.bats
bats test/unit/proxy-external-service.bats
bats test/unit/proxy-service.bats
bats test/unit/proxy-serviceaccount.bats

test/acceptance/proxy.bats adds a kind-based acceptance test covering pass-through requests, cache hits, tokenless auto-auth, and config-checksum rollout. Both suites are picked up automatically, since CI runs bats ./test/unit and bats ./test/acceptance over the whole directory.

helm lint passes.

PCI review checklist

  • I have documented a clear reason for, and description of, the change I am making.

  • If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

    The component is opt-in and disabled by default, and the default rendered output is unchanged. Reverting the pull request is sufficient.

  • If applicable, I've documented the impact of any changes to security controls.

    In the shipped default configuration the proxy is pass-through: clients present their own tokens and Vault enforces their identity and policies unchanged, so no authorization decision moves into the chart. The optional auto_auth pattern is shipped commented out and does move authorization to the proxy's own role; the values documentation states the trust-domain constraints that make it sound, and the accompanying loadBalancerSourceRanges support on the external Service exists so operators can restrict reachability. The proxy runs under its own ServiceAccount with a configurable securityContext, and the external Service is opt-in and disabled by default.

Vault Proxy provides a cluster-local, caching Vault API endpoint in front of
an in-cluster or external Vault server. Running it as a shared Deployment
rather than a per-pod sidecar centralises token and lease caching and renewal,
gives one place to monitor aggregate Vault traffic, and lets clients keep
reading cached responses while the Vault server is unreachable.

The component is opt-in: proxy.enabled defaults to false and is not governed
by global.enabled, so with the proxy disabled the rendered chart is unchanged.
It follows the injector's conventions and adds a Deployment, a ConfigMap
holding the tpl-rendered HCL config, a Service, a ServiceAccount and an
optional PodDisruptionBudget, along with the usual replicas, affinity,
topologySpreadConstraints, tolerations, resources and OpenShift knobs. A
checksum of the config is set as a pod annotation so a config change rolls the
pods. The Vault address resolves to global.externalVaultAddr when set and to
the chart's own Service otherwise.

Health probes default to TCP socket checks on the listener, with HTTP paths
available as an opt-in. An HTTP health check forwards through to Vault, so
during a Vault outage the kubelet would restart the proxy and destroy the cache
that is still serving requests.

An optional external Service, proxy.service.externalService.enabled, exposes
the proxy outside the cluster. It renders as its own template and is
independent of proxy.service.enabled, with its own type, port, targetPort,
annotations, externalTrafficPolicy and loadBalancerSourceRanges. The
service.loadBalancer helper now resolves `type` as a fallback to `serviceType`
so those fields render for services that use the `type` key; the only other
caller, the UI service, is unaffected.

Closes hashicorp#756
The test pinned hashicorp/vault:2.0.3 while values.yaml already defaults to
2.0.4, so it failed. Deriving the expected value from values.yaml keeps the
assertion meaningful without needing an update on every image bump.
The proxy could already be reached from outside the cluster through
proxy.service.externalService, which renders a LoadBalancer Service. An
Ingress is the HTTP-layer equivalent and is what most clusters already have a
controller for, so it completes the external exposure story rather than
forcing a cloud load balancer per proxy.

The new proxy.ingress.* values mirror server.ingress.*: enabled, labels,
annotations (YAML map or templated string), ingressClassName, pathType,
hosts, extraPaths and tls. activeService has no proxy equivalent and is
omitted, since the proxy has no active/standby split.

The Ingress backend is the proxy Service on proxy.service.port, so it renders
only when both proxy.enabled and proxy.service.enabled are true. That gate is
the new vault.proxyServiceEnabled helper, following vault.serverServiceEnabled.
proxy.ingress.enabled defaults to false, and the rendered chart with the proxy
disabled is unchanged.

Unlike server-ingress.yaml this is not gated on global.openshift. That gate
exists on the server because server-route.yaml is its OpenShift counterpart,
and there is no proxy Route to be exclusive with. OpenShift supports the
Ingress API, so gating would remove a working option and leave OpenShift
users with only the LoadBalancer Service.

The values documentation notes that anything able to reach the Ingress can
reach Vault through the proxy, and that with auto_auth plus
api_proxy.use_auto_auth_token those clients inherit the proxy's Vault role.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support deploying central Vault Agent HTTP Caching Proxy

1 participant