Skip to content

Transient Vault KVv2 autodetect failure is cached as KVv1 until restart #2149

Description

@skoppe

Observed via Nomad embedding consul-template v0.40.0.

If Vault is temporarily unreachable during the initial KVv2 autodetection for a secret "secret/..." template call, consul-template logs:

failed to check if <path> is KVv2, assume not

and then appears to cache that dependency as KVv1 for the lifetime of the dependency instance.

Once Vault connectivity returns, consul-template continues reading: /v1/secret/<path> instead of the correct KVv2 path: /v1/secret/data/<path>

This causes persistent 403 permission denied errors until the process is restarted. This can permanently block first template render after process start if the target mount is KVv2 and the policy only permits KVv2 data-path access. In Nomad, this can wedge allocation recovery because template prestart never completes.

Expected behavior

If KVv2 autodetection fails due to a transient startup error, consul-template should not permanently cache assume not KVv2 for that dependency instance.

Possible acceptable behaviors:

  • retry KVv2 detection later
  • keep KV version as unknown until detection succeeds
  • re-run autodetection after connectivity is restored
  • add an explicit option to force KV version

Actual behavior

A single transient autodetect failure poisons the dependency into KVv1 mode until process restart.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions