Observed via Nomad embedding consul-template v0.40.0.
If Vault is temporarily unreachable during the initial KVv2 autodetection for a secret "secret/..." template call, consul-template logs:
failed to check if <path> is KVv2, assume not
and then appears to cache that dependency as KVv1 for the lifetime of the dependency instance.
Once Vault connectivity returns, consul-template continues reading: /v1/secret/<path> instead of the correct KVv2 path: /v1/secret/data/<path>
This causes persistent 403 permission denied errors until the process is restarted. This can permanently block first template render after process start if the target mount is KVv2 and the policy only permits KVv2 data-path access. In Nomad, this can wedge allocation recovery because template prestart never completes.
Expected behavior
If KVv2 autodetection fails due to a transient startup error, consul-template should not permanently cache assume not KVv2 for that dependency instance.
Possible acceptable behaviors:
- retry KVv2 detection later
- keep KV version as unknown until detection succeeds
- re-run autodetection after connectivity is restored
- add an explicit option to force KV version
Actual behavior
A single transient autodetect failure poisons the dependency into KVv1 mode until process restart.
Observed via Nomad embedding consul-template v0.40.0.
If Vault is temporarily unreachable during the initial KVv2 autodetection for a
secret "secret/..."template call, consul-template logs:and then appears to cache that dependency as KVv1 for the lifetime of the dependency instance.
Once Vault connectivity returns, consul-template continues reading:
/v1/secret/<path>instead of the correct KVv2 path:/v1/secret/data/<path>This causes persistent 403 permission denied errors until the process is restarted. This can permanently block first template render after process start if the target mount is KVv2 and the policy only permits KVv2 data-path access. In Nomad, this can wedge allocation recovery because template prestart never completes.
Expected behavior
If KVv2 autodetection fails due to a transient startup error, consul-template should not permanently cache assume not KVv2 for that dependency instance.
Possible acceptable behaviors:
Actual behavior
A single transient autodetect failure poisons the dependency into KVv1 mode until process restart.