How to handle secret rotations in general?

[vknabel]

In metal-stack there are multiple kinds of secrets that do expire. But currently there is no uniform or automated way of managing these. Some do get rotated while running, some need to be rolled manually, some do so during deployments (or at least this is one idea we had).

This topic came up in multiple issues:

• Rotation of headscale API key metal-roles#373
• Create JWT token for duros controller metal-roles#366
• metal-stack/metal-apiserver
• NSQ
• ...

Regarding all of these, there are multiple ideas, all with their own pros and cons.

Approach	Pros	Cons
Rotate on Deploy	Less common failure	Still requires deployment, customer specific pipelines
Alerts prior expiration	Transparency, suitable for external secrets	Still requires rotation
Rotate in Clients	Automated	Requires metal-apiserver, not reflected by k8s secrets

While it isn't possible to solve all of these with the same approach, I think it makes sense to have a discussion around these and how we want to handle those cases.

What do you think? Once we found our preferred approach for each use case, we could apply it to all matching secrets in a uniform way.