Implement secret cache in a memory friendly manner

[black-dragon74]

Please open a tracker for this too

Originally posted by @Rakshith-R in #5984 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[black-dragon74]

I’ve been thinking about this, and the most straightforward approach seems to be using a TTL-based expiry. Other options tend to introduce additional overhead without providing benefits that justify it.

If cache size becomes a concern at scale, we could also consider an LRU-based approach. However, I doubt that will be necessary since there are relatively few distinct secrets fetched during CSI operations. Happy to be corrected if that assumption is off.

One potential downside, as with any caching strategy, is the risk of invalidation issues or stale data. That said, CSI secrets typically don’t change very frequently. Given that, a TTL of around 30 seconds seems like a reasonable starting point. We could also make this value configurable.

Would appreciate hearing your thoughts @ceph/ceph-csi-contributors