[bitnami/redis-cluster]: add preStop hook that gracefully fails over master nodes on pod termination

[sobotklp]

Description of the change

This is a fix for #23036

It implements graceful failover for terminations of Redis Cluster master pods. This will limit the impact of terminations and failover events, since Redis cluster clients will have the chance to update their topology before the previous pod terminates. This will improve uptime during planned maintenance events.

Benefits

Improved uptime by failing master pods over to other replicas.

Possible drawbacks

If PodDisruptionBudget isn't being used, it may initiate a failover to a node that is also about to also be terminated. The script will attempt to filter out replicas that aren't candidates for promotion.

Applicable issues

• fixes [bitnami/redis-cluster] Request failover when a master node gracefully shuts down #23036

Additional information

This is an attempt to mirror the graceful failover behaviour of the redis and valkey charts when using Sentinel.

Checklist

• Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
• Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
• Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
• All commits signed off and in agreement of Developer Certificate of Origin (DCO)

[carrodher]

Thank you for initiating this pull request. We appreciate your effort. This is just a friendly reminder that signing your commits is important. Your signature certifies that you either authored the patch or have the necessary rights to contribute to the changes. You can find detailed information on how to do this in the “Sign your work” section of our contributing guidelines.

Feel free to reach out if you have any questions or need assistance with the signing process.

[sobotklp]

Thank you for initiating this pull request. We appreciate your effort. This is just a friendly reminder that signing your commits is important. Your signature certifies that you either authored the patch or have the necessary rights to contribute to the changes. You can find detailed information on how to do this in the “Sign your work” section of our contributing guidelines.

Feel free to reach out if you have any questions or need assistance with the signing process.

Hi @carrodher . Thanks for the tips on signing my commits. I've rebased my commit and amended my change with my signature. Let me know if I can do anything else. :)

[sobotklp]

The script uses this basic logic on pod termination:

• If the node is a master node, then
• • Get the node ID
• • List the available replicas for the node ID
• • Select a replica from the list and attempt to call CLUSTER FAILOVER.

I tested this by deploying a 9-node, 3-shard cluster locally without authentication or TLS, then manually terminating master nodes.
I could see that the commands ROLE and CLUSTER MYID were called on the terminating master node, as expected.
On the target replicas, I saw the expected output in the logs:

 * Manual failover user request accepted.
 * Received replication offset for paused master manual failover: 1148
 * All master replication stream processed, manual failover can start.
 * Start of election delayed for 0 milliseconds (rank #0, offset 1148).
 * Starting a failover election for epoch 11.

[sobotklp]

The script uses this basic logic on shutdown:

• If the node is a master node, then
• • Get the node ID
• • List the available replicas for the node ID
• • Select a replica from the list and attempt to call CLUSTER FAILOVER.

I tested this by deploying a 9-node, 3-shard cluster locally without authentication or TLS, then manually terminating master nodes. I could see that the commands ROLE and CLUSTER MYID were called on the terminating master node, as expected. On the target replicas, I saw the expected output in the logs:

 * Manual failover user request accepted.
 * Received replication offset for paused master manual failover: 1148
 * All master replication stream processed, manual failover can start.
 * Start of election delayed for 0 milliseconds (rank #0, offset 1148).
 * Starting a failover election for epoch 11.

It's also possible to kubectl exec -it -- bash into a master node and run the script manually, initiating a manual failover without terminating the pod.

[xqianwang]

Should we make this configurable instead of fixed 10 seconds?

[sobotklp]

it could be configurable up to a maximum of {{- $.Values.redis.terminationGracePeriodSeconds }} I would think.

[sobotklp]

I updated this to wait $.Values.redis.terminationGracePeriodSeconds - 10 seconds

[xqianwang]

This should cover 0 replicas found. But maybe its better to add a warning message?
No replica found for failover; proceeding with shutdown

[github-actions]

This Pull Request has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thank you for your contribution.

[sobotklp]

@migruiz4 would you like me to do anything more here to validate this PR?

[github-actions]

This Pull Request has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thank you for your contribution.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Pull Request. Do not hesitate to reopen it later if necessary.

[jmtekin]

Hi. Will this change be abandoned?

[sobotklp]

Though it would be nice to have this functionality directly in the chart, I was able to add it to the existing chart by adding the following values overrides:

redis:
...
   lifecycleHooks:
      preStop:
        exec:
          command:
            - /bin/bash
            - -ec
            - /custom-scripts/prestop-redis-cluster.sh
    extraVolumeMounts:
      - name: custom-scripts
        mountPath: /custom-scripts
    extraVolumes:
      - name: custom-scripts
        configMap:
          name: custom-redis-scripts
          defaultMode: 0755

then creating my own ConfigMap name: custom-redis-scripts with the content of prestop-redis-cluster.sh.

[github-actions]

This Pull Request has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thank you for your contribution.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Pull Request. Do not hesitate to reopen it later if necessary.