Async replication: primary retains stale replica config after node restart, HAProxy rejects it

[AKhozya]

Description

After a Kubernetes node restart, the MySQL primary pod retains stale replication configuration from its PVC. MySQL auto-starts the SQL applier thread, which causes the HAProxy haproxy_check_primary.sh health check to reject the primary as NOT OK, resulting in <NOSRV> for the mysql-primary backend.

Environment

• Operator version: 1.0.0
• MySQL version: Percona Server 8.4.7-7.1
• Cluster type: async (2 replicas)
• Kubernetes: K3s v1.35.1
• HAProxy: percona/haproxy:2.8.18

Steps to Reproduce

1. Deploy a 2-node async replication cluster with HAProxy
2. Restart the Kubernetes node hosting the primary MySQL pod
3. After pod recreation, the primary starts with stale replica configuration

Observed Behavior

After node restart, the primary (mysql-0) has a stale SHOW REPLICA STATUS pointing to the old replica:

Source_Host: main-mysql-mysql-1.main-mysql-mysql.databases
Replica_IO_Running: No
Replica_SQL_Running: Yes
Seconds_Behind_Source: NULL

The HAProxy haproxy_check_primary.sh script (check_async function) requires both REP_IO_STATUS != 'ON' and REP_SQL_STATUS != 'ON' for a healthy primary:

if [[ ${SUPER_RO} == '0' ]] && [[ ${READ_ONLY} == '0' ]] && [[ ${REP_IO_STATUS} != 'ON' ]] && [[ ${REP_SQL_STATUS} != 'ON' ]]; then
    # OK

Since Replica_SQL_Running: Yes (stale thread), the check fails → <NOSRV> → all MySQL clients get "Connection lost: The server closed the connection."

HAProxy logs show:

{"frontend_name": "mysql-primary-in", "backend_name": "mysql-primary", "server_name":"<NOSRV>", "termination_state": "SC"}

The Percona CR stays in state: initializing indefinitely. Orchestrator sees the primary with Problems: ["not_replicating"].

Expected Behavior

After a node restart, the operator or Orchestrator should either:

1. Detect and clean up stale replica configuration on the primary (STOP REPLICA; RESET REPLICA ALL;)
2. Or configure skip-replica-start in the MySQL config to prevent auto-starting stale replication threads

Workaround

Adding skip-replica-start to the MySQL configuration via the CR prevents the issue:

mysql:
  configuration: |
    [mysqld]
    skip-replica-start

This prevents MySQL from auto-starting replication threads on startup. Orchestrator then properly handles START REPLICA only on the actual replica node.

Manual fix without config change: connect to the primary and run STOP REPLICA; RESET REPLICA ALL;

Root Cause Analysis

The MySQL data directory (on PVC) persists relay log info and replica configuration. When the primary pod restarts, MySQL reads this persisted state and starts the SQL applier thread. The Percona operator and Orchestrator do not detect or remediate this stale state on the primary.

Suggestion

Consider one of:

1. Operator-level fix: Add skip-replica-start to the default MySQL config for async clusters, since Orchestrator manages replication lifecycle
2. Orchestrator-level fix: Detect when the primary has active replication threads with no IO thread and run RESET REPLICA ALL
3. Init container fix: Add a step to the mysql-init container that cleans up stale replica config before MySQL starts

[egegunes]

@AKhozya I created K8SPS-661 to investigate and fix this.

[mayankshah1607]

@AKhozya I wasn't able to reproduce this issue at least on GKE.

But it might be worth noting that the pod's startup probe runs a bootstrap process (found in /opt/percona/bootstrap) that executes RESET REPLICA ALL if its the primary [1]. If this runs correctly, you shouldn't be seeing this issue.

I don't know if there's an easy way to get the logs from the startup probe, but looking at the failing pod events might give some idea. Or you could also try to exec into the pod and run /opt/perona/bootstrap manually to see if it succeeds.

[1] https://github.com/percona/percona-server-mysql-operator/blob/main/cmd/bootstrap/async/async_replication.go#L124

[egegunes]

Startup probe logs can be found in /var/lib/mysql/bootstrap.log.

[AKhozya]

Thanks for looking into this. The bootstrap logs at /var/lib/mysql/bootstrap.log confirm that RESET REPLICA ALL runs successfully when the pod is the primary (we see "I'm the primary." entries).

However, the issue is a race condition, not a bootstrap failure. The sequence is:

1. Pod restarts, MySQL starts with PVC data containing stale replica config
2. Without skip-replica-start, MySQL auto-starts the SQL applier thread before the startup probe runs
3. HAProxy health check (haproxy_check_primary.sh) runs and sees Replica_SQL_Running: Yes → rejects the primary → <NOSRV>
4. Bootstrap startup probe eventually runs RESET REPLICA ALL, but by then HAProxy has already marked the backend as down

The bootstrap RESET REPLICA ALL works correctly — the problem is that MySQL starts the stale replication thread before bootstrap gets a chance to clean it up.

Our workaround (skip-replica-start in [mysqld] config) prevents MySQL from auto-starting replication threads, which eliminates the race entirely. Orchestrator then handles START REPLICA only on the actual replica.

This might explain why it doesn't reproduce on GKE — the timing depends on how fast the startup probe runs vs how quickly HAProxy begins health checking. On our K3s cluster with local NVMe storage, MySQL starts very quickly (PVC is already local), making the race window larger.

Would adding skip-replica-start to the default MySQL config for async clusters be considered? Since Orchestrator manages the replication lifecycle anyway, there's no reason for MySQL to auto-start replication threads from stale PVC state.

If you need me to execute some commands/install "potential fix" - happy to do so :)

[egegunes]

@mayankshah1607 let's add skip-replica-start by default and see if there are any unwanted side effects. but I think @AKhozya is right and it should be okay.