[bug] Sentinel mode: pub/sub reconnect goroutine wedges permanently when first dial after failover is refused — silent delivery loss, /health green

[joelmsanto]

Describe the bug

In Redis Sentinel mode, when a Redis primary dies and the pub/sub reconnect's first dial hits the dead primary's address (connection refused), the pub/sub retry goroutine stops retrying permanently. From that moment the affected Centrifugo pod silently delivers nothing for that shard's channels: client WebSocket connections stay open, publishes keep returning 200 and advancing history offsets, /health stays green — but no subscriber on that pod receives another publication until the process is restarted.

This looks like the same failure class as #766 (v4: node kept stale Redis state after Sentinel failover, health check green, manual restart required), which was closed after an upgrade without a root cause. We reproduce it on the latest v6.9.1 and can supply the evidence that issue lacked: the exact log signature, Redis-side PUBSUB CHANNELS state, and the code path.

Versions

• Centrifugo v6.9.1 (official Docker image, unmodified), 3 replicas on GKE
• Redis 8.8.0 via Bitnami redis chart 27.0.15, Sentinel mode (1 master + 2 replicas + 3 sentinels, quorum 2), two independent shards
• Engine config (nothing else non-default besides namespaces/delta):

"engine": {
  "type": "redis",
  "redis": {
    "address": [
      "redis+sentinel://redis-shard0:26379?sentinel_master_name=mymaster",
      "redis+sentinel://redis-shard1:26379?sentinel_master_name=mymaster"
    ]
  }
}

(Startup logs engine_mode: sharded(2):sentinel-x2. Also reproduced logic-wise with a single shard being the affected one — the wedge is per shard, per pod.)

Steps to Reproduce

1. Centrifugo v6.9.1, Redis engine via redis+sentinel:// DSN, at least one client subscribed (so the shard has channels).
2. Drive continuous publish load (we used 200-command /api/batch requests at ~15–20 req/s ≈ 3–4K publishes/s).
3. Kill the Redis primary pod (kubectl delete pod <primary>). The pod IP dies instantly; Sentinel promotion takes ~10s — that gap is the race window.
4. Pods whose reconnect dial lands in that window (connection refused from the stale primary address) wedge permanently. Pods whose dial happens to succeed (e.g. the StatefulSet re-created the pod name fast enough) recover normally — same cluster, same config, same drill.

Reproduced multiple times. A quiet failover (no publish traffic at reconnect time, dial succeeds) recovers correctly every time — which is presumably why this survives in the wild mostly unnoticed.

Log evidence

Stuck pod (debug logging enabled — the absence of lines after 18:10:25 is the point; observed silent for 150+ seconds until manual restart):

{"level":"error","error":"EOF","pub_sub_shard":0,"shard":"redis-shard1:26379","time":"2026-07-19T18:10:24Z","message":"pub/sub connection error"}
{"level":"error","error":"EOF","shard":"redis-shard1:26379","time":"2026-07-19T18:10:24Z","message":"control pub/sub connection error"}
{"level":"error","error":"dial tcp 10.124.0.56:6379: connect: connection refused","pub_sub_shard":0,"shard":"redis-shard1:26379","time":"2026-07-19T18:10:25Z","message":"pub/sub subscribe error"}
{"level":"error","error":"dial tcp 10.124.0.56:6379: connect: connection refused","shard":"redis-shard1:26379","time":"2026-07-19T18:10:25Z","message":"control pub/sub subscribe error"}

10.124.0.56 is the killed primary's pod IP. Note what's missing after the subscribe error: no deferred "stopping Redis PUB/SUB" (it's a defer in the same function), and no further "running Redis PUB/SUB" — ever.

Healthy recovery on the same cluster (quiet failover, for contrast — this is what every failover should look like):

{"level":"error","error":"EOF","pub_sub_shard":0,"shard":"redis-shard1:26379","message":"pub/sub connection error"}
{"level":"debug","num_processors":4,"pub_sub_shard":0,"shard":"redis-shard1:26379","message":"stopping Redis PUB/SUB"}
{"level":"debug","num_processors":4,"pub_sub_shard":0,"shard":"redis-shard1:26379","message":"running Redis PUB/SUB"}
{"level":"debug","elapsed":"394.368µs","num_channels":2,"pub_sub_shard":0,"shard":"redis-shard1:26379","message":"resubscribed to channels"}

Redis-side confirmation of the outage: with ~100 channels per shard actively subscribed, redis-cli PUBSUB CHANNELS 'centrifugo.client.*' on the shard master showed 1 channel instead of ~100 after the incident (the 1 came from a freshly connecting client, whose own SUBSCRIBE works — masking the outage for new clients while existing ones starve).

Expected behavior

runForever() in broker_redis.go is designed to re-run runPubSub() every 250ms indefinitely, so a refused dial should just be retried until Sentinel finishes promotion (a few seconds). That is exactly what happens in Redis Cluster mode on the same cluster: under identical dead-IP dial conditions (dial tcp ...: connect: connection refused / i/o timeout on reconnect), cluster mode logged clean stopping → running → resubscribed to channels cycles within 1–2s on every one of four master kills we drilled. Only the Sentinel path wedges.

Where it appears to be stuck

Reading runPubSubLoop() (redis_pubsub_shared.go), the error path after a failed shard-channel subscribe is: log "pub/sub subscribe error" → return → deferred conn.Close() / cancel() (rueidis DedicatedClient obtained from client.Dedicate()) → deferred "stopping Redis PUB/SUB" debug log. We observe the error log but never the deferred debug log and never another runForever iteration — so the goroutine appears to hang inside the dedicated-connection teardown after a dial that never succeeded, specifically when the client is a Sentinel-managed rueidis client whose master is gone. (Possibly adjacent to the rueidis blocking classes referenced in the OnMessage hook comment, redis/rueidis#596.)

Happy to provide more logs, run patched builds against our reproduction, or test candidate fixes — the drill reproduces this in a couple of minutes on demand.

[FZambia]

Hello @joelmsanto

Thanks for the detailed report!

Will try to reproduce, but generally if you can help with it further - will be nice, as the bug may only reproduce under some specific conditions.

What can be helpful:

• Full reproducer repo that uses Docker/minikube/GKE - ideal so it's possible to try without fedback loop
• See https://github.com/centrifugal/centrifugo/wiki/Investigating-performance-issues - once you have pod non recovering, would be nice to get goroutine dumps from it - both /debug/pprof/goroutine?debug=1 and /debug/pprof/goroutine?debug=2. You need to run Centrifugo with debug.enabled set to true to access these goroutine dumps. Also you can send SIGABRT signal to process, the process dies with goroutine dump printed to stderr (should be visible in pod logs kubectl logs <pod> --previous) - in that case no debug.enabled required.

[FZambia]

I tried to reproduce it in GKE, and everything worked for me. While I found several things that could be improved in the Sentinel client, I couldn't reproduce this exact pattern or the issue where the pod gets stuck forever without resubscribing. So it seems we really need the above ^^

[FZambia]

While I was trying to reproduce this reported case I found several things to improve in Redis Sentinel integration.

Some of them were addressed in the underlying Redis driver Centrifugo uses:

• feat(sentinel): add opt-in TopologyRefreshInterval to reconcile topology periodically
• fix(sentinel): back off between failed topology refreshes
• fix(pipe): don't let Close hang when the command ring is full
• fix(sentinel): do not close a connection still referenced by mConn/rConn
• fix(singleflight): give waiting callers the real error of the run they waited on

Centrifugo now periodically makes PUB/SUB liveness probes internally to detect issues on app level:

• Verify Redis PUB/SUB delivery with liveness probes - Centrifugo now periodically publishes a probe on each PUB/SUB connection and verifies it is delivered back. If it isn't — which is exactly your symptom, a connection that looks alive but delivers nothing — the connection is torn down and rebuilt.

All of the above ship in v6.9.2.

Since I couldn't reproduce your exact symptom — there may still be something else going on. Please:

• Run your drill against v6.9.2 and see whether the wedge still happens (and, if it does, whether the liveness probe recovers the pod on its own).
• If it does recur, a goroutine profile from the wedged pod while it's silent is needed. Or full reproducer to run and see the issue independently (as I mentioned here).