Flaky test: ShardedDaemonProcessProxySpecs.ShardedDaemonProcessProxy_must_start_daemon_process_on_proxy

[Aaronontheweb]

Description

The test ShardedDaemonProcessProxySpecs.ShardedDaemonProcessProxy_must_start_daemon_process_on_proxy is intermittently failing in CI due to a timing/race condition.

Failure Details

Test: Akka.Cluster.Hosting.Tests.ShardedDaemonProcessProxySpecs.ShardedDaemonProcessProxy_must_start_daemon_process_on_proxy

Location: src/Akka.Cluster.Hosting.Tests/ShardedDaemonProcessProxySpecs.cs:95

Error:

Assert.Equal() Failure: Values differ
Expected: 2
Actual:   1

Root Cause: The test expects 2 daemon processes to start but only 1 was started within the timeout period.

The logs show:

[WARNING][10/01/2025 18:44:58.304Z][Thread 0016][ActorSystem(ShardedDaemonProcessProxySpecs)] AwaitAssert failed, timeout [00:00:03] is over after [6] attempts and [00:00:03.2718737] elapsed time

Evidence

• This failure occurred in PR Adding Akka.Persistence health checks #662 (https://github.com/akkadotnet/Akka.Hosting/actions/runs/18172208991/job/51729401764)
• The test is using AwaitAssert with a 3-second timeout
• All other tests passed, indicating this is an isolated flaky test
• Classic symptoms of a race condition in distributed systems testing

Suggested Fixes

1. Increase timeout: The 3-second timeout may be insufficient in slower CI environments
2. Add better diagnostics: Log the actual state when the assertion fails to understand what's happening
3. Review cluster formation timing: Ensure both nodes in the cluster are fully joined and ready before expecting daemon processes to start
4. Consider using TestKit probes: Use TestProbe to explicitly wait for specific cluster events before asserting on daemon process count

Impact

• Causes spurious CI failures unrelated to actual code changes
• Currently blocking PR Adding Akka.Persistence health checks #662 which contains persistence health check improvements

Related Code

The test is setting up a cluster with worker and proxy nodes, then expecting daemon processes to start on both. The timing between cluster formation and daemon process initialization appears to be the issue.