ProtoActor: distributed cache does not subscribe one local invalidator per cluster member

[DenDeline]

Summary

Elsa.Caching.Distributed.ProtoActor is intended to invalidate the process-local memory cache on every active cluster member.

The affected implementation creates a process-static random ClusterIdentity and subscribes that virtual identity to the change-token-signals topic:

• Affected StartLocalCacheActor subscription
• Local-cache virtual-actor registration
• Elsa LocalNodeStrategy

A ClusterIdentity identifies a virtual actor by identity and kind. It does not retain the address of the member that registered the subscription.

Proto.Actor resolves cluster-identity subscribers later, during publication:

• Resolve and group subscribers during publication

Resolution and placement are therefore independent of the member that originally registered the identity. Multiple subscribed identities can activate on the same member while another member has no local invalidator.

The implementation does not guarantee one cache invalidator per member.

There are two additional lifecycle problems:

1. The subscription runs inside BackgroundService.ExecuteAsync. BackgroundService.StartAsync does not wait for an incomplete ExecuteAsync task, so host startup is not gated on the Pub/Sub subscription acknowledgement. In .NET 10, the entire ExecuteAsync method additionally runs on a background thread:

   • .NET 10 BackgroundService

2. Proto.Actor's automatic failed-delivery and departed-member cleanup handles concrete PID subscribers:

   • Failed-delivery cleanup
   • Topology and startup cleanup

Elsa's virtual actor performs a best-effort unsubscribe from OnStopped, but this only runs when that activation stops:

• Affected LocalCache.OnStopped

Departure of the member that originally registered a cluster identity does not necessarily stop an activation hosted elsewhere or remove its subscription.

Steps to reproduce

A deterministic regression test can use two Elsa hosts and two Proto.Actor members in the same process:

1. Configure both hosts with the same cluster name and shared multi-member test provider.
2. Enable memory caching and the Proto.Actor distributed-cache transport.
3. Register a recording IChangeTokenSignalInvoker independently in each host.
4. Configure a shared capturing IKeyValueStore<Subscribers> for the topic actor.
5. Start both hosts.
6. Inspect the subscriber state for the change-token-signals topic.
7. Publish one batch containing:
   • a signal with a unique target key;
   • a second signal with a unique fence key.
8. Wait until both members observe the fence key.
9. Inspect the number of target-key invocations recorded by each member.

The same-process topology is deliberate. StartLocalCacheActor.ActorName is process-static, so both hosted services create the same ClusterIdentity. Proto.Actor stores subscribers in a set, producing one logical subscriber rather than one subscriber per member.

In separate processes, each process normally creates a different random identity, but those identities are still not tied to their registering members and can be placed unevenly.

The target and fence should be published together through Cluster.Publisher().PublishBatch. A successful PublishResponse confirms that the topic actor accepted the publication; it does not prove that every subscriber has processed it. Waiting for the fence provides bounded delivery coordination before target counts are asserted.

Expected behavior

• Each active cluster member owns one member-local cache invalidator.
• The Pub/Sub topic contains one concrete PID for each active member.
• Each PID address matches the address of its owning member.
• Host startup does not complete until the local PID subscription is acknowledged.
• In a stable two-member scenario, one published invalidation is processed once by each member.
• Gracefully stopping a member unsubscribes its PID and stops its local actor.
• Concrete PID subscriptions are eligible for Proto.Actor's failed-delivery and departed-member cleanup.

Actual behavior

• The topic contains cluster identities instead of member-local PIDs.
• In a two-member same-process test, the process-static identity collapses to one logical subscriber.
• In separate processes, the random identities are not tied to their registering members.
• Multiple identities can activate on one member while another member receives no invalidation.
• Host startup can complete before the subscription is acknowledged.
• Proto.Actor cannot associate a departed member with the cluster identity that member originally registered.

Proposed resolution

Replace the placement-managed LocalCache virtual grain with an internal member-owned MemoryCacheInvalidatorActor:

var props = cluster.System.DI().PropsFor<MemoryCacheInvalidatorActor>();
var pid = cluster.System.Root.SpawnNamedSystem(
    props,
    "$memory-cache-invalidator");

await cluster.Subscribe(
    Topics.ChangeTokenSignals,
    pid,
    cancellationToken);

The hosted-service lifecycle should:

1. Register the invalidator as an ordinary internal IActor.
2. Spawn one named system actor locally in each ActorSystem.
3. Subscribe the actor's concrete PID.
4. Await the subscription acknowledgement from IHostedService.StartAsync.
5. Stop the spawned actor and propagate the exception if subscription fails.
6. During StopAsync, unsubscribe the PID and always stop the actor, including when unsubscribe fails or is cancelled.
7. Stop the invalidator before the Proto.Actor cluster shuts down.
8. Remove the obsolete LocalCache grain contract, cluster kind, helper, and grain-codegen dependency.
9. Retain LocalCache.Messages.proto and LocalCacheMessagesReflection.Descriptor for remote Pub/Sub serialization.

The fixed actor name is safe across members because a PID includes both the actor-system address and the local actor ID.

Proto.Actor exposes dedicated PID subscription APIs for this lifecycle:

• PID subscription
• PID unsubscription

The invalidator does not need a cluster identity because it is member-owned infrastructure, not a globally addressable virtual entity.

Compatibility and breaking changes

This resolution intentionally introduces breaking API changes:

• LocalCache.proto and its generated public APIs are removed:
  • GrainExtensions.GetLocalCache
  • LocalCacheBase
  • LocalCacheClient
  • generated LocalCacheActor
• Public StartLocalCacheActor changes from BackgroundService to IHostedService.
• The legacy LocalCache cluster kind is no longer registered.
• The module no longer requires Proto.Cluster.CodeGen.

No compatibility shim is provided. Consumers that reference the removed APIs must update and recompile.

The API removal and hosted-service base-type change must be documented and versioned as breaking changes.

LocalCache.Messages.proto and LocalCacheMessagesReflection.Descriptor remain because invalidation messages still require remote serialization.

Upgrade limitation

This implementation does not migrate existing local-cache ClusterIdentity subscriptions.

A mixed-version rolling upgrade is unsupported. Every member running the old implementation must stop before members running the new implementation start.

A full cluster stop removes in-memory topic state. If a durable Pub/Sub subscriber store contains state created by the old implementation, the subscriber state for the change-token-signals topic must also be cleared before starting the new version.

Otherwise obsolete cluster identities can remain after their cluster kind has been removed and can interfere with publication.

Impact

In a multi-member deployment where every member owns a process-local memory cache, a missed invalidation can leave one member serving stale state until that cache entry expires.

Publication success and ordinary actor-system health checks do not establish that every member has an active cache-invalidating subscription.

Dependencies and related work

• Depends on #171, which is stacked on #168.
• #168 fixes #167 by starting the Proto.Actor cluster before dependent hosted services. Reverse hosted-service ordering also allows the invalidator to unsubscribe and stop before cluster shutdown.
• #171 fixes #170 by registering the cache-invalidation protobuf descriptor on the RemoteConfig attached to the ActorSystem.
• #172, which fixes #169, is related but is not a prerequisite. If durable subscriber storage already contains state from the legacy implementation, the upgrade limitation above applies.

Acceptance criteria

• Every member spawns one local system actor named $memory-cache-invalidator.
• The topic contains exactly one concrete PID for every active member.
• No new local-cache ClusterIdentity subscription is created.
• Each subscriber PID address matches the address of its owning member.
• Host.StartAsync does not complete before the topic actor acknowledges the local subscription.
• In a stable two-member test, one target invalidation is processed once by each member.
• Delivery assertions use bounded target-and-fence coordination rather than treating PublishResponse as subscriber-processing completion.
• Subscription failure stops the newly spawned actor.
• Graceful shutdown unsubscribes the member PID before cluster shutdown and stops the local actor.
• An unsubscribe failure does not prevent the local actor from being stopped.
• After one member stops, the topic contains only the remaining member's PID.
• An invalidation published after that shutdown reaches the remaining member and not the stopped member.
• Concrete PID subscribers remain eligible for Proto.Actor's departed-member cleanup.
• LocalCacheVirtualActorProvider registers no legacy LocalCache cluster kind.
• LocalCacheVirtualActorProvider continues to register LocalCacheMessagesReflection.Descriptor.
• The legacy grain contract, generated grain APIs, and grain-codegen dependency are removed as an explicit breaking change.

Verification

The accompanying integration tests cover:

• startup waiting for subscription persistence and acknowledgement;
• two members registering one named local PID each;
• subscriber PID addresses matching the member addresses;
• a target-and-fence batch being processed once by each member;
• graceful shutdown removing one member's PID and stopping its actor;
• publications after shutdown reaching only the remaining member;
• removal of the legacy cluster kind;
• retention of the cache-invalidation message descriptor.

Logs and screenshots

Not applicable. This is a distributed placement and lifecycle defect. It does not require an exception, warning, or failed publication response to occur.

Environment

• Affected Elsa Extensions commit: 4b672e3b78a7f063f87e7def77fa1947126c36bc
• Elsa package version: 3.7.0
• Proto.Actor version: 1.8.0
• .NET SDK: 10.0.302
• Operating system: macOS 26.5.2
• Architecture: ARM64
• Affected topology: two or more cluster members with process-local memory caches