Skip to content

queue: make the queue backend pluggable - #177

Open
marcopiraccini wants to merge 3 commits into
mainfrom
queue-pluggable-driver
Open

queue: make the queue backend pluggable#177
marcopiraccini wants to merge 3 commits into
mainfrom
queue-pluggable-driver

Conversation

@marcopiraccini

@marcopiraccini marcopiraccini commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Split queue processing into durable QueueStore, BrokerTransport, and Coordinator interfaces.
  • Keep PostgreSQL as the default store and transport, preserving the current behavior.
  • Add an outbox handoff with transport-aware publication tracking so pending messages survive crashes and queue-driver changes.
  • Move delivery state transitions behind the store, including acknowledgement, retry, dead-letter, claim release, and expired-delivery recovery.
  • Add backend conformance, redelivery, leadership, outbox, and driver-change coverage.

Dependency

This PR is stacked on #176, which has not been merged yet. Review and merge #176 first.

This introduces the extension points and retains PostgreSQL behavior. External Redis or SQS transports are intentionally left for the next phase.

Centralize the queue-name grammar in queue/names.ts, mirroring
@workflow/world's queue.ts: the spec allows an optional namespace between the
sentinel and the kind (__{namespace}_wkf_workflow_<id>, from
WORKFLOW_QUEUE_NAMESPACE), which three call sites did not account for.

routeMessage already matched the namespaced form with its own inline regex, but
the failure-finalization paths in the poller matched the bare
'__wkf_workflow_' / '__wkf_step_' literals, so a namespaced delivery skipped
failRun and its workflow name was sliced at the wrong offset. The replay
fallback in run-actions had the same literal. All of them now share one
definition, and the router's duplicate regex is gone with it.

Also wire the VersionExpired error, which was defined but never thrown: an
enqueue targeting an expired deployment version now answers 410 instead of
accepting a message routeMessage will refuse to dispatch, and the World client
implements the spec's optional isDeploymentUnavailableError() so the SDK can
re-route rather than burn ten delivery attempts. Draining versions still accept
work, matching routeMessage.

Signed-off-by: marcopiraccini <marco.piraccini@gmail.com>
Signed-off-by: marcopiraccini <marco.piraccini@gmail.com>
Signed-off-by: marcopiraccini <marco.piraccini@gmail.com>
@marcopiraccini marcopiraccini changed the title Queue pluggable driver queue: introduce pluggable queue backends Aug 28, 2026
@marcopiraccini marcopiraccini changed the title queue: introduce pluggable queue backends queue: make the queue backend pluggable Aug 28, 2026
@marcopiraccini
marcopiraccini marked this pull request as ready for review August 28, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants