Skip to content

feat(server): FIFO channels data plane (PUSH/POP/GET_CHAN_LEN) (PR 2) - #298

Merged
ortuman merged 6 commits into
mainfrom
feat/fifo-data-plane
May 13, 2026
Merged

feat(server): FIFO channels data plane (PUSH/POP/GET_CHAN_LEN) (PR 2)#298
ortuman merged 6 commits into
mainfrom
feat/fifo-data-plane

Conversation

@ortuman

@ortuman ortuman commented May 12, 2026

Copy link
Copy Markdown
Owner

Summary

Builds the FIFO data plane on top of PR 1's transition slice. Owners now PUSH elements to a FIFO channel; JOINed read-allowed members atomically POP them; owners query the queue depth with GET_CHAN_LEN.

The load-bearing contract: each POP advances and fsyncs the head cursor sidecar before writing POP_ACK. A crash between cursor fsync and the socket write loses that element consumer-side; no element is ever returned by two POP_ACKs. PUSH durability matches BROADCAST (synchronous flush when message_flush_interval == 0, batched otherwise). The cursor advance plus a tail-segment retention invariant in the message log keeps the seq counter recoverable across restart.

CLEAR (owner-driven drain / recovery) ships in a follow-up PR.

What changed

  • Protocol (crates/protocol): 6 new variants — PUSH/PUSH_ACK, POP/POP_ACK, GET_CHAN_LEN/CHAN_LEN. PUSH and POP_ACK are payload-bearing (BROADCAST/MESSAGE framing convention). 7 round-trip tests.
  • Channel manager (crates/server): Command::{Push,Pop,GetChannelLen} and matching handlers with the full error matrix (CHANNEL_NOT_FOUND, WRONG_TYPE, USER_NOT_IN_CHANNEL, FORBIDDEN, NOT_ALLOWED, QUEUE_EMPTY, QUEUE_FULL, POLICY_VIOLATION, CURSOR_RECOVERY_REQUIRED, NOT_IMPLEMENTED). QUEUE_FULL is gated by the logical cursor-to-tail depth, not the physical entry count. Adds PopVisitor next to the existing HistoryVisitor to capture a single entry into a pooled buffer.
  • C2S dispatch (crates/server/src/c2s/conn.rs): wire the three new messages.
  • Client (crates/client): push(channel, payload), pop(channel) -> (payload, timestamp), get_chan_len(channel) -> u32 on both compio and tokio C2S clients.
  • Metrics (under the narwhal prefix): fifo_pushes{result}, fifo_pops{result} with one label per outcome, and fifo_cursor_fsync_seconds histogram. Skipped fifo_queue_depth (use GET_CHAN_LEN instead).
  • Message::PopAck added to the send_message_with_payload payload-bearing variant assertion in crates/common.
  • Docs: docs/PROTOCOL.md gains the six wire messages; docs/architecture/channels/fifo-channels.md status flipped to "Partially Implemented (CLEAR pending PR 3)". CHANGELOG entry.

Tests added

crates/server/tests/c2s_fifo_channel.rs grows by 12 tests:

  • PUSH/POP roundtrip — payload and PUSH timestamp survive intact
  • QUEUE_EMPTY on POP against an empty FIFO
  • QUEUE_FULL via logical depth — fill cap, POP one, PUSH succeeds, PUSH again fails
  • PUSH from non-owner returns FORBIDDEN
  • POP from non-member returns USER_NOT_IN_CHANNEL
  • POP from member excluded by read ACL returns NOT_ALLOWED
  • GET_CHAN_LEN reflects PUSH/POP activity
  • GET_CHAN_LEN from non-owner returns FORBIDDEN
  • Competing consumers — two clients POPing the same FIFO each get a distinct element
  • Cursor durable across restart — PUSH 3, POP 1, restart, POP remaining: consumed entry must not resurface
  • POP force-flushes the log when the entry is not yet durable (60s message_flush_interval, POP still completes immediately)
  • PUSH durable with message_flush_interval=0 — PUSH, shutdown, restart, POP returns the pre-restart payload

Test plan

  • cargo fmt --all clean
  • cargo clippy --workspace --all-targets -- -D warnings clean
  • cargo test --workspace passes (all 200+ tests, including the 12 new FIFO data-plane tests plus the 11 PR 1 FIFO tests still green)
  • Skim the PR for anything that should bypass the assertion change in crates/common/src/conn/mod.rs

PUSH appends owner-authored elements to a FIFO channel's log; the logical
queue depth (cursor-to-tail span, not physical entry count) is gated by
`max_persist_messages`, returning `QUEUE_FULL` on overflow. PUSH durability
mirrors `BROADCAST`: synchronous flush+fsync when
`message_flush_interval == 0`, batched otherwise.

POP atomically returns the head element. The handler reads the entry at
`cursor.next_seq`, forces a synchronous log flush when the entry is not yet
durable (`next_seq > last_durable_seq`), advances the cursor sidecar with
an atomic tmp+fsync+rename, and only then writes `POP_ACK`. A crash between
cursor fsync and the socket write loses the element consumer-side; no
element is ever returned twice. `POP_ACK` carries the entry's PUSH
timestamp so consumers can reason about queue-induced latency.

GET_CHAN_LEN reports the underflow-guarded logical depth `log.last_seq -
cursor.next_seq + 1`. Owner-only.

Adds:
- `PUSH`/`PUSH_ACK`/`POP`/`POP_ACK`/`GET_CHAN_LEN`/`CHAN_LEN` protocol
  variants with round-trip tests.
- Mailbox commands and handlers on `ChannelManager` with full validation
  matrix (`CHANNEL_NOT_FOUND`, `WRONG_TYPE`, `USER_NOT_IN_CHANNEL`,
  `FORBIDDEN`, `NOT_ALLOWED`, `QUEUE_EMPTY`, `QUEUE_FULL`,
  `POLICY_VIOLATION`, `CURSOR_RECOVERY_REQUIRED`, `NOT_IMPLEMENTED`).
- C2S dispatch wiring plus `push()`, `pop()`, `get_chan_len()` methods on
  both compio and tokio clients.
- Metrics under the `narwhal` prefix: `fifo_pushes{result}`,
  `fifo_pops{result}` (one label per outcome), `fifo_cursor_fsync_seconds`
  histogram.
- 12 integration tests: PUSH/POP roundtrip, QUEUE_EMPTY/QUEUE_FULL via
  logical depth, owner-only PUSH, JOIN-required POP, read-ACL denied POP,
  GET_CHAN_LEN happy path + non-owner-FORBIDDEN, competing consumers
  (each gets unique element), cursor-durable-across-restart (no element
  resurfaces), POP force-flushes log when not durable, PUSH durable with
  zero flush interval.

Allow `Message::PopAck` through `ConnTx::send_message_with_payload`'s
payload-bearing variant assertion. Update `docs/PROTOCOL.md` with the six
new messages and flip `docs/architecture/channels/fifo-channels.md` to
"Partially Implemented (CLEAR pending PR 3)".

Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
@ortuman ortuman changed the title feat(server): FIFO channels data plane (PUSH/POP/GET_CHAN_LEN) (PR 2) feat(server): FIFO channels data plane (PUSH/POP/GET_CHAN_LEN) (PR 2) May 12, 2026
ortuman added 5 commits May 13, 2026 12:59
1. `Inner::roll_segment` now `sync_all()`s the active log before dropping
   the handle. Without this, any appends since the last `flush()` (including
   the entry that triggered the roll) sit in the kernel page cache while the
   segment is sealed; the subsequent `flush()` only fsyncs the new active
   segment, so `cached_durable_seq` advances past entries that are not
   actually on disk. Breaks the durability claim made by `PUSH_ACK` and
   `BROADCAST_ACK` (with `message_flush_interval == 0`) and risks the
   `cursor.next_seq <= log.last_seq() + 1` invariant FIFO recovery relies on.

2. `push_payload` no longer increments `channel.seq` before `message_log.append`
   completes. The previous order (peek + bump via `next_seq()`, then append)
   burned the seq on a transient append failure, leaving a gap between the
   in-memory counter and the log's tail. `MessageLog::read(from_seq, ...)`
   returns the first entry with `seq >= from_seq`, NOT strictly at `from_seq`,
   so the next POP at the burned seq would read the later entry, advance the
   cursor by one, and re-read the same entry on the next POP: silent duplicate
   delivery. The fix peeks `channel.seq` without bumping and commits the bump
   only after `append` returns Ok.

3. `pop_payload` adds a defensive seq match: `PopVisitor` now records the
   entry's seq, and the handler verifies `captured.seq == cursor_next_seq`
   before advancing. On mismatch (gap below the tail), log diagnostics and
   return `INTERNAL_SERVER_ERROR` instead of advancing. (2) prevents the
   condition from arising in practice; (3) catches future log-corruption
   regressions of the same shape and keeps the no-duplication contract
   load-bearing.

Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
The previous wording elided `MessageLog::read`'s parameters with an
ellipsis, which the repo's comment style rule disallows on newly added
or modified comments. Spell out the parameter names instead
(`from_seq, limit, visitor`).

Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
Synthesize the data-plane and transition-slice entries into shorter
summaries, and reclassify both as ENHANCEMENT (they extend the existing
channel system rather than introducing a wholly new top-level feature).

Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
Collapse the two unreleased FIFO entries into a single ENHANCEMENT line
covering the transition slice and the data plane together, and link the
two PRs (#292, #298) that landed them.

Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
Match the brevity of surrounding ENHANCEMENT entries.

Signed-off-by: Miguel Ángel Ortuño <ortuman@gmail.com>
@ortuman
ortuman merged commit 25fe7e4 into main May 13, 2026
2 checks passed
@ortuman
ortuman deleted the feat/fifo-data-plane branch May 13, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant