feat(mcp): headless sessions with playhead simulation + lag measurement#257
Draft
leszko wants to merge 1 commit into
Draft
feat(mcp): headless sessions with playhead simulation + lag measurement#257leszko wants to merge 1 commit into
leszko wants to merge 1 commit into
Conversation
The MCP server could only attach to a browser-created session, so realtime bugs (e.g. generation lagging behind the playhead) were only reproducible with the frontend. Add a headless mode: headless_start spawns a full PRIMARY WebSocket client (new headless_client.py, torch-free) that does what the browser does — init handshake, binary slice decode (zstd float16 deltas, kept byte-identical to the server SliceCodec mirror), a simulated audible playhead reported over the params channel, and the params_echo mirror so control-bus knob changes persist. The session registers like any other, so every existing MCP tool drives it unchanged. On top of the transport it measures the two lag signals per event: lead_s (where each fresh slice landed relative to the playhead; negative = behind the listener) and staleness_s (age of the audio under the playhead; ~buffer duration = a full lap behind). New tools: headless_start/stop/status, headless_lag_report, headless_observe, headless_seek, headless_set_playback (clock-skew + pause repro), and headless_dump_audio (the stream the simulated listener heard, stale audio audible as a user would hear it). Verified end-to-end against a live TRT backend: healthy stream shows lead ~0.2 s and zero stale ticks; a seek produces the expected transient staleness spike; knobs set through the control bus persist via the echo mirror. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Testing realtime behavior through the frontend is slow, and some bugs — e.g. music generation lagging behind the playhead — were only reproducible with a browser attached. The MCP server could only attach to a browser-created session because only the PRIMARY WS transport advances the server playhead (
StreamingSession.set_knobsis echo-only forCommandOrigin.EXTERNAL) and knob persistence relies on the browser'suseMcpMirrorcarrying echoes back.What
demos/realtime_motion_graph_web/headless_client.py(new, torch-free — numpy/zstandard/websockets only, runs inside the MCP stdio process):SliceCodecmirror,swap_ready/stem_assetshandling.PlayheadSim: simulated audible clock reported viaparamsticks like the browser; supports seek, pause, and rate skew (reproduces a fast client AudioContext eating the lead).params_echomirror: control-bus knob changes are merged and carried back on the next PRIMARY tick, so all existing MCP tools work against headless sessions unchanged.LagTracker: per-slicelead_s(where fresh audio landed relative to the playhead; negative = behind the listener) and per-tickstaleness_s(age of audio under the playhead; ~buffer duration = a full lap behind), plus a ring recorder of what the simulated listener heard.New MCP tools (
mcp_server.py):headless_start,headless_stop,headless_status,headless_lag_report,headless_observe,headless_seek,headless_set_playback,headless_dump_audio.Typical lag-bug repro:
Note: starting a headless session preempts a live browser session (one-session-per-pod policy), documented in the tool.
Testing
tests/unit/test_headless_client.py): playhead sim (advance/wrap/seek/pause/rate-rebase/swap-reset), circular lead folding, lag detection (healthy stream, stall + behind-playhead writes, window filtering, timeline rollup), and slice codec round-trips against the server-sideSliceCodec(anti-ghosting byte-identity invariant).tests/unit/test_ws_preemption.pyhas 2 failures that pre-exist onmain(SDK package move fallout), unrelated to this change.No wire-contract or knob-registry changes — no TS regen needed.
🤖 Generated with Claude Code