terminal-rl: tb-v2 task builders, train-batch dump tooling, cookbook pin#732
Draft
signalrush wants to merge 71 commits into
Draft
terminal-rl: tb-v2 task builders, train-batch dump tooling, cookbook pin#732signalrush wants to merge 71 commits into
signalrush wants to merge 71 commits into
Conversation
… with harbor-parity summarization Same terminal interaction model as terminus2 (tmux keystroke batches + screen observations), but the conversation rides native tool-calling rails: execute_commands tool calls, role:tool results, and reasoning_content echoed verbatim on assistant messages so think-mode models (DeepSeek-V4-Pro) retain reasoning across turns. Measured on Fireworks: the user-message path renders echoed reasoning to zero prompt tokens; the tool-calling path renders it fully. Context management is ported verbatim from harbor terminus_2.py so eval deltas stay attributable to the transport alone: - three-subagent summarization dance (summary -> fresh-context questions -> answers), prompt texts byte-identical to harbor's - reactive fallback chain on context-length errors: unwind whole turns from the tail -> full dance -> short summary -> no-LLM handoff -> "Technical difficulties" fabricated reply (episode never crashes) - proactive summarization when estimated free tokens < 8000; tokens estimated from real usage anchors + chars/4 instead of litellm's model map (whose unmapped-model 1M default silently disabled summarization in raw harbor) - context-length errors fast-fail out of the retry loop (a doomed payload retried 4x is just 4 identical failures) Knobs: enable_summarize, proactive_summarization_threshold, context_limit_tokens (env RLLM_TERMINUS_CONTEXT_LIMIT wins so eval runs can pin the model's real limit, e.g. 240000 for v4 on Fireworks). Validated end-to-end on TB2 (v4-pro, effort=max): the dance fired in production and the 207-step episode recovered to a PASS; 100% reasoning persistence across all episodes; blended pass@1 ~66% / pass@3 ~76% vs 54.3 for the classic user-message harness (official in-house: 67.9). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds tb-v2-debug / tb-v2-opus-pass-at-1 / tb-v2-full (and friends) to the dataset registry, backed by new hf_tasks_builder/local_tasks_builder that materialize task archives from HF (mobius-lab/tb-v2-tasks) or local dirs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gateway binds 0.0.0.0:<port> from a background thread/subprocess, so a bind conflict surfaced as an opaque 30s health-poll timeout — after minutes of Fireworks infra provisioning. Probe-bind the port at trainer init (before the backend provisions anything) and again in start(), and raise GatewayPortInUseError naming the holder-lookup command and, when the port is the tunnel daemon's upstream, the likely concurrent-run cause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Fireworks SDK's SSE stream can surface raw ssl.SSLError (e.g. bad_record_mac) unwrapped by httpx; the transient classifier missed it, failing the sampling call permanently on attempt 1 — 26% of episodes died to this in a single run. Classify ssl.SSLError as a network error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unpinned git ref resolved to cookbook HEAD, which now requires fireworks-ai>=1.2.0a86 and conflicts with our ==1.2.0a83 pin, making rllm[fireworks] unresolvable on fresh installs. Pin the commit recorded in uv.lock (the combo running in production). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings in PR rllm-org#727 (gateway separate-process mode, per-request CPU cuts, loop-health monitoring) and 751d0bd (continuous rollout dispatch). Conflict resolutions: - fireworks_engine.py: keep both 'import ssl' (our SSL retry fix) and upstream's 'import time' - gateway/manager.py: take upstream's _start_gateway_subprocess/_start_process rewrite, re-insert our preflight_gateway_port() call at the top of _start_process Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A non-zero verifier exit is the expected outcome for an unsolved task; the raised exception already carries the detail and callers pick the severity. Matches the docker backend, which already logged at debug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…1, 4 replicas) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ay memory bounding); keep all terminus2-native content The rllm-org#725 squash-merge and its upstream revert cancel out, so this merge preserves the full terminus2-native harness work while picking up the session-sharding front and the async-training gateway memory fix. Conflict resolution in gateway/manager.py: dropped the num_workers>1 NotImplementedError guard (rllm-org#728 implements it) and kept the port preflight at the top of _start_gateway_subprocess. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
signalrush
marked this pull request as draft
July 7, 2026 03:45
…29s; log response headers - The 90s retry budget's clock started before attempt 1, charging a long generation's own streaming time against it — any attempt failing after ~80s (exactly the mid-stream SSE truncations) got zero retries. Start the clock at the first failure instead. - 429 'service overloaded' was classified permanent (marker list had 502/503/425 but not 429); the SDK already burns its 300s internal window before we ever see it, so failing fast helped nothing. - Log the failed response's headers (Retry-After, rate-limit info is otherwise discarded with the exception). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e poisons rewards _push_b64 routed the payload-consuming execs through _exec_unchecked, so a failed task-files upload silently no-opped: the agent ran in a broken environment and its reward-0 episode entered training as a legitimate failure (ENV_DONE, no infra tag). Raising instead lets the setup hook classify SANDBOX_ERROR (excluded + retried) and revives the evaluator's dead AddTestsDirError branch. mkdir/chunk-append execs stay unchecked — a corrupt payload fails the final checked consume anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…arbor summarization Harbor's Terminus2 has enable_summarize but the driver never passed it, so summarization was permanently on. Plumb it through host-env-wins (same pattern as RLLM_TERMINUS_CONTEXT_LIMIT): =0 gives a plain ReAct loop where a context overflow ends the episode (still evaluated) instead of entering the lossy recover-grow-overflow cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ness 3.0, 100 turns, 2 trainer / 6 rollout replicas, 4 gateway workers, summarize off Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gauges - rllm.trainer.dump_batch_dir: dump each optimizer step's consumed batch in eval's episode-store layout (timestamped run root under ~/.rllm for relative paths, step_<N>/episodes/*.json + trajectory_groups.json), browsable with rllm view. Default null = off. - trainer loop-health line now shows the dispatch gate: groups in-flight vs staleness cap, buffer queue/partial/filtered/consumed, and a dispatch_blocked reason — the state that explains a decaying engine inflight while dispatch is frozen at the group cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hes, 4 rollout replicas mask_max_prompt_length_exceeded/mask_max_turns_exceeded=false so context-overflow and turn-cap episodes train as reward-0 group members instead of being silently dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nable_summarize knob, sandbox lifetime sizing Upstream landed its own RLLM_TERMINUS_ENABLE_SUMMARIZE plumbing (b342c9c) — take the upstream (canonical) implementation in terminus2.py/train.py and adapt qwen3p5-debug.sh to the cookbook's TERMINUS_ENABLE_SUMMARIZE knob. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> # Conflicts: # cookbooks/terminal-rl/train_fireworks.sh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… in both harnesses rllm eval has no agent-kwargs mechanism, so the field-only knob left eval runs unable to disable summarization. Host env wins when set (mirrors RLLM_TERMINUS_CONTEXT_LIMIT); the native harness also stops clobbering the parent's env-aware value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Distill deepseek-v4-pro terminus2 traces into deepseek-v4-flash via Fireworks LoRA SFT, preserving per-turn reasoning as trained <think> blocks: - build_distill_dataset.py: per-step curation folding reasoning_content into <think>...</think> content (stock from-eval drops it); registers a dataset. - sft_deepseek_flash.py: launches Fireworks SFT with the deepseek-correct config the CLI can't set (renderer=deepseekv3_thinking, v4 tokenizer with correct <think> id 128821, shape=deepseek-v4-flash-256k-lora, stepwise/ LAST_ASSISTANT_MESSAGE) via SFTSpec.overrides. - sft_deepseek_flash.sh: one-command pipeline wrapper. Format validated end-to-end: think block trained, history masked, correct EOS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ilent zero-weight training datum truncation is right-side, so a per-step row longer than max_length loses its target (weights all zero) and trains on nothing. terminus2 rows are long (p50~27k, p90~68k cumulative tokens), so ~37% would silently no-op at 32k. Filter rows over max_length at build time (using the real renderer+tokenizer), keep builder/launcher max_length in sync (default 131072, ~99.8% coverage). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # rllm/harnesses/terminus2.py
…terminus2-native Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nto terminus2-native Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All of a run's artifact trees (episode logs, batch dumps) now share one UnifiedTrainer._run_stamp instead of computing separate timestamps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
experiment+step alone is account-global and collides on every relaunch with the same experiment_name (ALREADY_EXISTS at each save; observed at steps 10-60 of the tb-v2-debug LoRA run). The snapshot name (step-N-<job suffix>) is already unique per run — embed it: <experiment>-step-N-<suffix>. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…port baking qwen3p5-debug-fullft.sh: lora_rank=0 on the POLICY_TRAINER shape, lr 1e-6, port 9101, thw2 tunnel — runs concurrently with the LoRA script (9091/thw1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…licy metrics) into terminus2-native Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records MoE expert routing at rollout and replays it in the trainer's forwards, removing routing nondeterminism from the trainer/sampler mismatch. Full client chain verified (SDK _tinker_r3_patch wire support, gateway cumulative-token carry, transform threading); server-side replay is covered by Fireworks' own router_replay e2e tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Spreads per-request gateway CPU across 8 worker processes (ports port+1..port+8). Port map stays collision-free: LoRA 9091-9099, full-FT 9101-9109. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n experiment names Script name, experiment_name, and dump_batch_dir now all carry the same identity (lora-r3 / fullft-r3), so W&B runs, batch dumps, and promoted models are attributable to the exact launcher variant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IS-weighted REINFORCE with a soft backward-KL trust region to the sampler (bwd_kl_coef=0.0025) instead of PPO clipping; bypass_mode=true since the loss ignores logp_old (no proximal forward needed — custom-loss path logs offpolicy metrics from rollout logprobs regardless). Verified via hydra compose: loss_params requires the + append prefix (empty struct dict). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(3.5x smaller) The batch dump wrote every Step twice: episodes/ (what rllm view reads) plus trajectory_groups.json (~1GB/step) which nothing in the codebase consumes — episodes already carry the training view (token ids, logprobs, advantages, weight_version). Dropped. EvalEpisodeStore.write now uses compact separators instead of indent=2: token-array-heavy episodes measured 12.9MB -> 3.7MB (3.48x) on a real dump; consumers (rllm view, json.load) are whitespace-agnostic. Verified round-trip through the store and the viewer's loader. Net: ~8GB/step -> ~2GB/step, and proportionally less synchronous serialization time on the trainer event loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…top dumping by default _dump_train_batch snapshots episode refs synchronously (reset_batch reassigns the fields right after) and offloads the multi-GB JSON serialization via asyncio.to_thread — measured 0.1ms on the loop vs ~2min blocking before. At most one dump in flight; a still-writing previous dump skips the next (dumps are diagnostics, never training data). Debug scripts no longer set dump_batch_dir (config default is null = off); opt back in per-launch by appending rllm.trainer.dump_batch_dir=<dir>. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…llm-org#760) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remaining content after splitting out the bugfixes (fireworks retry hardening → #748, sandbox fixes → #749, both merged):
local_tasks_builder.py(register a local directory of Harbor task dirs as a dataset — symlinks +dataset.toml, tasks never copied),hf_tasks_builder.py(same for HF-hosted task datasets), tb-v2 dataset catalog entries, and aTB_TRAIN_DATASETenv override incookbooks/terminal-rl/train.py.trainer.dump_batch_dir— dump each optimizer step's consumed batch in eval's episode-store layout (step_<N>/episodes/…), browsable withrllm viewexactly like eval runs.fireworks-aito1.2.0a87and track cookbook HEAD again (HEAD requires>=1.2.0a86; the old==1.2.0a83pin made the[fireworks]extra unresolvable and brokeuv run).qwen3p5-debug.sh,train_fireworks.shtweaks).These can merge together here, or be split further (pin / builders / dump) on request — each group is file-disjoint.
Test plan
Branch is fully synced with
terminal-rl(0 behind).pytest tests/harnesses tests/unified_trainer/test_custom_loss.py tests/sandboxgreen on the merged tree.🤖 Generated with Claude Code