Independent reviewer, multi-model debate, and best-of-N tournament

[Jacky628]

Problem

Reviewing and judging are currently performed by the same model that produced the artifact. Stage 18 asks the author model to review the paper it just wrote, and the Stage 20 quality gate asks it whether its own paper is good enough to pass. LLMs show a well-documented preference for their own output, so this makes both stages weaker than they look: a self-review rarely produces the kind of objection that would actually block a run.

What this adds

Three opt-in mechanisms that decouple generation from judgement. All three default to off — with no config changes, every stage behaves exactly as it does today.

1. Independent reviewer

llm.reviewer_model builds a reviewer client that never falls back to the author model. Set only the model to reuse the main provider with a different model, or add reviewer_provider / reviewer_base_url / reviewer_api_key(_env) for a fully separate provider (e.g. generator = GPT, reviewer = Claude).

Used by Stage 18 (peer review) and Stage 20 (quality gate); both fall back to the generator when it is unset.

Stage 18 also writes review_provenance.json — author_model, judge_model, independent_reviewer — plus an HTML comment header in reviews.md, so a finished run can be audited for reviewer independence after the fact rather than taken on trust.

2. Multi-model debate (pipeline/debate.py)

Roles argue with distinct models, rebut each other for debate_rounds, then an independent judge ranks the positions and a synthesizer writes the final text from that ranking.

Separating ranking from synthesis is the part that matters. When a single model both scores the positions and writes the summary, the output collapses into vague consensus — the disagreements that make a debate useful get averaged away. Letting the judge only rank (which keeps the anti-self-preference property) and having the stronger model write from that ranking preserves them.

The panel reuses models you already configure: primary_model + reviewer_model + fallback_models, deduplicated, each cloned into its own single-model client. Records debate_record.json.

3. Best-of-N tournament (pipeline/tournament.py)

Generates N candidates from diverse stances, an independent judge scores and ranks them, and the single winner proceeds — the pipeline stays linear, with one canonical artifact per stage.

Blank generations are dropped before judging rather than scored. This is a guard worth having: a judge handed an empty candidate will happily invent a rationale and a score for it, and an empty artifact can then win. Records tournament_record.json.

Stage 8 routes tournament → debate → the existing single-model multi-perspective path. That last path is untouched and remains the default.

Notes for review

• Both engines depend only on the standard library — no new dependencies.
• Stage 18 deliberately does not use the debate synthesizer split. Reviewer independence is the entire point of that stage; a synthesizer would put the author model back in the loop.
• Two sub-prompts are added (tournament_rank, debate_rebuttal). hypothesis_synthesize already existed and is reused.
• These paths multiply LLM calls when enabled, which is why they are opt-in. tournament_candidates < 2 disables the tournament.
• config.researchclaw.example.yaml is not touched in this PR; happy to document the knobs there if you'd prefer them advertised.

Testing

2900 passed, 0 failed, 56 skipped

Baseline on main at the time of branching (e2e23c9) was 2863 passed, 1 failed — the failure is a pre-existing timing-sensitive test, test_hitl_advanced.py::TestFileWait::test_poll_with_delayed_response, which passed on this run. The delta is the 36 new tests.

New coverage: debate engine (role retry, split synthesis path, judge/synthesizer wiring), tournament engine (empty-candidate drop, candidate-count effective value, ranking parse), and reviewer construction (fallback to generator, independent provider resolution, Stage 18 provenance).

🤖 Generated with Claude Code

[Jacky628]

Hi @Jiaaqiliu @huaxiuyao — flagging this for whenever you have review bandwidth, no rush.

A few notes that may make it cheaper to review:

• Nothing changes by default. All three mechanisms are opt-in and default to off; the existing single-model paths in Stages 8/18/20 are untouched. Merging this cannot alter the behaviour of any current run.
• No new dependencies — both engines use only the standard library.
• Full suite green: 2900 passed, 0 failed, 56 skipped, on top of e2e23c9.
• Overlap with open PRs: this touches llm/client.py and llm/__init__.py, which feat: add MiniMax regional endpoint presets #303 and fix(llm): surface the real error when retries are exhausted, cap OSError backoff #304 also touch. fix(llm): surface the real error when retries are exhausted, cap OSError backoff #304 edits _call_with_retry, a different region entirely. feat: add MiniMax regional endpoint presets #303 edits PROVIDER_PRESETS and the body of from_rc_config, whereas this PR appends a new classmethod after that function and two builders at the end of __init__.py — so they should merge cleanly either way, but I'm happy to rebase if you take feat: add MiniMax regional endpoint presets #303 first.

Also happy to split this into smaller PRs if that's easier to review. The three pieces are separable:

• reviewer — a separately configured model answers Stage 18 peer review and the Stage 20 quality gate instead of the model that wrote the paper, and each run records which model judged it. This matters most at Stage 20: a gate the author model decides for itself is not really a gate, and a self-review rarely produces the kind of objection that would actually stop a run.
• debate — roles argue with distinct models and rebut each other, then an independent judge ranks the positions and a synthesizer writes the final text from that ranking. A single model arguing with itself can only surface what it already believes, whereas disagreement between models is real signal; splitting ranking from writing is what preserves it, because one model doing both averages the disagreements away into consensus.
• tournament — N candidates are generated from different stances, an independent judge scores them, and only the winner proceeds, so the pipeline stays linear. Single-shot generation at these stages has high variance and nothing downstream can recover from a weak hypothesis set; best-of-N spends sampling to fix that, and keeping one canonical artifact per stage means the gain costs nothing in pipeline complexity.

The reviewer client is a dependency of the other two — both use it as the judge — so it would need to go first.

[Jiaaqiliu]

Merged. Large feature PRs usually get held for a slower review; this one landed because the risk surface is genuinely small, and I verified that rather than taking the "all three default to off" claim at face value:

reviewer_model           = ''
reviewer_provider        = ''
debate_enabled           = False
tournament_enabled       = False
debate_rounds            = 1
tournament_candidates    = 3

Parsed from an empty config. With no config change, Stage 8/18/20 take the same path they did before, and build_reviewer_llm/build_panel_llms return None/[] on any error rather than propagating — so a misconfigured reviewer degrades to current behaviour instead of failing a run.

The premise is sound and worth stating plainly: asking the author model whether its own paper passes the quality gate is not a gate. Self-preference in LLM judging is well enough documented that a self-review's silence is not evidence of quality.

Three design choices stood out as better than the obvious version:

• Splitting ranking from synthesis. A single model doing both averages the disagreement away, which is the one thing a debate is for. Letting the judge only rank preserves the anti-self-preference property while the synthesizer keeps the sharp edges.
• Dropping blank generations before judging. A judge handed an empty candidate will invent a rationale and a score for it. This is the kind of failure that produces a confident, entirely fabricated winner.
• Stage 18 deliberately not using the synthesizer split. Putting the author model back in the loop would quietly defeat the stage's entire purpose. Good that this is a documented decision rather than an oversight.

review_provenance.json is the part I'd highlight for future work — a run being auditable for reviewer independence after the fact is worth more than the mechanism being present, because it makes the claim falsifiable.

On the note about config.researchclaw.example.yaml: yes, please document the knobs there in a follow-up. Opt-in features that aren't advertised tend to stay unused.

Post-merge full suite: 2956 passed, no regressions (baseline before this batch: 2869). The pre-existing test_hitl_advanced.py::TestFileWait flake you mentioned didn't reproduce here.

[Jacky628]

Thanks for the quick turnaround on #316 — and for verifying the "all three default to off" claim instead of taking it on trust. For a PR that size that's the right instinct, and your parse of the actual defaults is a better guarantee than anything I could have asserted in the description.

The config.researchclaw.example.yaml follow-up you asked for is #317. Writing it turned into two more PRs, because I documented what is actually wired up rather than what the engines can do — and that surfaced that the mechanisms only ever reached Stage 8:

• docs(config): document the reviewer / debate / tournament knobs #317 — documents the nine settings. Beyond listing them it records the parts a user cannot infer from field names: the tournament takes priority over debate on Stage 8, and a panel needs >= 2 distinct models after deduplication or Stage 8 quietly falls back to the single-model path.
• feat(review): extend debate to Stages 14/18 and the tournament to Stage 9 #318 — extends debate to Stage 14 (result analysis) and Stage 18 (peer review), and the tournament to Stage 9 (experiment design). Stage 18 passes no synthesizer, for exactly the reason you highlighted. Stacked on docs(config): document the reviewer / debate / tournament knobs #317: merge that first and this one reduces to a single commit.
• feat(llm): pass reasoning effort through to reasoning models #319 — reasoning_effort passthrough, independent of the other two. It matters most for the judge roles: scoring is the kind of task where more deliberation changes the verdict rather than the wording. One call is left to you and flagged in the description — o1 is not covered because it is absent from _NEW_PARAM_MODELS.

On review_provenance.json: agreed, and that framing — a claim that can be falsified after the fact — is a useful lens for the rest of this. No rush on any of these.