pdd detect --stories verdict depends on model strength: pass criterion is 'empty changes list' from an open-ended prompt

[agarwal-ishaan]

Summary

pdd detect --stories passes only when detect_change returns an empty changes_list. That makes the verdict depend on model strength rather than on whether the prompts satisfy the contract.

Evidence

Three runs, same story, same contract, same seven prompts, only --strength varied:

run	strength	model	verdict	quality of findings
1	0.2	gemini-3-flash-preview	FAIL	2 genuine gaps — error precedence when several gates fire, and repair-directive content. Both became numbered contract rules.
2	0.2	gemini-3-flash-preview	FAIL	noise — asked to "strengthen R15 to ensure signature_details carries the exact declared target, preventing the repair loop from having to infer the shape", when R15 already ended "...rather than having to infer it". Also asked to retain a rule that already existed, and to add a "Critical for AC5" label.
3	0.9	claude-sonnet-4-6 → fell back to gemini-3.5-flash	PASS	empty changes list

Run 1 earned its keep. Run 2 shows what the pass criterion does when there is nothing left to find.

Analysis

The pass condition is "the model volunteers zero changes" in response to an open-ended what would you change? — a question LLMs are strongly biased to answer with something. So:

• a weaker model fails a correct prompt set
• the same inputs flip verdict on strength alone
• the operator must exercise judgement to reject suggestions. In run 1 I rejected 3 of 5 on technical grounds: one asked for text already present verbatim in interface_check R9, another wanted "checksum match" where the rule already said the strictly stronger "byte-identical". Applying pdd fix unreviewed would have made the prompts worse.

A gate that measures the model rather than the code is not a sound gate, though it is a genuinely useful advisor.

Resolution (in progress)

pdd/story_criteria.py on worktree-issue-5-story-criteria replaces the open-ended question with a bounded one, per its own docstring:

Every acceptance criterion in the story contract is classified exactly once as satisfied, unsatisfied, or unclear, and a satisfied verdict must quote the prompt text that satisfies it. Only unsatisfied fails the story; unclear is advisory, and a criterion the evaluator skipped entirely is unevaluated — an incomplete run, never a pass.

That matches the per-criterion classification this issue argued for, including the citation requirement, and adds an unevaluated state that closes a hole the original report missed: a criterion the evaluator silently skipped must not read as a pass.

Branch contents: pdd/story_criteria.py (428 lines), tests/test_story_criteria.py (413), story_criteria_LLM.prompt and story_criteria_python.prompt, wiring in pdd/commands/analysis.py and pdd/user_story_tests.py, plus architecture.json, .pdd/expected-managed.json, .pdd/sync-ownership.json and the accompanying manifest.py re-pin.

Related

Surfaced while validating the story attached to #2374.

[agarwal-ishaan]

Target 08/11

[agarwal-ishaan]

Measured gap: the bounded gate is unsound below ~strength 0.5

Implementing the proposal in this issue (PR #2390) and then testing it against a live provider turned up a limitation worth recording here, because it changes how the issue's two proposals relate to each other.

Setup

Story story__pdd_bug.md + its contract (4 acceptance criteria) against prompts/commands/analysis_python.prompt. A "regressed" variant deletes every statement constraining --clean-restart — both the MUST raise click.UsageError sentence and the "agentic mode only" qualifier — so AC3 (reject --clean-restart outside agentic GitHub issue mode) is genuinely unsatisfied while AC1/AC2/AC4 are untouched. 5 runs per cell.

Result

strength	model	correct prompts	regressed prompts
0.2	gemini-3-flash-preview	PASS 5/5 ✅	PASS 5/5 ❌
0.5	gemini-3.5-flash	PASS 5/5 ✅	FAIL 5/5 ✅
0.7	gemini-3.5-flash	PASS ✅	FAIL ✅
0.9	gemini-3.5-flash	PASS ✅	FAIL ✅

At strength 0.2 the gate misses a real regression, reproducibly, and does so confidently — AC3 comes back satisfied 5 runs out of 5, not unclear. Above that tier it is correct in both directions, 5/5 each.

Why the citation requirement does not catch it

The proposal's citation requirement is doing its job, and still does not prevent this. The evidence quoted in support of the wrong verdict was:

evidence: Options: --manual, --output, --language, ...

That is genuine, verbatim prompt text — so the automated verbatim check passes. But it merely mentions --clean-restart; it says nothing about rejecting it, which is what AC3 requires.

So the citation makes the error visible rather than impossible. A human scanning the report sees a criterion about rejecting a flag "supported" by a list of flags and spots it immediately — that is how this was found. That matches what the issue claims for it ("the citation makes each verdict checkable") and no more; it is an audit aid, not a correctness guarantee.

Consequence for the proposal

The issue offers bounded classification as the fix and a strength floor as the fallback ("failing that, the minimum fix is to pin a floor strength for story validation and state in the output which model produced the verdict"). The measurement says these are complementary, not alternatives:

• a strength floor could never have rescued the legacy gate — in the same experiment it returned FAIL on correct prompt sets at 0.2, 0.5 and 0.9, so there is no strength at which it discriminates
• a strength floor does rescue the bounded gate, because the bounded gate is correct above 0.5 and wrong below it

So the fallback proposal is worth implementing on top of the main one, not instead of it. Concretely: warn (or refuse) when story validation runs below a floor strength, and keep the model provenance line PR #2390 already prints (judged by <model>) so a cheap-model verdict is never mistaken for a strong one.

Caveat on the measurement

Strengths 0.5, 0.7 and 0.9 all resolved to gemini-3.5-flash in my environment (Anthropic credits exhausted, so the Claude tiers failed over to Gemini). This is therefore two model tiers rather than four: the 0.2-vs-rest contrast is real, the 0.5-vs-0.9 comparison is not informative, and a Sonnet-class model is untested. The exact position of the cliff between 0.2 and 0.5 is also unmeasured.

[agarwal-ishaan]

Root cause is structural: the validator and the change propagator share one engine

Working through this issue surfaced a cause underneath the one it reports. Recording it here rather than opening a separate issue, since the fix belongs with this work.

The finding

pdd detect --stories is already a validator in every respect except its engine:

• its own document schema pdd.detect.stories.v1
• its own verdict vocabulary PASS / FAIL / UNKNOWN
• its own exit codes, including 3 for "incomplete-evaluation"
• read-only, non-interactive semantics, an exact-scope manifest

None of that belongs to a change detector. But the engine underneath it was detect_change() running detect_change_LLM — the same function and the same prompt used for change propagation. The only difference was how the caller read the result: standard mode treats the change list as the deliverable; story mode treated "list is empty" as a verdict.

That is the whole defect in one line. A change list is the right answer to "what should change?" and a bad proxy for "is this correct?", because an empty list is the only passing value and nothing in that prompt was ever asked to produce one — its task steps walk the model toward "generate at least three implementation plans → select one → list the prompts to change."

Evidence that it is structural, not model quality

In the controlled experiment posted above, the legacy gate returned FAIL on correct prompt sets at strength 0.2, 0.5 and 0.9. There is no strength at which it discriminates between a correct and a regressed prompt set. A stronger model does not fix a question that cannot be answered with "nothing".

The surface is misplaced too

Story authoring lives under pdd story (add, link, list). Story validation lives under pdd detect --stories. The namespace seam is in the wrong place, and pdd story verify — the obvious home — does not exist. pdd verify is unrelated (it runs a verification program against generated code, with repair attempts).

Proposed shape

Three layers, only one of which is missing:

1. Engine — exists. PR fix(detect --stories): gate stories on per-criterion verdicts, not model silence #2390 adds evaluate_acceptance_criteria(...) -> CriteriaEvaluation, which returns a per-criterion satisfied/unsatisfied/unclear list, and changes_from_verdicts(), which renders the unsatisfied rows into the existing {prompt_name, change_instructions} shape. Verification already produces a structured change list as a by-product.
2. Front door — missing. pdd story verify, alongside pdd story add/link/list. A subcommand, not a flag: a flag would bolt verification onto detect and deepen the mismatch.
3. Callers — rewiring. run_user_story_fix currently plans a fix with detect_change (user_story_tests.py:2084) and then verifies it with the criteria gate (:2180) — two different questions in one loop. It can apply "strengthen this wording" edits the gate never asked for and will not credit. Pointing it at the engine makes every applied edit traceable to a named failed criterion, and drops that path from 3 LLM calls to 1.

Why this inverts the dependency usefully

Today change propagation is the primitive and verification is derived from it (badly). Reversed, verification is the primitive and the change list falls out of it: each proposed change traces to one failed criterion, with a citable reason, instead of being a free-form rewrite suggestion.

This applies to the story-driven path. It does not generalize to standard pdd detect, whose input is an arbitrary change description with no contract and therefore nothing to verify against — that remains a genuinely different question and detect_change stays as-is for it.

Scope

Being addressed under this issue rather than split out. detect_change() itself is not modified; pdd detect --stories keeps working (with --legacy-detect as the escape hatch).