pdd detect --stories verdict depends on model strength: pass criterion is 'empty changes list' from an open-ended prompt

[agarwal-ishaan]

Summary

pdd detect --stories passes only when detect_change returns an empty changes_list. That makes the verdict depend on model strength rather than on whether the prompts satisfy the contract.

Evidence

Three runs, same story, same contract, same seven prompts, only --strength varied:

run	strength	model	verdict	quality of findings
1	0.2	gemini-3-flash-preview	FAIL	2 genuine gaps — error precedence when several gates fire, and repair-directive content. Both became numbered contract rules.
2	0.2	gemini-3-flash-preview	FAIL	noise — asked to "strengthen R15 to ensure signature_details carries the exact declared target, preventing the repair loop from having to infer the shape", when R15 already ended "...rather than having to infer it". Also asked to retain a rule that already existed, and to add a "Critical for AC5" label.
3	0.9	claude-sonnet-4-6 → fell back to gemini-3.5-flash	PASS	empty changes list

Run 1 earned its keep. Run 2 shows what the pass criterion does when there is nothing left to find.

Analysis

The pass condition is "the model volunteers zero changes" in response to an open-ended what would you change? — a question LLMs are strongly biased to answer with something. So:

• a weaker model fails a correct prompt set
• the same inputs flip verdict on strength alone
• the operator must exercise judgement to reject suggestions. In run 1 I rejected 3 of 5 on technical grounds: one asked for text already present verbatim in interface_check R9, another wanted "checksum match" where the rule already said the strictly stronger "byte-identical". Applying pdd fix unreviewed would have made the prompts worse.

A gate that measures the model rather than the code is not a sound gate, though it is a genuinely useful advisor.

Resolution (in progress)

pdd/story_criteria.py on worktree-issue-5-story-criteria replaces the open-ended question with a bounded one, per its own docstring:

Every acceptance criterion in the story contract is classified exactly once as satisfied, unsatisfied, or unclear, and a satisfied verdict must quote the prompt text that satisfies it. Only unsatisfied fails the story; unclear is advisory, and a criterion the evaluator skipped entirely is unevaluated — an incomplete run, never a pass.

That matches the per-criterion classification this issue argued for, including the citation requirement, and adds an unevaluated state that closes a hole the original report missed: a criterion the evaluator silently skipped must not read as a pass.

Branch contents: pdd/story_criteria.py (428 lines), tests/test_story_criteria.py (413), story_criteria_LLM.prompt and story_criteria_python.prompt, wiring in pdd/commands/analysis.py and pdd/user_story_tests.py, plus architecture.json, .pdd/expected-managed.json, .pdd/sync-ownership.json and the accompanying manifest.py re-pin.

Related

Surfaced while validating the story attached to promptdriven#2374.