fix(test --from-story): make the traceability fallback visible, not silent - #2393
Draft
agarwal-ishaan wants to merge 1 commit into
Draft
Conversation
…ilent When a story contract declares no ## Entry Point, `pdd test --from-story` silently degrades from a behavioural test to a text-pinning one. Nothing told the user the guarantee was weaker: generation printed no warning, the gate reported story-regression-ok either way, and `pytest -m story` passed either way. Every signal read "protected" for a test that never executes the code the story is about. - generated tests now declare PDD_STORY_TEST_MODE, and the generator records the mode on GeneratedStoryTest (exposed via as_dict() and .is_behavioral) - `pdd test --from-story` warns at generation time when it falls back, naming the missing ## Entry Point and how to fix it - the gate reports story-regression-traceability-only distinctly from story-regression-ok, and `pdd story --with-regression-status` labels it has-traceability-test - a story claimed by both a behavioural and a traceability test is reported behavioural: one text pin must not mask real cover - absence of the mode constant means "unknown" and is treated as behavioural, so hand-written tests are never downgraded Traceability-only is a narrower description of the same present-and-fresh outcome, not a failure: `offending` and `StoryRegressionEvaluation.passed` still accept it, so no caller's gate silently tightens. Also removes a dead assertion from the generated traceability test. It looped over clauses extracted from the bundle and asserted each appears in that same bundle. `_bundle_hash()` covers the same bytes, and normalization only strips metadata comments, trailing whitespace and blank lines -- so no edit can remove a clause while preserving the hash, and a failing hash halts the test before the loop runs. The loop could not fail under any input. Clauses are kept as PDD_STORY_ORACLE_CLAUSES / PDD_STORY_NEGATIVE_CLAUSES documentation constants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2392
Problem
When a story contract declares no
## Entry Point,pdd test --from-storysilently degrades from a behavioural test to a text-pinning one. Nothing tells the user the guarantee is weaker than the feature advertises:pdd/story_test_generation.pycontains nowarn/logger/click.echocall of any kindstory-regression-okeither waypytest -m storypasses either wayThe gate is already scrupulous about not overclaiming — its verdict constant is literally
story-regression-present, with a source comment that it "must never claim a test passed". The gap was that nothing distinguished present and behavioural from present and documentary.Change
PDD_STORY_TEST_MODE; the generator records it onGeneratedStoryTest(surfaced viaas_dict()and.is_behavioral)pdd test --from-storywarns at generation time on the fallback, naming the missing## Entry Pointand how to fix itstory-regression-traceability-onlydistinctly fromstory-regression-ok, andpdd story --with-regression-statuslabels ithas-traceability-testThe gate reads the mode with the same AST scan it already uses for story hashes (
_module_string_constants), so nothing is imported or executed.Traceability-only is a narrower description of the same present-and-fresh outcome, not a failure.
StoryRegressionGateResult.offendingandStoryRegressionEvaluation.passedboth still accept it, so no caller's gate silently tightens.Dead assertion removed
The generated traceability test looped over clauses extracted from the bundle and asserted each appears in that same bundle. That loop cannot fail under any input:
_bundle_hash()covers the same bytes the clauses were extracted from_normalized_bundleonly strips metadata comments, trailing whitespace and blank lines, so no edit can remove a clause while preserving the hashasserthalts the test before the loop runsSo it is dead code, not merely redundant. The clauses are retained as
PDD_STORY_ORACLE_CLAUSES/PDD_STORY_NEGATIVE_CLAUSESdocumentation constants, and the traceability test now makes the one assertion it can honestly make, with an actionable failure message.This drops the generated traceability
test_countfrom 2 to 1 where a contract has## Negative Cases, because the second test's only distinct content was that same dead loop.Correction to the issue's Impact section
Issue #2392 (and its companion #2391) imply the repo's own stories are unprotected — "deleting the module the story is about leaves it green". That is not true of this repository, and I want the record straight before anyone acts on it.
All 11 tests in
tests/story_regression/are hand-written, not generated, and they do import and exercise real code (pdd.commands.analysis.bug,pdd.commands.modify.change,pdd.commands.maintenance.sync, …). Zero generated text-pins exist in the tree.So these are real defects in
pdd test --from-storythat will affect anyone who uses it, but this repo's story lane is not currently exposed. The severity is "the generator produces a weak artifact and doesn't say so", not "the safety net is fake".Follow-up not addressed here
Running
pdd test --from-storyagainst any existing story would overwrite a hand-written behavioural test with a text pin. I hit this accidentally while preparing this PR — a regeneration pass replaced 9 hand-written behavioural tests with text pins before I caught it in the diff (reverted, nothing lost).That is a silent data-loss path in a shipped command, and arguably more serious than the issue this PR fixes. It is deliberately out of scope here because guarding it changes CLI behaviour and deserves its own decision — happy to file it or fix it on request.
Tests
236 passing across
test_story_test_generation.py,test_story_test_generator.py,test_story_regression_gate.py,test_story_regression.py,test_coverage_contracts.py, and thetests/story_regressionlane.Two existing tests were updated rather than worked around, both because they encoded the behaviour this PR corrects:
test_story_regression_gate_detects_missing_passing_and_staleassertedSTATUS_PASSINGfor a generated text-pin; it now asserts the sharperSTATUS_STORY_REGRESSION_TRACEABILITY_ONLYand that.passed/has_regression_teststay true, pinning the "narrower, not failing" guarantee.test_pdd_test_from_story_cli_without_entry_point_writes_text_pinasserted2 passed; now1 passed, plus an assertion that the mode constant is emitted.No new files under
pdd/prompts/, so this PR does not touch the managed-unit registration path.🤖 Generated with Claude Code