Skip to content

fix(test --from-story): make the traceability fallback visible, not silent - #2393

Draft
agarwal-ishaan wants to merge 1 commit into
promptdriven:mainfrom
agarwal-ishaan:fix/issue-2392-story-test-mode-signal
Draft

fix(test --from-story): make the traceability fallback visible, not silent#2393
agarwal-ishaan wants to merge 1 commit into
promptdriven:mainfrom
agarwal-ishaan:fix/issue-2392-story-test-mode-signal

Conversation

@agarwal-ishaan

Copy link
Copy Markdown
Contributor

Fixes #2392

Problem

When a story contract declares no ## Entry Point, pdd test --from-story silently degrades from a behavioural test to a text-pinning one. Nothing tells the user the guarantee is weaker than the feature advertises:

  • pdd/story_test_generation.py contains no warn/logger/click.echo call of any kind
  • the gate reports story-regression-ok either way
  • pytest -m story passes either way

The gate is already scrupulous about not overclaiming — its verdict constant is literally story-regression-present, with a source comment that it "must never claim a test passed". The gap was that nothing distinguished present and behavioural from present and documentary.

Change

  • generated tests declare a module-level PDD_STORY_TEST_MODE; the generator records it on GeneratedStoryTest (surfaced via as_dict() and .is_behavioral)
  • pdd test --from-story warns at generation time on the fallback, naming the missing ## Entry Point and how to fix it
  • the gate reports story-regression-traceability-only distinctly from story-regression-ok, and pdd story --with-regression-status labels it has-traceability-test
  • a story claimed by both a behavioural and a traceability test is reported behavioural — one text pin must not mask real cover
  • absence of the constant means "unknown" and is treated as behavioural, so hand-written tests are never downgraded

The gate reads the mode with the same AST scan it already uses for story hashes (_module_string_constants), so nothing is imported or executed.

Traceability-only is a narrower description of the same present-and-fresh outcome, not a failure. StoryRegressionGateResult.offending and StoryRegressionEvaluation.passed both still accept it, so no caller's gate silently tightens.

Dead assertion removed

The generated traceability test looped over clauses extracted from the bundle and asserted each appears in that same bundle. That loop cannot fail under any input:

  • _bundle_hash() covers the same bytes the clauses were extracted from
  • _normalized_bundle only strips metadata comments, trailing whitespace and blank lines, so no edit can remove a clause while preserving the hash
  • and when the hash assertion fails, the assert halts the test before the loop runs

So it is dead code, not merely redundant. The clauses are retained as PDD_STORY_ORACLE_CLAUSES / PDD_STORY_NEGATIVE_CLAUSES documentation constants, and the traceability test now makes the one assertion it can honestly make, with an actionable failure message.

This drops the generated traceability test_count from 2 to 1 where a contract has ## Negative Cases, because the second test's only distinct content was that same dead loop.

Correction to the issue's Impact section

Issue #2392 (and its companion #2391) imply the repo's own stories are unprotected — "deleting the module the story is about leaves it green". That is not true of this repository, and I want the record straight before anyone acts on it.

All 11 tests in tests/story_regression/ are hand-written, not generated, and they do import and exercise real code (pdd.commands.analysis.bug, pdd.commands.modify.change, pdd.commands.maintenance.sync, …). Zero generated text-pins exist in the tree.

So these are real defects in pdd test --from-story that will affect anyone who uses it, but this repo's story lane is not currently exposed. The severity is "the generator produces a weak artifact and doesn't say so", not "the safety net is fake".

Follow-up not addressed here

Running pdd test --from-story against any existing story would overwrite a hand-written behavioural test with a text pin. I hit this accidentally while preparing this PR — a regeneration pass replaced 9 hand-written behavioural tests with text pins before I caught it in the diff (reverted, nothing lost).

That is a silent data-loss path in a shipped command, and arguably more serious than the issue this PR fixes. It is deliberately out of scope here because guarding it changes CLI behaviour and deserves its own decision — happy to file it or fix it on request.

Tests

236 passing across test_story_test_generation.py, test_story_test_generator.py, test_story_regression_gate.py, test_story_regression.py, test_coverage_contracts.py, and the tests/story_regression lane.

Two existing tests were updated rather than worked around, both because they encoded the behaviour this PR corrects:

  • test_story_regression_gate_detects_missing_passing_and_stale asserted STATUS_PASSING for a generated text-pin; it now asserts the sharper STATUS_STORY_REGRESSION_TRACEABILITY_ONLY and that .passed / has_regression_test stay true, pinning the "narrower, not failing" guarantee.
  • test_pdd_test_from_story_cli_without_entry_point_writes_text_pin asserted 2 passed; now 1 passed, plus an assertion that the mode constant is emitted.

No new files under pdd/prompts/, so this PR does not touch the managed-unit registration path.

🤖 Generated with Claude Code

…ilent

When a story contract declares no ## Entry Point, `pdd test --from-story`
silently degrades from a behavioural test to a text-pinning one. Nothing told
the user the guarantee was weaker: generation printed no warning, the gate
reported story-regression-ok either way, and `pytest -m story` passed either
way. Every signal read "protected" for a test that never executes the code the
story is about.

- generated tests now declare PDD_STORY_TEST_MODE, and the generator records
  the mode on GeneratedStoryTest (exposed via as_dict() and .is_behavioral)
- `pdd test --from-story` warns at generation time when it falls back, naming
  the missing ## Entry Point and how to fix it
- the gate reports story-regression-traceability-only distinctly from
  story-regression-ok, and `pdd story --with-regression-status` labels it
  has-traceability-test
- a story claimed by both a behavioural and a traceability test is reported
  behavioural: one text pin must not mask real cover
- absence of the mode constant means "unknown" and is treated as behavioural,
  so hand-written tests are never downgraded

Traceability-only is a narrower description of the same present-and-fresh
outcome, not a failure: `offending` and `StoryRegressionEvaluation.passed`
still accept it, so no caller's gate silently tightens.

Also removes a dead assertion from the generated traceability test. It looped
over clauses extracted from the bundle and asserted each appears in that same
bundle. `_bundle_hash()` covers the same bytes, and normalization only strips
metadata comments, trailing whitespace and blank lines -- so no edit can remove
a clause while preserving the hash, and a failing hash halts the test before the
loop runs. The loop could not fail under any input. Clauses are kept as
PDD_STORY_ORACLE_CLAUSES / PDD_STORY_NEGATIVE_CLAUSES documentation constants.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Traceability fallback in pdd test --from-story is silent; no signal distinguishes a documentary story test from a behavioural one

1 participant