Summary
When pdd test --from-story cannot build a behavioural test (no ## Entry Point in the contract), it silently degrades to a traceability test. Nothing anywhere tells the user the guarantee they received is weaker than the one the feature advertises.
Evidence
generator: no warning emitted on the fallback path (pdd/story_test_generation.py:270-300)
gate: reports story-regression-ok either way
lane: pytest -m story passes either way
story_regression_gate.py already has "traceability-only" vocabulary, but it is used exclusively for legacy hashless markers — not to distinguish a generated behavioural test from a generated text-matching one.
Impact
The user-visible sequence is:
- write a story
- run
pdd test --from-story → test generated, no warning
- run the gate →
story-regression-ok
- run
pytest -m story → passes
Every signal says "protected". The test does not execute a single line of the code the story is about. This is exactly the shape of failure the story feature exists to catch:
A safety check that stops firing does not produce an error — it produces a successful-looking run.
Worth noting the gate itself is scrupulous about not overclaiming — its verdict constant is literally story-regression-present, with a source comment that it "must never claim a test passed". The gap is that nothing distinguishes present and behavioural from present and documentary.
Proposal
Any one of these helps; all three would be better.
- Warn at generation time:
contract declares no ## Entry Point; generating a traceability-only test that will not exercise the code.
- Record the mode — in the marker, a module constant, or the generated docstring — so
evaluate_stories can report story-regression-traceability-only distinct from story-regression-ok.
- Surface it in
pdd checkup coverage, so "35 stories linked, 1 behavioural" is visible instead of hidden.
Also worth fixing while here
The fallback test's second assertion is circular: expected is a list of clauses extracted from the contract, and bundle is the story+contract text read at runtime, so it asserts the contract contains its own sentences. It can only fail when the bundle-hash assertion on the preceding line has already failed. It adds no signal and implies coverage that is not there.
Related
Surfaced while reviewing the story attached to promptdriven#2374. Companion to the ## Entry Point template issue.
Summary
When
pdd test --from-storycannot build a behavioural test (no## Entry Pointin the contract), it silently degrades to a traceability test. Nothing anywhere tells the user the guarantee they received is weaker than the one the feature advertises.Evidence
story_regression_gate.pyalready has "traceability-only" vocabulary, but it is used exclusively for legacy hashless markers — not to distinguish a generated behavioural test from a generated text-matching one.Impact
The user-visible sequence is:
pdd test --from-story→ test generated, no warningstory-regression-okpytest -m story→ passesEvery signal says "protected". The test does not execute a single line of the code the story is about. This is exactly the shape of failure the story feature exists to catch:
Worth noting the gate itself is scrupulous about not overclaiming — its verdict constant is literally
story-regression-present, with a source comment that it "must never claim a test passed". The gap is that nothing distinguishes present and behavioural from present and documentary.Proposal
Any one of these helps; all three would be better.
contract declares no ## Entry Point; generating a traceability-only test that will not exercise the code.evaluate_storiescan reportstory-regression-traceability-onlydistinct fromstory-regression-ok.pdd checkup coverage, so "35 stories linked, 1 behavioural" is visible instead of hidden.Also worth fixing while here
The fallback test's second assertion is circular:
expectedis a list of clauses extracted from the contract, andbundleis the story+contract text read at runtime, so it asserts the contract contains its own sentences. It can only fail when the bundle-hash assertion on the preceding line has already failed. It adds no signal and implies coverage that is not there.Related
Surfaced while reviewing the story attached to promptdriven#2374. Companion to the
## Entry Pointtemplate issue.