Skip to content

Traceability fallback in pdd test --from-story is silent; no signal distinguishes a documentary story test from a behavioural one #4

Description

@agarwal-ishaan

Summary

When pdd test --from-story cannot build a behavioural test (no ## Entry Point in the contract), it silently degrades to a traceability test. Nothing anywhere tells the user the guarantee they received is weaker than the one the feature advertises.

Evidence

generator: no warning emitted on the fallback path (pdd/story_test_generation.py:270-300)
gate:      reports story-regression-ok either way
lane:      pytest -m story passes either way

story_regression_gate.py already has "traceability-only" vocabulary, but it is used exclusively for legacy hashless markers — not to distinguish a generated behavioural test from a generated text-matching one.

Impact

The user-visible sequence is:

  1. write a story
  2. run pdd test --from-story → test generated, no warning
  3. run the gate → story-regression-ok
  4. run pytest -m story → passes

Every signal says "protected". The test does not execute a single line of the code the story is about. This is exactly the shape of failure the story feature exists to catch:

A safety check that stops firing does not produce an error — it produces a successful-looking run.

Worth noting the gate itself is scrupulous about not overclaiming — its verdict constant is literally story-regression-present, with a source comment that it "must never claim a test passed". The gap is that nothing distinguishes present and behavioural from present and documentary.

Proposal

Any one of these helps; all three would be better.

  1. Warn at generation time: contract declares no ## Entry Point; generating a traceability-only test that will not exercise the code.
  2. Record the mode — in the marker, a module constant, or the generated docstring — so evaluate_stories can report story-regression-traceability-only distinct from story-regression-ok.
  3. Surface it in pdd checkup coverage, so "35 stories linked, 1 behavioural" is visible instead of hidden.

Also worth fixing while here

The fallback test's second assertion is circular: expected is a list of clauses extracted from the contract, and bundle is the story+contract text read at runtime, so it asserts the contract contains its own sentences. It can only fail when the bundle-hash assertion on the preceding line has already failed. It adds no signal and implies coverage that is not there.

Related

Surfaced while reviewing the story attached to promptdriven#2374. Companion to the ## Entry Point template issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions