Skip to content

Story contracts cannot produce behavioural tests: template omits ## Entry Point / ## Seams #3

Description

@agarwal-ishaan

Summary

pdd test --from-story can generate a behavioural test that imports a callable, invokes it, and asserts the ## Oracle / ## Negative Cases as Python expressions. It only does so when the story's contract declares a machine-readable ## Entry Point. Otherwise it falls through to a traceability test that only checks text.

That behavioural path is effectively unreachable today, because the contract template does not offer the sections the generator needs.

Evidence

sections in user_stories/contracts/template.contract.md matching "Entry Point" or "Seams":  0
contracts in user_stories/contracts/ declaring "## Entry Point":                            1 of 35

The template offers Covers, Context, Acceptance Criteria, Oracle, Non-Oracle, Negative Cases, Non-Goals, Candidate Prompts, Notes. _llm_generate_story_contract generates from that template, so essentially every contract in the repo lands in the fallback.

pdd/story_test_generation.py:273-282 documents the two modes:

If the contract declares a machine-readable ## Entry Point, generate a behavioral test that calls the entry point and asserts the ## Oracle / ## Negative Cases as Python expressions over result … Contracts without an Entry Point fall through to [traceability].

Two subsystems designed against different section vocabularies that never meet.

Impact

A generated story test for a contract without an Entry Point asserts only:

  1. the story+contract bundle hash is unchanged
  2. each Oracle/Negative clause appears in the bundle — where the bundle is the story+contract text those clauses were extracted from

So the test never imports the code under test. Deleting the module the story is about leaves it green. The story becomes a change-detector for a markdown document rather than a regression oracle for behaviour — which is the failure mode the story feature exists to prevent.

Proposal

  1. Add ## Entry Point and ## Seams to user_stories/contracts/template.contract.md.
  2. Teach _llm_generate_story_contract to populate them from the story's linked prompts (the declared <pdd-interface> of a pdd-story-prompts target is a natural source for the entry point).
  3. Backfill existing contracts opportunistically — regeneration already happens when a story changes.

Related

Surfaced while reviewing the story attached to promptdriven#2374. See also the companion issue on the silent fallback.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions