fix: pdd-change Step 6 should use <pdd-dependency> tags, not <include> tags

[agarwal-ishaan]

Summary

pdd change Step 6 built its module-dependency context by scanning <include> tags (build_dependency_graph()), but <include> tags are LLM context, not declared architectural dependencies. A module can truly depend on another via <pdd-dependency> without ever <include>-ing it, and can <include> another module's example purely for style without depending on it at all -- so the old graph both missed real dependents and fabricated false ones.

1. Prompts (pdd/prompts/sync_order_python.prompt, pdd/prompts/agentic_change_orchestrator_python.prompt): specify a new build_dependency_graph_from_architecture() function and wire Step 6's dependency context to use it.
2. pdd/sync_order.py: implements build_dependency_graph_from_architecture(architecture_path) -> Dict[str, List[str]], reading architecture.json's dependencies field (populated from <pdd-dependency> tags by architecture_sync.py) instead of scanning <include> tags. Generated via pdd generate --incremental.
3. pdd/agentic_change_orchestrator.py: _build_dependency_context() now takes architecture_path and calls the function above instead of the include-based build_dependency_graph(prompts_dir) -- the actual Step 6 behavior fix.

Test plan

• pytest tests/test_sync_order.py -- 52 passed
• pytest tests/test_agentic_change_orchestrator.py -- 265 passed
• Manual repro (pdd-issue-1807-repro/): _build_dependency_context() now correctly shows payment_status -> affects: customer_service (previously missing) and no longer fabricates legacy_wizard -> affects: onboarding
• Confirmed no regression: sibling modules (agentic_sync, sync_graph_order_consistency) that also import from sync_order still import cleanly

Fixes #1807

[gltanaka]

/heal

[gltanaka]

Thanks for consolidating the earlier prompt and implementation PRs. The architecture-based direction is sensible, but this version has two merge-blocking correctness problems:

1. architecture.json.dependencies is not currently a <pdd-dependency>-only data source. merge_auto_deps_includes_into_architecture() intentionally writes module/example <include> targets into the same array, and existing tests require that behavior. Consequently, reading this field cannot prove that an include-only module edge is excluded. The new Step 6 test hand-writes JSON without the include edge, so it does not exercise the real prompt → architecture → Step 6 path.

2. build_dependency_graph_from_architecture() uses extract_module_from_include() for architecture identities. That helper drops directories and rejects _LLM.prompt files. This loses real declared dependencies in the current repository. For example, commands/gate_python.prompt -> gate_python.prompt collapses to gate -> gate and is discarded as a self-edge; dependencies such as agentic_update_python.prompt -> agentic_update_LLM.prompt are also omitted.

Please resolve the dependency provenance/semantics and use a path-preserving architecture identity that supports LLM modules. Add regressions for:

• a real prompt-to-architecture-to-Step-6 flow distinguishing <pdd-dependency> from context-only <include>;
• path-qualified modules with the same basename;
• _LLM.prompt dependencies;
• root plus nested architecture files.

The focused suites pass (317 tests), but they cover only flat _python.prompt names and do not exercise these production shapes. Follow-up scope should also address the issue's requested prompting-guide examples and the retained top-30/top-10 truncation.

[agarwal-ishaan]

Implemented the above changes, requesting you to please review

[gltanaka]

Re-review of current head 8c4338c9 against origin/main (f25a0139), including the complete PR/issue history, superseded PRs #2373/#2375, all commits, the cumulative diff, architecture.json, prompt sources, prompting guide, and the PR-loop runbook. There are no inline review threads (0 resolved / 0 unresolved). The core provenance correction is directionally right, but this is not merge-ready.

Blocking findings

1. P1 — the verification-profile transition fails on the actual PR base.

   pdd/sync_core/verification.py:2782 validates the protected active requirement rotations before it can apply the candidate retirement added in .pdd/verification-profile-rotations.json:676. On this exact head, the production path
   load_verification_profiles(build_unit_manifest(root, base_ref="origin/main", head_ref="HEAD"))
   raises VerificationProfileError: protected active requirement transition rules are ambiguous because protected origin/main has two active code_generator_main_python.prompt/python rows. The current hash-only reconciliation test does not exercise the real base-to-head load.

   Please either land/rebase onto the prerequisite protected retirement and remove the duplicate candidate retirement, or add a narrowly byte-bound bridge that can validate this exact transition before protected-row uniqueness is enforced. Add the exact origin/main -> PR head production-loader regression.

2. P1 — Step 6 still drops most real dependency context.

   pdd/agentic_change_orchestrator.py:2284-2290 retains the old [:30] module and [:10] dependent slices, and pdd/prompts/agentic_change_orchestrator_python.prompt:329 explicitly requires them. Against this repository's real architecture-derived graph, 135 dependency targets have dependents, but only 30 are rendered: 105 targets and 8 additional dependent names disappear. In particular, the newly preserved agentic_update -> agentic_update_LLM edge exists in the graph, but agentic_update_LLM ranks 68th and is absent from the actual Step 6 context. Thus the prior _LLM omission remains at the user-visible boundary.

   Render all edges in a compact deterministic representation (or another complete representation with explicit fidelity guarantees) and add >30-target and >10-dependent regressions at _build_dependency_context, not only at the graph helper.

3. P1 — nested architecture support is only partially implemented and the requested real flow is still untested.

   pdd/sync_order.py:244,261 assumes prompt files live under the architecture file's own prompts/ or pdd/prompts/. That contradicts the established contract in pdd/architecture_sync.py:118-138 / its tests: a nested architecture.json may use an ancestor project's prompts/ directory. A deterministic reproduction with repo/prompts/auth_python.prompt and repo/backend/functions/architecture.json returns {'auth': [], 'log': []} and warnings instead of auth -> log. The valid root entry context/python_preamble.prompt is likewise misresolved as pdd/prompts/context/python_preamble.prompt.

   The advertised “real prompt -> architecture” tests do not cover this boundary: tests/test_sync_order.py:827-850 says it does not hand-write architecture JSON, but lines 842-848 do exactly that; tests/test_agentic_change_orchestrator.py:10784 does the same. Neither runs architecture sync/auto-deps, where include-derived edges make architecture.json.dependencies mixed-provenance.

   Resolve prompts through the governing/nearest supported prompt root (and safe architecture-relative paths), then add a deterministic production-flow regression that creates architecture state through sync/auto-deps and invokes Step 6. Cover root and nested architectures, including the ancestor-prompts shape.

4. P2 — the ancillary include-parser fix still treats fenced examples as directives.

   pdd/sync_core/includes.py:211 and :264 skip only a tag whose immediately preceding character is a backtick. <include> and <include-many> inside triple-backtick or ~~~ fenced examples are still returned as real references, contrary to docs/prompting_guide.md:1404; this can make closure validation/fingerprinting require example-only files. The new tests cover inline code only.

   Exclude both Markdown fence styles while preserving the supported legacy <path> include grammar, and add fenced XML/include-many regressions.

5. P2 — the linked issue's prompting-guide acceptance work remains absent.

   #1807 explicitly asks for practical examples explaining when a real module relationship uses <pdd-dependency> and when files are only LLM context via <include>. This PR does not change docs/prompting_guide.md. Its current line 1645 also says <include> “does NOT affect architecture,” while the implementation and this PR correctly acknowledge that auto-deps can write include-derived edges into the architecture dependencies array.

   Add the requested customer-service/payment-status versus refund-policy/tone-context examples and reconcile that mixed-field behavior in the guide. If that work is intentionally deferred, this PR should not close #1807; link a narrower issue instead.

Prior-request checklist

Prior request	Implementing evidence	Current status
Stop Step 6 using the include graph	4b6349dfa	satisfied
Do not trust mixed architecture.json.dependencies as pdd-dependency-only	626002541, fresh prompt-tag parsing	satisfied
Preserve path-qualified same-basename identities	626002541; tests/test_sync_order.py:877-895	satisfied
Preserve _LLM.prompt dependencies	626002541; graph test at :898-914	partially satisfied — graph keeps it, Step 6 truncates it
Root + nested architecture behavior	626002541; local-prompts-only test at :917-935	partially satisfied — supported ancestor prompt roots fail
Real prompt -> architecture -> Step 6 regression	claimed by 626002541	not addressed — tests hand-write architecture JSON
Prompting-guide distinction and examples	no implementing commit/diff	not addressed
Remove top-30/top-10 loss	no implementing commit; slices remain	not addressed
Verification-profile reconciliation	849fdd5ac, 460998285, 7abac24eb	regressed/blocking — production base-to-head load fails
Ignore documentation examples in include parsing	8c4338c97	partially satisfied — inline works, fenced examples regress
Initial direct architecture-field design / first restamp	superseded by 626002541; 0181bd446 reverted by 315bd833	obsolete with valid supersession

Validation evidence: 323 focused orchestrator/sync-order tests passed; 21 architecture/auto-deps boundary tests passed; GitHub's Unit Tests, Public CLI Regression, Story Regression, Package Preprocess Smoke, Repo Bloat Docker E2E, and heal checks are green. Those green checks do not cover the failing production profile load or the deterministic Step 6/nested/fence reproductions above.