feat(adf): telemetry-driven 'adf suggest' diagnostics + co-occurrence heuristic#245
Merged
Merged
Conversation
…access.adf The module named the private Stackbilt-dev/stackbilt_llc repo, AEGIS's disambiguation firewall, CodeBeast's DATA_AUTHORITY class, the aegis-web consumer pattern, and private issue links -- Stackbilt's actual internal service architecture, checked into the public Charter repo. Contradicts this repo's own OSS policy (no product logic; framework patterns and generic utilities only). Kept the generic sensitivity-tier pattern (6 tiers, constraints, advisory) as legitimate OSS content; removed the Stackbilt-specific instantiation (repo paths, service names, issue refs). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…diagnostics ADF's manifest routing already computes buildTriggerReport() (which triggers matched/missed) but discarded it; telemetry only recorded command name + exit code, nothing about which modules resolved. - packages/adf: export buildTriggerReport publicly (was internal to manifest.ts), needed by the on-demand consumers below. - telemetry.ts: new AdfResolutionEvent type + recordAdfResolutionEvent(), a sibling event type in the existing .charter/telemetry/events.ndjson file, discriminated by eventType. Adds an optional sessionId (from CHARTER_SESSION_ID env, or an explicit override) for future joining. - Hooked into the three places a task resolves modules: adf-bundle.ts, adf-context.ts (both the --bundle path and the previously-bare list-only path), and serve.ts's getProjectContext MCP tool (which now mints one sessionId per server process so all tool calls in an agent session share it). - New 'charter adf suggest' command: report-only diagnostics over the persisted telemetry -- dead modules (never fire), recurring unmatched keywords (stopword-filtered), and modules that load yet a downstream validate/drift/evidence call still fails nearby (joined by sessionId, or a forward-only time-window fallback). No patch ops are generated: attributing a failure or keyword to a specific module's trigger list would require knowing which module's constraint was violated, and nothing in validate/drift/evidence telemetry records that today. 23 new tests in adf-suggest.test.ts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a narrower, self-contained mechanism on top of the report-only detectors: if an unmatched keyword frequently co-occurs (same task) with a keyword that already triggers module M, M is a *candidate* to adopt it as a new trigger. This is weaker evidence than a failure correlation -- 'these words tend to show up together', not 'this caused a violation' -- so it's kept separate from the failure-backed signals and clearly labeled as a heuristic in both the report output and the help text. --emit-ops <file> writes non-ambiguous candidates as REPLACE_BULLET ops grouped per module (candidates targeting the same module are merged into one op -- applying two separate ops at the same list index in sequence would silently clobber the first, a real bug caught via end-to-end testing with 'adf patch' before this shipped). Nothing is ever applied automatically; ambiguous candidates (tied top module) are reported but never turned into an op. Known limitation, called out in the code and CLI help: this heuristic can propose low-signal candidates (e.g. a co-occurring content word with no real relevance) and, if accepted repeatedly, mildly self-reinforces toward trigger-list growth on frequently-loaded modules -- the opposite of manifest.adf's own 'prefer smallest relevant module set' rule. Human review before applying is the load-bearing safeguard, not a formality. 10 new tests (33 total in adf-suggest.test.ts). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Correctness fixes: - buildSuggestReport's dead-module/near-miss counters incremented once per (module, trigger) pair instead of once per resolution event, so a module with N declared triggers reached --min-occurrences after a single invocation while a single-trigger module needed N. Dedup module and keyword sets per event before counting. - hasCorrelatedFailure scanned the full command-event history per resolution event (O(resolutions x commands)); precompute the (typically much smaller) failed-command subset once instead. - readEvents()'s command-event validation only checked timestamp, weaker than telemetry.ts's own reader (which also requires commandPath/exitCode) -- a malformed line could silently count as a valid command outcome. Match the stricter check. - tokenizeTask() and the MCP getProjectContext keyword split didn't dedupe, unlike adf-context.ts's keyword handling -- a single task with a repeated word could alone cross --min-occurrences. Dedupe at the source. - Unified sessionId derivation: telemetry.ts's getSessionId() treats an empty CHARTER_SESSION_ID as unset (via ||), but serve.ts independently read the env var with ??, which does not. Export getSessionId and use it in both places. Cleanup: - adf-named-scaffolds.ts's TYPED_DATA_ACCESS_SCAFFOLD still contained the Stackbilt-internal specifics (registry repo path, AEGIS/CodeBeast names, issue links) that PR #245 removed from .ai/typed-data-access.adf -- running adf create typed-data-access would have reintroduced the leak in any new repo. Genericized to match, and updated the tests that asserted on the old content. - Added formatTriggerEntry() to packages/adf (inverse of parseTriggerEntry) as the single source of truth for the ON_DEMAND bullet format; adf-suggest --emit-ops now uses it instead of a hand-rolled duplicate serializer. - Bumped packages/adf and packages/cli to 1.8.0 (new public export buildTriggerReport/formatTriggerEntry, new adf suggest subcommand) per this repo's strict-semver convention (CLAUDE.md, minor for new features). 804/804 tests passing (803 + 1 new OSS-policy-leak regression test). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
typed-data-access.adfnamed Stackbilt-internal repo paths, service names, and issue links — removed, kept the generic sensitivity-tier pattern.buildTriggerReport()output (which ADF triggers matched/missed) is now recorded to.charter/telemetry/events.ndjsonalongside existing command events, joinable by session.charter adf suggest: new report-only command surfacing dead modules, recurring unmatched keywords, and load-but-violated correlations from that telemetry.--emit-ops: opt-in, human-gated detector proposing new trigger keywords based on co-occurrence with existing triggers; writesREPLACE_BULLETops (grouped per-module to avoid a clobbering bug found during manualadf patchtesting). Nothing is ever auto-applied. Known limitation (documented in code/CLI help): can propose low-signal candidates and mildly self-reinforces trigger-list growth on frequently-loaded modules — human review is the load-bearing safeguard.Real module-attribution (validate/drift/evidence recording which module's constraint they check) remains the better long-term investment if less human review toil is the goal; this heuristic is a stopgap, not a replacement.
Test plan
dist/bin.js adf bundle/adf contextagainst a scratch.charterdir--emit-opsoutput applies cleanly viaadf patchwithout clobbering co-located candidates🤖 Generated with Claude Code