Skip to content

context-engine-audit-response-swarm-plan.md

Latest commit

 

History

History
513 lines (336 loc) · 41.8 KB

context-engine-audit-response-swarm-plan.md

File metadata and controls

513 lines (336 loc) · 41.8 KB
Raw

Context Engine Audit Response — Swarm Plan

Status: Recommendation-only roadmap (no coding). Open-source enhancements to the application; AI providers may be vendor-managed or open-source. Produced via multi-agent GPT-5.4 review (ai-engineer, backend-architect, workflow-optimizer) and finalized with swarm-planner.

Scope

Deliver a dependency-aware, parallel-executable roadmap that closes the remaining application-level gaps in the Context Engine using open-source-friendly engineering improvements where possible, while explicitly acknowledging that MCP Resources/Prompts, /mcp HTTP transport, and scoped retrieval are already shipped and only require verification/consolidation. AI provider support is provider-agnostic: vendor-managed providers such as OpenAI are in scope.

In scope

  • Safety/runtime hardening (transport, content, cancellation, concurrency).
  • Workspace/index integrity (ignore files, watcher, persisted state, secret boundaries).
  • Retrieval quality verification, calibration, and polyglot expansion.
  • Provider contract freeze + multi-provider adapter architecture (OpenAI first, then additional compatible/local providers as needed).
  • Navigation/discoverability improvements (SCIP-first symbol graph).
  • Governance/docs/repo-root consolidation.

Out of scope

  • Any code changes in this session (recommendations only).
  • Net-new product features unrelated to retrieval, safety, provider architecture, navigation, or governance.
  • Re-implementing MCP Resources/Prompts, /mcp HTTP transport, scoped retrieval (already present).

Minimum viable result

A ranked, testable backlog where each phase has: entry criteria, exit criteria, measurable gate, rollback trigger, and owner surface. Phases 0–1.5 are the MVR floor; everything else is sequenced behind them.

Dependency map (DAG)

P0 (scope freeze, fixture pack)
  └─► P1 (safety/transport/runtime)
        └─► P1.5 (workspace/index integrity)
              └─► P2 (retrieval verify + calibrate + polyglot)
                    └─► P2.5 (provider contract freeze)
                          └─► P3 (multi-provider adapters)
                                └─► P4 (navigation + surface simplification)
                                      └─► P5 (governance/docs/root hygiene)

Strict blockers: P0 → P1 → P1.5 → P2 → P2.5 → P3. P4 and P5 may begin preparatory recommendation drafting in parallel with earlier phases, but their final acceptance waits for P3.

Parallelizable slices

The following recommendation workstreams can be drafted concurrently by independent subagents because they touch disjoint surfaces and depend only on P0 artifacts:

Slice Parallel with Rationale
S-A: Transport + HTTP security spec S-B, S-C src/http/ vs retrieval/runtime surfaces
S-B: Content safety (secret scrub + injection boundary) S-A, S-C src/reviewer/, provider call sites
S-C: Runtime cancellation + concurrency S-A, S-B src/runtime/ focus
S-D: Workspace/index integrity S-A, S-B, S-C src/internal/retrieval/, watcher
S-E: Retrieval verify/calibrate matrix S-F measurement only, no adapters
S-F: Polyglot chunking rollout plan S-E tree-sitter grammar selection
S-G: Provider contract document — needs S-E outputs
S-H: Multi-provider adapter matrix (OpenAI first, then compatible/local providers) — needs S-G frozen
S-I: Navigation (SCIP-first) + surface consolidation S-J src/mcp/tools/
S-J: Governance/docs/root hygiene S-I docs/, root layout

Each slice produces one recommendation artifact (markdown section or sub-doc) that can be reviewed independently before being folded into the final roadmap.

Task list

Phase 0 — Audit correction and scope freeze (blocking)

  • T0.1 Reclassify shipped items (MCP Resources/Prompts, /mcp, scoped retrieval) as "verify/consolidate".
  • T0.2 Freeze open-gap backlog; rank by (safety, quality, provider flexibility, DX).
  • T0.3 Freeze measurement baseline: multilingual fixture pack + benchmark normalization + per-phase exit criteria.
  • T0.4 Define success metrics per theme (recall@n/nDCG, deterministic safety coverage, surface reduction count).

Exit: Single corrected backlog + fixture pack locked; owners named; every later item tied to a measurable gate.

Phase 1 — Safety, transport, runtime hardening

  • T1.1 Freeze /mcp + /api/v1 security expectations (loopback/auth defaults, origin policy, size/time bounds, structured errors).
  • T1.2 Enforce secret scrubbing before index/review/provider calls.
  • T1.3 Treat retrieved content + external grounding as untrusted; define prompt-boundary handling.
  • T1.4 Propagate cancellation across retrieval, workers, providers, HTTP.
  • T1.5 Define bounded concurrency, shutdown, partial-write protection, fail-closed rules.

Exit: Threat-model checklist + deterministic tests specified; operator docs drafted; rollback defined.

Phase 1.5 — Workspace/index integrity

  • T1.5.1 Ignore-file correctness (.gitignore, .contextignore, generated-file exclusions) across index/watcher/refresh.
  • T1.5.2 Watcher churn correctness (rename/move/delete, stale cache, ordering).
  • T1.5.3 Persisted state integrity (compat, crash recovery, stale locks, no-regression).
  • T1.5.4 Secret-boundary handling in indexed content.

Exit: Deterministic churn tests specified; compat checks specified; no-regression rules documented.

Phase 2 — Retrieval quality foundation

  • T2.1 Polyglot chunking rollout (tree-sitter: Python → Go → Rust → Java → C# → broader).
  • T2.2 Embedding baseline calibration; enforce no-silent-downgrade from model-backed to hash.
  • T2.3 Reranker calibration and tune/replace decision.
  • T2.4 Extend eval with multilingual slices + adversarial checks + fixed pass/fail thresholds.
  • T2.5 Tie polyglot chunking to rebuild/migration/eval-reset rules.

Exit: Before/after report on fixed fixture pack; reproducible benchmark outputs; verification-before-replacement evidence.

Phase 2.5 — Provider contract and evaluation freeze

  • T2.5.1 Provider capability contract (flows, parity, error taxonomy, timeouts, cancellation, health).
  • T2.5.2 Privacy/network boundary: local-guaranteed vs opt-in external vs unsupported.
  • T2.5.3 Evaluation matrix: provider-specific gates + rollback triggers.

Exit: Contract doc + conformance matrix frozen; privacy boundary operator-readable.

Phase 3 — Multi-provider adapter architecture

  • T3.1 Generalize provider contract beyond openai_session.
  • T3.2 Harden OpenAI as the primary provider path (timeouts, cancellation, health/readiness, structured errors, model/config surface).
  • T3.3 Add the next provider path only where it adds clear value: generic OpenAI-compatible endpoint first, then optional local/self-hosted profiles (Ollama, llama.cpp server, LM Studio) if desired.
  • T3.4 Replace spawn-per-call with persistent HTTP/IPC where possible.
  • T3.5 Clarify provider policy in docs + .env.example: default provider, opt-in external endpoints, unsupported paths.

Exit: Provider matrix + startup/healthcheck coverage per supported backend; docs make the provider/privacy boundary obvious; no adapter ships without passing frozen gates.

Phase 4 — Navigation and discoverability

  • T4.1 Symbol graph: SCIP-first evaluation; LSP/ctags fallback only if simpler/sufficient.
  • T4.2 Consolidate overlapping search/review tools into canonical entry points.
  • T4.3 Align tools/prompts/resources/manifest with simplified surface.

Exit: Smaller public surface with no capability loss; updated manifest tests/snapshots specified; nav-query tests improved.

Phase 5 — Governance, docs, repo hygiene

  • T5.1 Docs reduction to canonical entry points; archive historical plans.
  • T5.2 Governance right-sizing (keep correctness/quality/compat gates; archive granular reporting).
  • T5.3 Repo-root hygiene (move tool/client/stateful artifacts out of root).

Exit: Short canonical docs map; reduced CI surface with preserved regression protection; cleaner root.

Validation and rollback

  • Every phase has: entry criteria (prior phase exit), exit criteria (testable), rollback trigger (what reverts), and ownership surface.
  • No silent downgrade: embedding/reranker fallbacks must be visible + logged.
  • Frozen fixture pack: retrieval-quality work cannot be accepted without before/after on the P0 fixture pack.
  • Contract-before-adapter: no P3 adapter merges before P2.5 contract freeze.
  • Archive-not-delete: governance cleanup archives historical artifacts; never deletes institutional memory outright.

Risks and tradeoffs

  • Polyglot chunking dependency load → mitigate with incremental rollout + fixture-gated acceptance.
  • Provider fragmentation → mitigate with strict contract + capability matrix.
  • Stronger models raising costs → mitigate with explicit quality/perf profiles + visible fallbacks.
  • Tool consolidation breaking workflows → mitigate with additive aliases + deprecation windows.
  • Governance cleanup losing context → mitigate by archiving, not deleting.
  • State/watcher/transport compat regressions → mitigate with explicit rollback + compat matrix + no-regression tests.
  • Privacy boundary blur from provider expansion → mitigate with frozen boundary contract before adapter rollout.

Rollout priority (capacity-constrained)

  1. Phase 0 → 2. Phase 1 → 3. Phase 1.5 → 4. Phase 2 → 5. Phase 2.5 → 6. Phase 3 → 7. Phase 4 → 8. Phase 5.

Execution handoff

This plan is executed via the parallel-task skill using the slice table above. Each slice is a bounded recommendation artifact (no code). A final consolidation pass (GPT-5.4) reviews the merged output before the roadmap is considered finalized.

Appendix A — Recommendations — Phase 1 / 1.5 (Safety, Transport, Runtime, Workspace Integrity)

Produced by parallel GPT-5.4 subagent (rec-safety), grounded in direct inspection of src/http/, src/reactive/guardrails/, src/runtime/, src/watcher/, src/mcp/, and existing tests.

S-A — Transport + HTTP security spec

Current state: /mcp uses @modelcontextprotocol/sdk streamable HTTP; /api/v1 exists with CORS limited to VS Code webviews + loopback; MCP auth is an optional authHook; errors are minimal {error,statusCode}.

Gaps: no enforced loopback bind (app.listen(port) binds broadly); no express.json size limit; no explicit request/headers/idle timeouts or slowloris policy; no rate limiting; error envelopes not a frozen taxonomy; CORS credentials: true without cookie-auth need.

OSS recommendations:

  1. Default-bind 127.0.0.1/::1; honor CE_HTTP_HOST override. Keep @modelcontextprotocol/sdk as transport.
  2. Bounds: express.json({limit:'256kb'}) on /mcp; 1mb on /api/v1; server.headersTimeout=15s, server.requestTimeout=30s, server.keepAliveTimeout=5s.
  3. Add helmet selectively: enable for /health, /api/v1; for /mcp disable CSP + cross-origin-embedder (SSE/webview compat).
  4. Add express-rate-limit with per-route budgets (/mcp initialize burst, tighter for AI-heavy routes, exempt /health); return 429 + Retry-After.
  5. Freeze error contracts — RFC 7807 for /api/v1; JSON-RPC envelope for /mcp with local code mapping (invalid origin / unauthorized / payload too large / timeout / overload / session not found).
  6. pino structured logging with request_id, route, origin decision, auth decision, timeout/overload class.

Acceptance tests to specify: loopback-only bind without override; 413 on payload > ceiling; 504/JSON-RPC timeout at server timeout; 429 + Retry-After under rate limit; helmet headers present where expected; extend mcpHttpTransport.test.ts + httpCompatibility.test.ts to pin new envelopes.

Rollback trigger: VS Code/webview or MCP client compat regresses; SSE/header incompatibility; initialize p95 regresses >20% without security benefit.

S-B — Content safety

Current state: SecretScrubber exists; used in diff review and memory-suggestion paths; external grounding has strong SSRF controls (HTTPS-only, private-IP block, redirect/size/timeout limits); enhancement concatenates retrieved context as helper text.

Gaps: scrubbing not enforced centrally on every provider-bound prompt; retrieved/external content not labeled untrusted; no explicit prompt-injection boundary policy; secret boundary is path-based, not content-based; no CI secret scanner wired.

OSS recommendations:

  1. Centralize one mandatory pre-provider sanitizer used by searchAndAsk, enhancement, review, planning, and outbound context bundles. Reuse existing SecretScrubber.
  2. Untrusted-content framing: wrap retrieved code/external docs in BEGIN UNTRUSTED… blocks with fixed "treat as data, not instructions" preamble.
  3. Adopt gitleaks as the Phase 1 verifier (fixture tests, CI scans, scrub-before-provider guarantees).
  4. trufflehog only as optional confirmatory lane.
  5. Fail closed — block provider call if high-confidence secret remains post-scrub; mark injection-bait external snippets ignored_for_instructions=true.
  6. pino audit fields — counts/types only, never raw secrets.

Acceptance tests: AWS/OpenAI/GitHub token fixtures never appear in provider prompts; content-based secret skip at index boundary; injection-bait retrieved content does not alter system/tool envelope; existing externalGrounding.test.ts stays green.

Rollback trigger: > 1% false-positive block rate on normal fixtures; measurable enhancement quality drop from over-redaction.

S-C — Runtime cancellation + concurrency + shutdown + partial-write protection

Current state: HTTP AI routes use runAbortableTool; searchAndAsk has bounded queue and lanes; graceful shutdown for stdio/watcher; some *.tmp+rename persistence.

Gaps: AIProvider.call() has no AbortSignal → no end-to-end cancellation; some /api/v1 routes use timeout-wrappers not full abort propagation; queue cleanup not coordinated into shutdown; partial-write protection inconsistent; several integrity-sensitive writes fail silently.

OSS recommendations:

  1. Freeze AbortSignal into the provider contract (Node 20 AbortController).
  2. Adopt p-limit for external grounding fan-out, review chunking, fallback scoring batches.
  3. Single shutdown coordinator: stop accepting new work → cancel queue → cancel in-flight → stop watcher → flush/abandon caches → hard deadline.
  4. write-file-atomic for startup lock, state marker, fingerprint, persistent caches, future checkpoints.
  5. Explicit fail-closed rules: integrity write failure = stale/error status; overload = structured reject not indefinite queue; aborted work must not publish partial artifacts.

Acceptance tests: abort-to-provider kill < 500ms; SIGINT/SIGTERM drains or rejects with retry guidance; saturated queue returns deterministic 429/503; crash during write never yields truncated accepted file; cancellation consistent across index/search/context/enhance/review.

Rollback trigger: hung shutdowns; spurious provider failures under normal load; throughput regression without overload improvement.

S-D — Workspace/index integrity

Current state: .gitignore + .contextignore loaded; watcher normalization + batching; discoverFiles recursive walk; index-state compat checks (schema/provider/fingerprint/feature-flags); workspace startup lock with stale recovery.

Gaps: ignore semantics are not full .gitignore (no negation, custom matching); discovery and watcher use different rule stacks; no deterministic rename/move contract; persisted recovery mixed; no orphan .tmp recovery policy; secret boundary still path-based.

OSS recommendations:

  1. Replace custom ignore parsing with ignore (real .gitignore semantics including negation); share one compiled engine across discovery/watcher/refresh/scoped-refresh.
  2. Replace manual recursive walk with fast-glob using shared ignore rules + dotfile policy.
  3. Keep chokidar; freeze rename/move as ordered unlink(old)+add(new) with post-pair coherence contract.
  4. write-file-atomic across .context-engine-context-state.json, fingerprint, persistent caches, retained locks.
  5. Content-based secret boundary check before persistence (SecretScrubber + gitleaks fixtures) — block indexing even when path passes.
  6. pino integrity telemetry (normalized event sequences, stale-cache repair, schema reset, stale-lock recovery).

Acceptance tests: ignore parity between discovery and watcher on root-anchored / negation / directory-only / wildcard fixtures; deterministic rename/move/delete-recreate/rapid-save behavior; crash during write → previous valid artifact or safe reset; corrupt-lock-file recovery; .ts file with secret is excluded + deterministic event.

Rollback trigger: ignore-rule parity breaks current indexing expectations; watcher churn causes missed/duplicate updates; frequent false stale-state resets.

Recommended implementation order inside Phase 1/1.5: S-A → S-C → S-B → S-D (expose less → cancel correctly → sanitize consistently → make workspace deterministic).

Appendix B — Recommendations — Phase 2 (Retrieval Quality: Verify, Calibrate, Polyglot)

Produced by parallel GPT-5.4 subagent (rec-retrieval). Governing rule: verify before replace.

S-E — Retrieval verify/calibrate matrix

Current state: Embedding runtime tracks configured-vs-active-runtime + degradation signals but fails open to hash (src/internal/retrieval/embeddingRuntime.ts); reranker gated with fail-open metadata (rankingCalibration.ts, retrieve.ts); quality profile enables dense + transformer embeddings but not tree-sitter or cross-encoder reranker; deterministic holdout/quality gate with dataset/fixture hash + provenance exists (scripts/ci/check-retrieval-holdout-fixture.ts, bench-provenance.ts, bench-compare.ts); current fixture pack is 5 cases, English-heavy (config/ci/retrieval-quality-fixture-pack.json).

Gaps: several latency/resource gates are literals, not measured; generate-retrieval-quality-telemetry.ts is deterministic scaffolding not runtime telemetry; benchmark normalization only partial (no multilingual, no macro-avg); no multilingual fixture composition; "no silent downgrade" only partially enforced.

OSS recommendations:

Verification matrix (measure before replace):

Layer Keep Compare against
Embeddings Xenova/all-MiniLM-L6-v2 via @huggingface/transformers BAAI/bge-small-en-v1.5, bge-base-en-v1.5, thenlper/gte-small, intfloat/e5-small-v2, one multilingual slot (BAAI/bge-m3 or multilingual-e5-small)
Reranker cross-encoder/ms-marco-MiniLM-L6-v2 BAAI/bge-reranker-base, OSS Jina rerankers
Runtime in-product @huggingface/transformers / ONNX fastembed as offline sweep only; Ollama optional smoke

Measure: P@1, MRR@10, Recall@{5,10}, nDCG@10; blank-result / timeout / error rates; embedding fallback rate, rerank fail-open rate, active≠configured rate; rerank invocation rate / win rate / latency; p50/p95 per profile; RSS growth; normalization receipts (dataset / fixture-pack / workspace / index / feature-flag hashes).

No-silent-downgrade rules: transformer→hash fallback = CI fail in quality; rerank fail-open = artifact/gate degraded; any calibration-run fallback = fail (not warn); fast/balanced may fail open for availability but never silently.

Multilingual fixture pack: keep repo-local pack for PR gate; add Phase 2 frozen pack — 6 code languages × 6+ NL locales, per code-language slice = 20 cases (8 semantic + 4 identifier + 4 relationship + 2 paraphrase + 2 adversarial) + 24 cross-lingual overlays. Nightly calibration pack from CodeSearchNet / MTEB code subsets / CSN-WikiSQL / BEIR code slices (report-only until stable).

Normalization: repo-relative path only; dedupe by retrieved path; same k=10; Unicode-aware query normalization; macro-avg by language slice first then overall.

Fixed Phase 2 gates: internal PR pack — nDCG@10 ≥ 0.78, MRR@10 ≥ 0.75, Recall@10 ≥ 0.90, P@1 ≥ 0.60. Polyglot macro — nDCG@10 ≥ 0.70, Recall@10 ≥ 0.85, P@1 ≥ 0.50, worst-language Recall@10 ≥ 0.60. Calibration fallback rate = 0%; rerank fail-open ≤ 1%. Latency p95 — fast ≤ 350ms / balanced ≤ 700ms / rich ≤ 1200ms; rerank p95 overhead ≤ 250ms; RSS growth ≤ 25%.

Rollback trigger: active embedding runtime falls back in quality; rerank fail-open > 1%; provenance/hash mismatch; any fixed gate misses 2 consecutive runs; p95 regresses > 20% vs approved baseline.

Dependency notes: extend existing provenance (bench-provenance.ts, bench-compare.ts); replace literal latency checks with measured values; treat approved_baseline_report_path as actual comparison source; do not replace current embedding/rerank default until verified stack misses fixed gates.

S-F — Polyglot chunking rollout

Current state: Heuristic boundary chunker (chunking.ts) + tree-sitter for TS/JS/TSX/JSX only; unsupported silently falls back; chunk index ties cache validity to parser id/version + chunking params; dense/vector indexes rebuild on embedding/model/dim change.

Gaps: no polyglot grammar registry; no parser-coverage metric; current tree-sitter fallback too permissive; no explicit grammar→re-chunk→re-embed→eval-reset coupling.

OSS recommendations: Core = tree-sitter (native, server-first; web-tree-sitter only if native build friction material). Grammars — tree-sitter-python → -go → -rust → -java → -c-sharp → broader. Never mix native + WASM outputs under same parser id. Per-language enablement: add grammar → map declaration/import/class/function node kinds → collect heuristic baseline → side-by-side chunk diff + retrieval eval → enable only if slice passes gates.

Per-language rollout gates: tree-sitter coverage ≥ 95%, heuristic fallback on supported ≤ 5%, target-language Recall@10 ≥ 0.80 / nDCG@10 ≥ 0.68 / P@1 ≥ 0.45; regression vs heuristic baseline no worse than -2%; chunk count drift ≤ 35% unless approved; added p95 retrieval ≤ +15%.

Existing-language safety gate: Recall@10 / nDCG@10 / P@1 each regress ≤ 2%.

Rollback trigger: coverage < 90%; supported-language heuristic fallback > 10%; chunk drift exceeds bounds without quality win; rebuild/latency regresses > 20%; enabled language misses frozen gate.

Migration coupling: grammar add/change / declaration-mapping change → parser-registry version bump → chunk-index invalidation → full re-chunk → full dense/LanceDB re-embed → reset approved baseline for affected slices. Grammar version bump only → shadow diff first. Embedding change without chunk change → dense/vector rebuild only. Reranker-only change → no re-chunk / no re-embed but full eval rerun required.

Appendix C — Recommendations — Phase 2.5 / 3 (Provider Contract + Multi-Provider Adapters)

Produced by parallel GPT-5.4 subagent (rec-providers).

Evidence snapshot: provider is single-id text-only (AIProviderId='openai_session', returns {text,model}); factory rejects non-Codex; only impl is Codex CLI spawn-per-call via temp file; queue abort works only while waiting — no AbortSignal in provider contract; offline-only mode blocks the sole AI provider; .env.example + docs are Codex-only.

S-G — Provider capability contract freeze

Gaps: no frozen capability vocabulary; no structured privacy boundary; no in-flight cancellation; no portable health/readiness; no stable error taxonomy for HTTP backends; no "capability absent vs silent fallback" rule.

Recommendations (do not ship any adapter before freeze):

Area Recommendation
Identity stable providerId, backendFamily, model, transport
Capabilities generate, structuredJson, streaming, toolCalls, embeddings, health, readiness, localGuaranteed
Core methods generate(req), health(), readiness(), dispose(); embed() only if declared
Request timeoutMs, absolute deadline, AbortSignal, response mode (text|json|tools), model override, workspace metadata
Response text|json|toolCalls, model, finishReason, usage?, latencyMs, warnings, privacy class
Validation zod for config / capability / response
HTTP stack undici / native fetch (pooled); ofetch acceptable for wrappers
OpenAI-compat @openai/openai SDK with custom baseURL

Frozen policy rules: capabilities declarative (no silent emulation); embeddings optional not implied (absent → keep local embedding runtime); health ≠ readiness (readiness verifies target model); cancellation end-to-end (queue + in-flight HTTP abort); privacy class operator-readable at startup + docs.

Privacy classes: local_guaranteed (loopback/unix/pipe only, no DNS/WAN egress, local model); external_opt_in (any non-loopback, requires explicit operator policy); unsupported (spawn-per-call CLI, ambiguous relay, missing contract surfaces).

Evaluation gates: prompt fidelity (critical — searchAndAsk structured results still parseable); streaming parity (conditional — chunk ordering, final-assembly equivalence, abort latency); tool-call parity (conditional — schema round-trip, deterministic errors); embeddings parity (conditional — dimension / batch / timeout, holdout impact); timeout/cancellation behavior (critical — distinguish queue-timeout / connect-timeout / read-timeout / provider-timeout / cancellation).

Rollback trigger: adapter needs provider-specific exceptions to core semantics; local_guaranteed cannot be machine-determined; in-flight cancellation cannot be enforced; embeddings/tool-calls need silent fallback to appear supported; error mapping collapses distinct failures into unknown.

Docs deliverables: provider contract doc (capability/envelope/error taxonomy); privacy boundary matrix; operator startup surface (id / baseURL / privacy class / health+readiness / selected model); migration note replacing "OpenAI-only / drop Ollama" copy in docs/updates-openai-roadmap-summary.md + .env.example.

S-H — Multi-provider adapter rollout order

Rollout order:

  1. OpenAI first — make the existing provider path production-grade under the frozen contract. Cover timeout/deadline semantics, in-flight cancellation, health/readiness, structured errors, operator-visible model/config, and removal of implicit spawn-per-call assumptions where possible.
  2. Generic OpenAI-compatible second — reuse the same contract for any endpoint speaking the OpenAI-style API. This covers vLLM, LocalAI, text-generation-webui (OpenAI ext.), LM Studio, and llama.cpp when compatible. Impl: @openai/openai SDK + custom baseURL; undici/native fetch for health + streaming; zod validation. Health/readiness: /v1/models + model-present check.
  3. Native/local profiles only if needed — add Ollama native, llama.cpp server-specific, or LM Studio shims only where the generic compatible path is insufficient or where local-only/privacy requirements justify the extra maintenance.

Per-adapter gates: prompt fidelity match; streaming final-text parity + mid-stream abort; tool-call disabled unless contract tests pass; embeddings only advertised if dim/batch/timeout pass; distinct connect/read/deadline/cancel outcomes; local_guaranteed path proves zero remote egress; no false-green when endpoint up but model unavailable.

Rollback trigger: local_guaranteed config still routes externally; readiness passes while required model absent; in-flight abort leaks a live request; structured-output break; embeddings advertised but parity-fail or silent-fallback; adapter needs spawn-per-call wrapper.

Docs deliverables: one canonical matrix (provider/profile → transport → privacy class → health probe → embeddings support → streaming/tool-call); copy-paste env for each target; troubleshooting page ("why this config is external_opt_in", "why readiness failed", "how model-loaded checks work"); document OpenAI as the default path unless operators opt into another backend.

Bottom line: Freeze contract + privacy classes + in-flight cancellation first. Then harden OpenAI → add generic OpenAI-compatible support → add native/local profiles only when they solve a real need. Persistent HTTP/IPC over spawn-per-call where possible. Capabilities never implied.

Appendix D — Recommendations — Phase 4 / 5 (Navigation, Discoverability, Governance, Repo Hygiene)

Produced by parallel GPT-5.4 subagent (rec-ux).

Evidence snapshot: tool_manifest advertises 43 tools / 17 capability groups / 4 prompts / 3 resource kinds; tool registration centralized in src/mcp/server.ts; discoverability metadata cross-links overlapping search + review tools; navigation is retrieval/chunk-first (no SCIP/LSP/ctags in src/); 131 active Markdown files in docs/, 66 already archived, 9 root .md files plus debug/state artifacts; CI spread across 3 workflows with duplicated install/build blocks.

S-I — Navigation / discoverability / simpler MCP surface

Gaps: no symbol graph — exact symbol/file→symbol/reference nav depends on retrieval + file reads; too many peer entry points; manifest doesn't distinguish recommended / advanced / alias; search+review overlap increases onboarding cost.

OSS recommendations:

Make navigation SCIP-first internally: order = scip-typescript first → scip-python / scip-java / scip-go as workspace mix demands → scip-ctags + universal-ctags for uncovered types → keep tree-sitter for chunking fallback → avoid LSP unless materially simpler than shipping the SCIP indexer.

Consolidate to canonical entry points (keep compatibility, simplify what's promoted):

Area Canonical Keep but demote
Find code codebase_retrieval semantic_search
Read exact get_file —
Assemble context get_context_for_prompt enhance_prompt as advanced
Default review review_auto review_changes, review_git_diff
Deterministic/CI review review_diff check_invariants, run_static_analysis as expert subtools
Long-running review reactive_review_pr status/pause/resume/telemetry as session subtools

Simplify manifest without capability loss: expose one view with recommended_tools / advanced_tools / compat_aliases / deprecated_tools. Prompts/resources link to canonical tools only. Keep resource kinds at 3 until symbol graph stabilizes.

Do not add a new nav tool first. Use SCIP data to improve ranking/context inside codebase_retrieval + get_context_for_prompt + optional get_file follow-up suggestions. Only add a nav-specific MCP surface later if existing tools cannot express the workflow cleanly.

Acceptance gates: promoted search+review entry points reduced from 12+ overlapping → 6 canonical; manifest exposes ≤ 10 recommended tools while aliases remain callable; exact-symbol + exact-file hit rate improves on fixed nav query set; discoverability.test.ts and manifest snapshots stay green.

Rollback trigger: SCIP indexing raises index time/state size without nav accuracy improvement; client discoverability parity loss; review/search success drops because demoted tools became harder to reach.

Deprecation window: ≥ 2 releases or 90 days; demoted tools remain callable as aliases; default docs/manifest views hide from first-time users; manifest marks alias_of / deprecated_after / replacement; remove only after smoke + usage evidence show no active dependency.

S-J — Governance / docs / repo-root hygiene

Gaps: no canonical docs map; blurry active-vs-archive boundary; root mixes source-of-truth docs with transient/generated/local artifacts; CI includes some governance/reporting gates too granular for required PR lane.

OSS recommendations:

Default: plain GitHub README consolidation now (not a docs site yet — lowest maintenance, best fit for engineering-heavy Markdown-native repo). If later needed: MkDocs Material (skip Docusaurus unless product-marketing/blog/versioned-site behavior is needed).

Target canonical docs map: README.md / ARCHITECTURE.md / CHANGELOG.md / docs/README.md (map) / docs/MCP_CLIENT_SETUP.md / docs/WINDOWS_DEPLOYMENT_GUIDE.md / docs/MEMORY_OPERATIONS_RUNBOOK.md / docs/BENCHMARKING.md / docs/REVIEW_CONTRACTS.md or consolidated docs/OPERATIONS.md / docs/archive/INDEX.md. Everything else merged or archived.

CI right-sizing (reuse existing GitHub Actions): one required PR workflow (correctness + compat + review); one perf workflow with PR-smoke + nightly-full + manual release dispatch (fold release_perf_gate.yml into perf_gates.yml). Required PR lane keeps: build/setup, contract/compat, MCP smoke, deterministic review_diff, critical SLOs. Move to nightly/manual: governance artifact completeness, rollout-readiness reporting, rollback-drill evidence, report-generation heavy gates.

Move root artifacts (archive, don't delete):

Current root Destination
agents-debug-*.log artifacts/logs/agents/
latest artifacts/pointers/latest
.context-engine-*.json, .context-engine-startup.lock .context-engine/state/
.augment-*.json, .augment-plans/ .context-engine/external/augment/
root *-swarm-plan.md, vibe-coder-memory-mode-plan.md docs/archive/swarm-plans/
read/ working-note docs docs/archive/working-notes/read/
plan/ working-note docs docs/archive/working-notes/plan/
.claude/, .gemini/ (if repo-owned examples) examples/mcp-clients/{claude,gemini}/ or stop tracking

Acceptance gates: active docs ≤ 10 pages; root .md reduced from 9 → canonical core; root debug/state artifacts = 0; every moved doc reachable from docs/archive/INDEX.md; ≤ 2 workflow files, ≤ 2 required PR jobs.

Rollback trigger: onboarding worse because primary docs harder to find; inbound links break without stubs; required CI loses regression-catching power; CI duration/regression signal degrades materially.

Deprecation window: old doc paths → stub/redirect pages for 2 releases or 90 days; old workflow names → wrappers/manual shims for one release; mirror old artifact paths briefly; archive is long-term location. Archive — never delete — institutional memory.

Consolidation status

All four parallel recommendation artifacts (S-A/B/C/D, S-E/F, S-G/H, S-I/J) have been produced by independent GPT-5.4 subagents grounded in direct repo inspection, and merged into this document. A final GPT-5.4 review pass followed — see Appendix E.

Appendix E — Final Review Consolidation (GPT-5.4, 3 reviewers)

Three independent GPT-5.4 reviewers (ai-engineer, backend-architect, workflow-optimizer) returned "ship with fixes". The following fixes are now part of the roadmap and override any softer wording in earlier sections.

E.1 Transport / SSE compatibility (fixes Appendix A § S-A)

  • /mcp MUST be exempt from the aggressive connection timeouts proposed for /api/v1. Required values: for /mcp SSE/long-lived GET streams, server.keepAliveTimeout ≥ 65s, no per-request server.requestTimeout (or set to 0), server.headersTimeout ≥ 70s. /api/v1 retains the proposed 15/30/5 seconds.
  • Rate limiting on /mcp MUST be session-aware: initialize/reconnect gets its own bucket; in-session JSON-RPC calls are NOT rate-limited in a way that interrupts an open stream.
  • helmet policy on /mcp MUST disable CSP, crossOriginEmbedderPolicy, and crossOriginResourcePolicy to preserve SSE + webview compatibility.

E.2 End-to-end cancellation during migration (fixes Appendix A § S-C + Appendix C § S-G)

  • Temporary rule for the Phase 3 migration window: any spawn-per-call provider that remains live (including current Codex/openai_session path) MUST support kill-on-abort and kill-on-shutdown. A provider that cannot propagate cancellation to its subprocess MUST be marked unsupported in the frozen contract and removed from the default configuration, not left as a silent exception.
  • Phase 1 exit is NOT met until end-to-end abort is demonstrated against the currently-active provider, not just the future OSS adapters.

E.3 Windows path contract (fixes Appendix A § S-D)

The unified ignore/discovery/watcher engine MUST freeze:

  • Case-insensitive matching on Windows (case-sensitive on POSIX).
  • Drive-letter normalization (canonical uppercase drive letter).
  • Path separator normalization to forward slashes internally; backslash preserved only at OS-boundary calls.
  • UNC / \\?\ long-path support (Windows paths > 260 chars), including in persisted-state keys.

Parity tests between fast-glob, chokidar, and persisted state MUST use the same normalized form.

E.4 Multi-tenant / concurrent-request safety (fixes Appendix C § S-G)

Add to the frozen provider contract:

  • Per-request auth/header isolation (no shared mutable auth state).
  • Declared maxInFlight concurrency + backpressure behavior.
  • Connection-pool and socket-reuse limits.
  • Model-selection isolation between simultaneous requests.
  • Circuit-breaker / brownout semantics (half-open probes, failure window, cool-down).

Adapter shipping gate includes a soak test proving these under concurrent load.

E.5 Retrieval candidate specificity (fixes Appendix B § S-E)

Replace the generic phrase "OSS Jina rerankers" with exact open-weight model IDs from the candidate set at evaluation time (e.g. jinaai/jina-reranker-v1-tiny-en, jinaai/jina-reranker-v1-turbo-en, or successor OSS-weighted releases). Drop the entry if no current Jina release ships open weights; BAAI/bge-reranker-base and cross-encoder/ms-marco-MiniLM-L6-v2 remain the primary comparators.

E.6 Measurable rollback triggers (fixes multiple appendices)

Replace vague language with concrete thresholds:

Original (vague) Replacement (measurable)
"initialize p95 regresses > 20% without security benefit" "/mcp initialize p95 regresses > 20% vs approved baseline across ≥ 50 samples AND no critical CVE mitigation is attributable to the change"
"measurable enhancement quality drop from over-redaction" "enhancement pack MRR@10 regresses > 3% OR user-visible 'empty context' rate rises > 5% vs approved baseline"
"spurious provider failures under normal load" "provider-initiated failure rate > 1% over a rolling 500-request window at ≤ default concurrency"
"client discoverability parity loss" "any pinned discoverability snapshot test fails OR a removed tool lacks a resolvable alias"
"onboarding worse" "time-to-first-successful-tool-call in a fresh-clone smoke drill increases > 25%"
"regression signal degrades materially" "required-lane catch rate on the canary failure-corpus drops below 95% of pre-consolidation rate"

E.7 Phase-exit cross-phase receipts (fixes phase-level coupling)

Each phase's exit now requires the following linked receipts, not just local acceptance:

  • P1 exit ⇒ end-to-end abort evidence for the live provider (used by P2.5/P3 contract).
  • P2 exit ⇒ no-silent-downgrade evidence (used by P3 adapter gates).
  • P2.5 exit ⇒ frozen contract document referenced by every P3 adapter PR.
  • P3 exit ⇒ each adapter carries a link back to the P2 fixture-pack run that validated its embeddings/rerank parity.
  • P4 exit ⇒ SCIP-backed nav quality delta measured on the same Phase 2 fixture pack used for retrieval, not a new pack.

A phase cannot "pass locally" without the downstream consumer phase confirming receipt validity.

E.8 Governance right-sizing correction (fixes Appendix D § S-J)

  • Governance artifact completeness, rollout-readiness reporting, and rollback-drill evidence MUST remain required on the release / tag lane, even if removed from the PR lane.
  • The "≤ 2 required PR jobs" constraint MUST explicitly preserve: Phase 1 transport/cancellation checks, Phase 2 retrieval no-silent-downgrade checks, and Phase 2.5/3 provider conformance checks. CI consolidation may merge these into fewer jobs but may not drop them.
  • A "required-lane catch rate on canary failure-corpus" telemetry must be produced so consolidation does not silently regress regression protection (tie-in with E.6).

E.9 Deprecation usage-evidence gate (fixes Appendix D § S-I / S-J)

Before final removal of any demoted tool, deprecated doc path, or legacy workflow:

  • Dependency/usage evidence MUST be gathered (access logs, telemetry, or grep-in-known-consumers).
  • Removal is blocked if any active dependency is observed.
  • Window extension is the default response; outright removal requires documented zero-usage evidence.

E.10 Fixture-pack PR-vs-nightly disambiguation (fixes Phase 0 + Appendix B § S-E)

  • PR-blocking pack: the existing small in-repo fixture pack (≤ 10 cases), covering TS/JS primary slices only. Remains the required gate on PR CI. Keeps PR latency bounded.
  • Phase 2 multilingual/polyglot pack (~144 cases): runs on nightly and release lanes, NOT on every PR. A per-PR "multilingual smoke" (≤ 15 representative cases) runs when retrieval/chunking code paths are touched.
  • The phrase "frozen fixture pack" at the phase-level refers to both — the PR pack is frozen for PR gating, the multilingual pack is frozen for release gating.

E.11 P4 ↔ Phase 2 coupling (fixes DAG)

P4 (navigation / SCIP-first) is now shown as coupled to the Phase 2 fixture pack and nav-query test set, not only to P3 completion. SCIP rollout changes retrieval behavior and MUST re-run the Phase 2 gates with the SCIP-backed path enabled. DAG update:

P3 ─┐
    ├─► P4 (requires P2 fixture-pack re-run with SCIP enabled)
P2 ─┘

E.12 Owner surface (fixes phase-level gap noted by ai-engineer reviewer)

Every phase MUST carry an owner field in the execution tracker (not duplicated into prose here). Owner surface is "the code area a single reviewer is accountable for," e.g. src/http/ for S-A, src/ai/providers/ for S-G/H, src/internal/retrieval/ for S-E/F. Phase exit review cannot be approved without a named owner.

Final verdict after incorporating Appendix E fixes: ship as-is as a recommendation-only roadmap. Implementation work against this roadmap remains out of scope for this session per the original "no coding" constraint.