ax will solve MCP and plugin visibility across agent harnesses

[Necmttn]

Problem

MCP and plugin failures are often invisible at the exact layer users need to debug. A server can be configured but unauthenticated, authenticated but not initialized, initialized but missing tools, schema-accepted but not injected into the thread, or injected but not visible to the model. Current harness surfaces usually collapse those states into "MCP broken" or "tool missing".

Ax should keep a provider-neutral inventory timeline so users and agents can answer why a capability was or was not available during a run.

Ax feature

Build an MCP/plugin capability inventory that records:

• configured, installed, enabled, authenticated, initialized, tools-listed, schema-accepted, callable, current-thread, and model-visible states
• hidden/disabled reason, last error, source config, resolved command, cwd/env class, and plugin root/source
• thread/run visibility snapshots so historical runs are explainable
• MCP telemetry boundaries: metadata always queryable, prompt/response payloads opt-in
• local debug/export view for governed inspection without uploading private content

Public threads this should help

• [Windows] Bundled Browser/Chrome/Computer Use plugins enabled in config but not exposed as callable tools openai/codex#26133
• Codex Desktop authenticates OAuth MCP server but never imports tools into threads; auth_status stays unsupported openai/codex#20009
• Codex CLI Ignores Available MCP Servers Until Manually Prompted openai/codex#29146
• feat(mcp): respect TRACELOOP_TRACE_CONTENT=false in standard MCP instrumentor traceloop/openllmetry#4189
• Plugin PostToolUse hook relative command resolves from workspace cwd openai/codex#26675
• RFC: DynamoDB MCP Server Telemetry Collection awslabs/mcp#2382
• [Bug]: /context detail shows ~62k untracked provider/runtime overhead on fresh OpenAI Codex gpt-5.5 session openclaw/openclaw#86819

Acceptance criteria

• Ax can show a per-run MCP/plugin capability matrix with one row per server/tool/plugin.
• Missing tools have a precise unavailable state, not a generic absent state.
• Historical sessions preserve the capability inventory that was visible at the time.
• Plugin command failures include source, resolved command, cwd, and safe env diagnostics.
• Payload capture remains opt-in while metadata remains useful by default.

Generated with ax - https://github.com/Necmttn/ax

[Necmttn]

Additional public threads from the outreach run that point at the same MCP/plugin visibility gap:

• SEP-2787: Tool call attestation modelcontextprotocol/modelcontextprotocol#2787 - correlation handles for session, turn, tool call, and parent/child lineage.
• Evidence-gated context retrieval planner for agent handoffs donovan-yohan/relay-ide#796 - evidence items need capability state and explicit gap kinds for unavailable provider events.
• PreToolUse hooks do not fire for the internal patch/edit tool openai/codex#26733 - expose raw tool names plus normalized file-write/edit capabilities for hook matchers.
• Add support for Agent hooks microsoft/copilot-intellij-feedback#1653 - hook parity matrix for discovery path, event naming, payload shape, and response effects.
• [FEATURE]: MCP tool calls should carry current session context (session_id) anomalyco/opencode#33035 - stable session id plus per-call MCP provenance fields in _meta.
• [tpp][codex] pass multi-agent metadata to MCP tools call openai/codex#27495 - versioned allowlist for MCP turn metadata and leakage tests.
• [FR] Environment reproducibility for AI coding agent traces mlflow/mlflow#21584 - runtime context detection through installed inventory and loaded-vs-invoked skills.
• Collect AI agent skill-usage telemetry for Aspire skills and MCP tools microsoft/aspire#18008 - skill_loaded vs skill_invocation, skill-relative ids, and hook telemetry dedupe.
• 🗺️ [agents] tracing skills and tools Arize-ai/phoenix#13570 - trace skills as available/loaded/applied with provenance links.
• PostToolUse hook cockroachdb/claude-plugin#23 - plugin launcher path diagnostics for unresolved ${CLAUDE_PLUGIN_ROOT}.
• [BUG] MCP tools connected but not exposed to agent anomalyco/opencode#33027 - staged MCP visibility diagnostics from connected through agent-visible/invocation-allowed.
• MCP startup should not block tool listing or thread startup openai/codex#29321 - MCP discovery as degraded capability inventory with freshness, startup status, and timeout/error reason.

Generated with ax - https://github.com/Necmttn/ax

[Necmttn]

Implementation plan

Proposed shape: build this as a provider-neutral capability inventory, not as an MCP-only special case. Treat MCP servers, MCP tools, plugins, plugin-bundled tools/skills/hooks, and harness-native tools as capability subjects with observations over time. The read model should answer: "what was configured, what stage did it reach, why did it stop, and was it visible in this session/thread?"

Existing seams to reuse:

• Config/artifact inventory: apps/axctl/src/ingest/claude-config.ts already writes metadata-only guidance_config_artifact rows for Claude settings, .mcp.json, plugin manifests, plugin hooks, plugin skills, and safe env-key names.
• Provider/session backbone: apps/axctl/src/ingest/provider-events.ts, agent_session, agent_event, normalized session/turn/tool_call.
• Runtime telemetry: apps/axctl/src/otel/*, especially metadata-only otel_log_event.attrs, plus queries/harness-run-context.ts / queries/harness-tool-events.ts projections.
• Provider parity bookkeeping: apps/axctl/src/ingest/provider-parity.ts + scripts/check-provider-parity.ts.
• Read surfaces to extend: CLI command modules under apps/axctl/src/cli/commands/, MCP registry in apps/axctl/src/mcp/tools.ts, dashboard/server contracts, and Studio routes.

Data model proposal:

• capability_subject: stable identity for one capability. Fields: provider, kind (mcp_server | mcp_tool | plugin | plugin_tool | skill | hook | native_tool), name, namespace, source_kind, source_ref, plugin_id, plugin_root_safe, content_hash, created_at, updated_at.
• capability_observation: append/upserted observation of a subject at a point in time. Fields: subject, provider, session?, agent_session?, observed_at, source_config?, resolved_command_safe?, cwd_safe?, env_keys_json, env_class, hidden_reason?, disabled_reason?, last_error_excerpt?, payload_policy, and explicit stage fields for configured, installed, enabled, authenticated, initialized, tools_listed, schema_accepted, callable, current_thread, model_visible with values like yes | no | unknown | n/a.
• capability_edge relation: server/plugin exposes tool/skill/hook; keeps matrix rows explainable without duplicating parent metadata.
• Optional later capability_payload_capture: opt-in payload pointer only, never default. Metadata rows stay useful without prompt/response bodies.

Vertical slices:

1. Config-only inventory baseline. Project existing guidance_config_artifact rows into capability subjects/observations for Claude MCP/plugin config, metadata-only. Add provider-parity row showing Claude supported and other harnesses as explicit gaps. No payload capture.
2. Cross-provider config scanners. Add config artifact discovery for Codex/OpenCode/Cursor/Pi where local config is available, following the same safe-path/env-key discipline as claude-config.ts.
3. Runtime observations. Use provider events and OTLP attrs to move capabilities from configured to initialized/tools-listed/callable where signals exist. Missing raw signals should be first-class unknown or raw-signal-unavailable, not inferred.
4. Session/thread snapshots. During ingest, materialize per-session capability matrix rows from observations at the session window. Historical backfill should be marked best-effort when only current config exists.
5. Debug/export read surface. Add ax capabilities ... or ax mcp ... only after command naming is decided. Output should be row-oriented with state columns, source config, hidden/disabled reason, last error, safe command/cwd/env diagnostics. Add matching MCP tool and Studio panel once the CLI/query contract is stable.
6. Opt-in payload boundary. Add a gated capture mode later, with retention and redaction tests. Keep metadata default always-on.

Files/modules likely touched:

• packages/schema/src/schema.surql, packages/schema/src/schema.test.ts, packages/schema/src/types.ts
• new apps/axctl/src/capabilities/ or apps/axctl/src/ingest/capabilities/ modules for state vocabulary, projectors, statement builders, and tests
• apps/axctl/src/ingest/claude-config.ts and new non-Claude config scanners
• apps/axctl/src/ingest/stage/registry.ts, apps/axctl/src/ingest/run.ts for the new ingest stage and progress labels
• apps/axctl/src/queries/capability-matrix.ts plus tests
• apps/axctl/src/cli/commands/*, apps/axctl/src/mcp/tools.ts, dashboard contract/server, and Studio route once the read contract is accepted
• apps/axctl/src/ingest/provider-parity.ts and scripts/check-provider-parity.ts

Risks:

• model_visible is not directly observable in every harness. It must remain unknown unless the provider exposes a real signal or ax owns the injection path.
• Reconstructing historical visibility from current config can lie. Historical rows need provenance like observed_from=current_config vs observed_from=session_event.
• Per-tool snapshots can grow quickly. Prefer session-grain snapshots, dedupe subjects by stable key/content hash, and index by session, provider, kind, state.
• Safe diagnostics are easy to overexpose. Store env key names/classes and safe paths, not env values or raw command payloads.
• MCP/tool naming varies across providers. Raw provider names and normalized capability ids both need to survive.

Open maintainer decisions before code beyond a config-only prototype:

• Command/product name: ax capabilities, ax mcp, or under existing harness/doctor surfaces?
• Should plugins, skills, hooks, MCP, and native tools share one matrix, or should MCP/plugin be separate views over shared tables?
• What exact evidence is sufficient for authenticated, schema_accepted, current_thread, and model_visible per provider?
• Should the first shipped slice create new capability_* tables, or first expose existing guidance_config_artifact rows as a read-only config inventory?
• What is the opt-in payload UX: install-time setting, env var, command flag, or per-export flag?

PR decision for this pass: I would skip opening a draft PR until the table/command semantics above are confirmed. The only low-risk code slice I see is a config-only projection from existing guidance_config_artifact, but even that bakes in naming and matrix boundaries that this issue should settle first.

---

Generated with ax.