🩺 Telemetry-MCP-Okahu

A Self-Healing AI Agent That Fixes Its Own Bugs — Using Only Production Traces

No guessing. No print-debugging. No local logs. Just telemetry.

💡 The Big Idea

Most AI-coding demos show an agent building an app. This one shows something far more interesting: an agent debugging a broken production service the way a senior SRE would — by reading distributed traces from an observability platform.

The repo ships with a deliberately broken Text-to-SQL API (analyst.py). Every LLM call it makes is auto-instrumented by Monocle and exported to Okahu Cloud. The self-healing agent (@analyst_v3) is then pointed at the failing test suite with one hard rule:

You may not guess. Every fix must be justified by evidence from an Okahu Cloud trace, fetched through the hosted Okahu MCP.

The agent runs the tests, watches them fail, pulls the traces, reads the actual request/response spans, and fixes the code — one evidence-backed bug at a time — until everything is green.

🐛 The Three Planted Bugs

#	Bug	Symptom	What the Trace Reveals
1	Invalid model name — `model="gpt-5.4-typo"`	Every call throws `openai.NotFoundError`	The span's request attributes show the bogus model name sent to the API
2	Wrong response attribute — `response.choices[0].text`	`AttributeError` after Bug 1 is fixed	The span's response payload shows content lives at `.message.content`
3	Schema mismatch — system prompt teaches `customers`/`products`, real DB has `users`/`orders`	`sqlite3.OperationalError: no such table`	The span's captured prompt + LLM output expose SQL targeting phantom tables

The bugs are layered: fixing one uncovers the next. The agent can't fix them in a single lucky shot — it must iterate: test → trace → diagnose → archive → fix → repeat.

🔬 Why Trace-Driven Debugging Matters

Trace-driven, not guess-driven. The agent queries the platform (/okahu:get_latest_traces), not local logs or stack traces. This mirrors how real production incidents are diagnosed.
Infrastructure-native. Observability arrives via a hosted MCP server — the agent talks to Okahu Cloud exactly the way it would talk to any other tool.
Auto-instrumented telemetry. Because the app uses the official openai SDK, Monocle captures every LLM span with zero manual instrumentation code.

⚠️ The One Rule of Monocle Instrumentation

Monocle can only auto-instrument supported SDKs:

Works ✅	Does NOT work ❌
`openai` Python SDK	Raw `requests.post()` to LLM APIs
`google-genai` SDK	Direct `httpx` / `aiohttp` calls
`langchain` framework	Custom API wrappers without SDK instrumentation
`llama-index` framework

Always call the LLM through the OpenAI SDK directly:

from openai import OpenAI

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
)

If you swap this for a raw HTTP call, no traces reach Okahu Cloud and the entire self-healing loop goes blind.

🏗️ Architecture

flowchart LR
    subgraph App["Buggy Text-to-SQL API"]
        A[main.py<br/>FastAPI] --> B[analyst.py<br/>3 planted bugs]
        B --> C[(sales.db<br/>users / orders)]
    end
    B -- "openai SDK call" --> D[GPT-4o]
    B -. "Monocle auto-instrumentation" .-> E[Okahu Cloud]
    subgraph Agent["@analyst_v3 (OpenCode)"]
        F[Run pytest] --> G[Query hosted Okahu MCP]
        G --> H[Diagnose from trace evidence]
        H --> I[Archive + Fix analyst.py]
        I --> F
    end
    G <-- "/okahu:get_latest_traces" --> E
    F -. "exercises" .-> A

📁 Repository Layout

telemetry-mcp-okahu/
├── analyst.py               # 🐛 The buggy Text-to-SQL analyst (the patient)
├── main.py                  # FastAPI wrapper (POST /query, GET /health)
├── test_analyst.py          # Test suite that exposes all 3 bugs
├── setup_db.py              # Creates + seeds sales.db (users / orders)
├── reset_demo.py            # Restores the buggy state before each demo
├── boilerplate.py           # ✅ Known-good reference patterns for the agent
├── demo/analyst_buggy.py    # Canonical buggy source used by reset_demo.py
├── versions/                # Agent archives analyst_vN.py here before each fix
├── .opencode/agent/analyst_v3.md   # The self-healing agent definition
├── opencode.example.json    # Hosted Okahu MCP config for OpenCode
├── .env.example             # Required environment variables
└── requirements.txt

🚀 Step-by-Step Setup

1. Environment Variables

Set up your keys in the telemetry-mcp-okahu/.env file. You will need an OpenAI API Key and an Okahu API Key for telemetry.

cd telemetry-mcp-okahu
echo 'OPENAI_API_KEY="your-openai-key"' > .env
echo 'OPENAI_MODEL="gpt-4o"' >> .env
echo 'OKAHU_API_KEY="your-okahu-key"' >> .env
echo 'MONOCLE_EXPORTER="okahu"' >> .env

2. Install Dependencies

Create a virtual environment and install required packages:

python3 -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install monocle_apptrace monocle_test_tools openai fastapi uvicorn python-dotenv pytest

3. Initialize the Database

Run the setup script to create and seed the sales.db database:

python setup_db.py

4. Configure OpenCode MCP

Update your global OpenCode config (~/.config/opencode/opencode.json) to use the hosted Okahu MCP:

{
  "mcp": {
    "okahu": {
      "type": "remote",
      "url": "https://mcp.okahu.ai/mcp",
      "headers": {
        "x-api-key": "your-okahu-api-key-here"
      },
      "enabled": true
    }
  }
}

Then re-authenticate:

opencode mcp logout okahu
opencode mcp auth okahu

If prompted to re-authenticate, select "Yes".

🎬 Usage

Reset the Demo (Run Before Each Test)

Always reset to the buggy state before starting a new demo:

python reset_demo.py

This restores analyst.py with all 3 bugs, clears versions/, and rebuilds sales.db.

Run the Self-Healing Agent

Open your OpenCode terminal in the telemetry-mcp-okahu/ directory and run:

"@analyst_v3 Fix the buggy Text-to-SQL API:

The analyst.py, test_analyst.py, and main.py files already exist but have bugs.

1. Run Tests: Execute `pytest test_analyst.py -v` to see failures.
2. Analyze Traces: Wait 5s, then query Okahu MCP (/okahu:get_latest_traces)
   with workflow_name='text_to_sql_analyst_v3'.
3. Fix Loop:
   - Archive current analyst.py to versions/analyst_vN.py
   - Fix the bug based on trace analysis (check boilerplate.py for correct patterns)
   - Record the trace ID used to diagnose each fix
   - Run tests again
   - Repeat until all tests pass
4. Final Report: Output a summary table of all issues fixed with their
   associated trace IDs.

Rules: No debug files. Debug only via Okahu MCP traces. Always call the MCP
tool to get the logs from traces, do not use the local logs in the terminal"

What a Successful Run Looks Like

❌ 4 failed, 1 passed   → trace shows model 'gpt-5.4-typo' → fix model name
❌ 4 failed, 1 passed   → trace shows response shape       → fix .message.content
❌ 2 failed, 3 passed   → trace shows SQL on 'customers'   → fix schema prompt
✅ 5 passed             → final report with trace IDs

Bug	Root Cause (from trace)	Fix	Trace ID
Invalid model	Request span: `model=gpt-5.4-typo` → 404	`model="gpt-4o"`	`trace_abc...`
Response attr	Response span: content at `message.content`	`.choices[0].message.content`	`trace_def...`
Schema mismatch	Output span: `SELECT ... FROM customers`	Prompt now teaches `users`/`orders`	`trace_ghi...`

Try the API Directly

uvicorn main:app --reload --port 8000
curl -X POST http://localhost:8000/query \
     -H "Content-Type: application/json" \
     -d '{"question": "How many users are there?"}'

🧪 The Test Suite

Test	Exposes
`test_database_has_real_schema`	Sanity: `sales.db` really has `users` + `orders`
`test_generate_sql_returns_select`	Bug 1 (invalid model) → Bug 2 (wrong attribute)
`test_sql_targets_real_tables`	Bug 3 (phantom `customers`/`products` schema)
`test_end_to_end_count_users`	Full pipeline correctness (8 users)
`test_end_to_end_order_revenue`	Aggregation executes against the real schema

🔮 Why This Pattern Is the Future

Today's AI agents fail silently and get "fixed" by prompt tinkering. This POC demonstrates the alternative: agents wired into the same observability stack as the rest of your infrastructure, capable of closing the loop from failure → telemetry → root cause → verified fix — autonomously, and with an audit trail of trace IDs for every change they make.

🔗 Links

🌐 Live project site: https://tirth1263.github.io/telemetry-mcp-okahu/
🔭 Okahu Cloud: https://www.okahu.ai/
🧿 Monocle (Linux Foundation): https://github.com/monocle2ai/monocle
🤖 OpenCode: https://opencode.ai/

Built by Tirth Rank · If this repo helped you understand trace-driven self-healing agents, ⭐ star it!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

🩺 Telemetry-MCP-Okahu

A Self-Healing AI Agent That Fixes Its Own Bugs — Using Only Production Traces

💡 The Big Idea

🐛 The Three Planted Bugs

🔬 Why Trace-Driven Debugging Matters

⚠️ The One Rule of Monocle Instrumentation

🏗️ Architecture

📁 Repository Layout

🚀 Step-by-Step Setup

1. Environment Variables

2. Install Dependencies

3. Initialize the Database

4. Configure OpenCode MCP

🎬 Usage

Reset the Demo (Run Before Each Test)

Run the Self-Healing Agent

What a Successful Run Looks Like

Try the API Directly

🧪 The Test Suite

🔮 Why This Pattern Is the Future

🔗 Links

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.opencode/agent		.opencode/agent
demo		demo
docs		docs
versions		versions
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
analyst.py		analyst.py
boilerplate.py		boilerplate.py
main.py		main.py
opencode.example.json		opencode.example.json
requirements.txt		requirements.txt
reset_demo.py		reset_demo.py
setup_db.py		setup_db.py
test_analyst.py		test_analyst.py

Folders and files

Latest commit

History

Repository files navigation

🩺 Telemetry-MCP-Okahu

A Self-Healing AI Agent That Fixes Its Own Bugs — Using Only Production Traces

💡 The Big Idea

🐛 The Three Planted Bugs

🔬 Why Trace-Driven Debugging Matters

⚠️ The One Rule of Monocle Instrumentation

🏗️ Architecture

📁 Repository Layout

🚀 Step-by-Step Setup

1. Environment Variables

2. Install Dependencies

3. Initialize the Database

4. Configure OpenCode MCP

🎬 Usage

Reset the Demo (Run Before Each Test)

Run the Self-Healing Agent

What a Successful Run Looks Like

Try the API Directly

🧪 The Test Suite

🔮 Why This Pattern Is the Future

🔗 Links

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages