Skip to content

hermes-integration.md

Latest commit

 

History

History
71 lines (54 loc) · 2.3 KB

hermes-integration.md

File metadata and controls

71 lines (54 loc) · 2.3 KB
Raw

Optional Hermes Integration Sketch

The Burn router is not currently wired into live Hermes Agent by default. It is a standalone Rust/Burn classifier that can become an optional pre-router in front of Hermes prompt/tool construction.

Current state

  • Live Hermes still builds tool schemas through toolsets.py / model_tools.py.
  • No production hook in run_agent.py, model_tools.py, or config.yaml calls this router yet.
  • The router is safe to run as an advisory sidecar because it returns a route hint, not an autonomous action.

Recommended integration

Add an opt-in config block:

routing:
  burn_router:
    enabled: false
    binary: /path/to/hermes-burn-tool-router/target/release/hermes-burn-tool-router
    model: /path/to/hermes-burn-tool-router/tool_router.safetensors
    confidence_threshold: 0.72
    fallback: full_surface

Then, before tool schema construction:

message
  → Burn router predicts category + confidence
  → if confidence >= threshold, map category to enabled_toolsets
  → build reduced tool schema for first LLM pass
  → if the model asks for unavailable tools or confidence is low, fall back to full surface

Why not hard-gate immediately?

Hard-gating is risky. Misrouting a user request can hide the correct tool from the main LLM. The sane first production mode is:

  1. Observe-only: log router category/confidence next to actual tool calls.
  2. Hint mode: bias/reorder toolsets while still keeping fallback available.
  3. Narrow mode: only for very high-confidence categories and short obvious requests.

Local shim

cargo build --release
scripts/route_hint.py "search X for trending Base coins"

Example output:

{
  "category": "x_search",
  "confidence": 0.98,
  "enabled_toolsets": ["x_search"],
  "mode": "narrow"
}

Expansion direction

  • Collect real Hermes session text → actual tool call labels from local session DB.
  • Train on real labels instead of only synthetic patterns.
  • Add top-k category output for ambiguous tasks.
  • Add regression tests for high-risk confusions:
    • search files vs web search
    • run /model vs shell run
    • send message vs ask clarification
    • x_search vs generic web research
  • Keep fallback-full-surface as default until real-world false-negative rate is measured.