A CI harness that tests voice AI agents over real phone calls, automatically.
🌐 Live web dashboard: voxharness-web.vercel.app — run tests and watch them stream in real time (latency gauge, live transcript, animated scorecard). Frontend source: Aarti-panchal01/voxharness-web.
Voice teams test their agents by calling their own number and listening. Every code change, by hand, in every language they support. It does not scale, it is not reproducible, and the number that actually hangs callers up, the dead air before the agent responds, never shows up in a unit test.
No human ever picks up a phone. voxharness is the caller (the customer): it plays each scripted customer line with Sarvam TTS, listens to your voice AI agent's reply with Sarvam STT, measures the latency between the two, checks for repetition, scores naturalness with Groq, then moves to the next turn. Your agent is the only thing on the other end. The only thing a human touches is the scorecard at the end.
voxharness turns that manual test into one command you can run in CI:
# --real places an actual call (needs credentials + a deployed media server);
# drop --real to run the verified mock path with zero setup.
voxharness dial --real --number +91XXXXXXXXXX --scenario examples/basic-hindi-kyc.yamlIt dials your agent, runs a scripted conversation from a YAML file, and scores three things that come straight from shipping multilingual voice in production:
- Response latency — does the agent start answering within budget (target sub-1.4s)?
- Repetition — does the agent ever ask the same question twice in one call?
- Naturalness — how human vs robotic does it sound?
Then it prints a scorecard and returns a non-zero exit code if a hard check regressed, so it fails your build the same way a broken test would.
The 2026 voice-AI stack has good testing SaaS (Cekura, Hamming, Coval), but they are closed platforms you upload your agent to. There is no open, self-hostable harness that runs inside your own CI, over the real telephony path (jitter, codecs, end-of-turn timing), with Indian languages as a first-class citizen. Latency measured on a clean datacenter socket is not the number that matters. Latency measured over the phone is.
voxharness is that harness.
# core (mock mode, zero credentials)
pip install -e .
# real mode (places actual phone calls)
pip install -e ".[real]"Requires Python 3.10+.
voxharness ships with mock drivers, so you can see it work end to end before wiring up any APIs:
voxharness dial --scenario examples/basic-hindi-kyc.yaml --mock====================================================================
voxharness scorecard :: basic-hindi-kyc (hindi)
====================================================================
Turn 1 user: Namaste, mera naam Aarti hai
agent: Namaste! Main aapki kaise madad kar sakti hoon?
latency PASS 909ms / 1400ms budget (+491ms)
repetition PASS no earlier turn echoed
naturalness PASS 92/100 (min 70)
Turn 2 user: Mujhe apna account band karna hai
agent: Zaroor, main aapka account detail dekh rahi hoon.
latency PASS 895ms / 1400ms budget (+505ms)
repetition PASS no earlier turn echoed
naturalness PASS 92/100 (min 70)
--------------------------------------------------------------------
avg latency: 902ms avg naturalness: 92/100
OVERALL: PASS
====================================================================
Add --json for a machine-readable scorecard you can post on a PR.
A scenario is a small YAML file. A teammate who does not write Python can read and edit it.
version: 1
name: basic-hindi-kyc
language: hindi # hindi | kannada | telugu | tamil | english
naturalness_min: 70 # optional, 0-100
repetition_threshold: 0.85 # optional, 0-1 similarity to count as a repeat
wait_for_greeting: true # optional: hear the agent's opening greeting first
turns:
- speak: "Namaste, mera naam Aarti hai"
expect_response_within_ms: 1400
expect_no_repetition: true
- speak: "Mujhe apna account band karna hai"
expect_response_within_ms: 1400speak— the user line voxharness says to your agent.expect_response_within_ms— latency budget for that turn (defaults to 1400).expect_no_repetition— fail the turn if the agent's reply echoes an earlier one.wait_for_greeting— settruefor agents that greet before the caller speaks; voxharness listens to and discards the greeting so turn 1 is measured against your line, not the "hello."
More examples in examples/.
| Metric | What it measures | Gates CI? |
|---|---|---|
| Latency | Time from end of user speech to first agent audio, vs the per-turn budget. | Yes (hard) |
| Repetition | Similarity of each agent reply to every earlier reply; flags echoes. | Yes (hard) |
| Naturalness | 0-100 human-vs-robotic score from an LLM judge. | Warning by default; --fail-on-naturalness to gate |
Naturalness is a model judgement, so it is non-deterministic in real mode. It
is a warning by default on purpose: a harness that fails your build on a fuzzy
score is a harness teams rip out. Opt in with --fail-on-naturalness when you
want it enforced.
How latency is measured, precisely. t0 is the moment voxharness finishes
streaming your prompt audio into the call; prompt frames are paced against a
real-time deadline, so t0 tracks playback-complete within Twilio's jitter
buffer. Onset (t1) is the first frame of a sustained voiced run (3+ frames by
default), not a single spike, so line echo and comfort noise can't fake a
sub-100ms "response." Latency is t1 − t0. Sub-frame precision via Twilio's
mark echo is on the roadmap.
Mock mode (default) runs everything in-memory and deterministically. No phone, no keys, no network. This is what CI runs, and it is the fully verified path in v0.1.
Real mode drives an actual call, fully automated, with no human on either
end — Twilio carries the audio, Sarvam does STT + TTS, Groq scores naturalness.
The media server (voxharness serve) is a WebSocket server Twilio's Media
Streams connects to: it plays each customer line into the call, uses energy VAD
to catch the exact moment your agent starts speaking (real time-to-first-token),
collects and transcribes the agent's reply, and scores it. It needs a public
wss:// URL — a live instance runs on Render at wss://voxharness.onrender.com/media
(see DEPLOY.md).
There are two directions, both zero-human:
Outbound — voxharness calls your agent (voxharness dial --real). voxharness
originates a Twilio call to your agent's number, plays the customer turns, and
polls for the scorecard.
export VOXHARNESS_PUBLIC_WS_URL=wss://voxharness.onrender.com/media
export TWILIO_ACCOUNT_SID=... TWILIO_AUTH_TOKEN=... TWILIO_FROM_NUMBER=+1XXXXXXXXXX
export VOXHARNESS_STREAM_TOKEN=$(openssl rand -hex 16) # match the server's token
# --number is YOUR VOICE AGENT'S phone number (e.g. an Inverix Raya line).
voxharness dial --real --number +91XXXXXXXXXX --scenario examples/inverix-raya-test.yamlInbound — your agent calls voxharness. Point your voxharness Twilio number's
"A call comes in" webhook at https://<your-server>/twiml/inbound. When your
agent dials that number, Twilio hits the webhook, voxharness answers by bridging
the call's audio to the media server, plays the customer, and scores the agent —
exactly the same pipeline, no human answering. Configure the inbound scenario
with VOXHARNESS_INBOUND_SCENARIO_B64 (base64 of a scenario JSON) or pass
?scenario_b64=... on the webhook URL.
The token now rides as a Stream
<Parameter>(validated in thestartevent), not a URL query string — a query on thewss://URL can break Twilio's Media Streams handshake (error 31920).
Security: the media server's
/mediaand/resultsendpoints are unauthenticated unlessVOXHARNESS_STREAM_TOKENis set on the server. On any public deployment, set it (an open/medialets anyone spend your Sarvam/Groq credits; an open/resultsexposes call transcripts). See DEPLOY.md.
What is verified: the full media pipeline — Twilio wire protocol, mu-law codec, VAD, latency measurement, transcription hand-off, and scoring — is exercised end to end by an in-process server test with a simulated Twilio client (
tests/test_media_server.py), so it runs in CI with no phone and no keys. A live call additionally needs a deployed server, Sarvam/Groq keys, and a real agent number to dial. voxharness never reports a measurement it did not take: a no-response turn fails, it is never given a fake latency.
voxharness runs on every push out of the box via .github/workflows/ci.yml:
it runs the test suite and then the mock smoke scenario, failing the build on a
non-zero scorecard. To gate a real voice regression, add your credentials as
repository secrets and swap the smoke step for --real.
- name: voxharness smoke
run: voxharness dial --scenario examples/english-smoke.yaml --mockEverything hangs off one seam: AgentConnection.send_user_utterance(text, lang) -> AgentResponse{transcript, latency_ms}. The runner speaks each turn and
collects replies; it never knows whether it is talking to a real phone or a
mock. Metrics are pure functions over the collected turns, so they are fast and
impossible to make flaky.
Mock mode (local, keyless):
scenario.yaml ─▶ runner ─▶ MockAgentConnection ─▶ replies
│
├─▶ latency / repetition / naturalness ─▶ scorecard ─▶ exit code
Real mode (over the phone):
dial --real ─▶ Twilio call ──(Media Streams wss)──▶ media server
│ play TTS (Sarvam) into call
│ VAD ─▶ time-to-first-token
│ collect agent audio ─▶ STT (Sarvam)
▼
CallSession ─▶ metrics + Groq naturalness
▼
/results/{callSid} ◀── dial polls ─▶ scorecard
The state machine (media/call_session.py) is synchronous and transport-free,
so it is unit-tested with synthetic frames; the async server is a thin shell.
Twilio Media Streams media server✅ shipped (voxharness serve).Greeting-first agents✅wait_for_greetinglistens past the opening line.mark-echo t0 for sub-frame latency precision (see "How latency is measured").- Barge-in: measure interruption handling when caller and agent overlap.
- DPDP / consent-flow assertions: check the agent captured and logged consent before processing personal data, per India's 2026 rules.
- Network-condition injection (packet loss, jitter, codec degradation).
- p50 / p95 latency percentiles across repeated runs.
- More language packs and accent coverage.
pip install -e ".[dev]"
pytestMIT. See LICENSE.