A DSPy-optimized LLM-judge that catches the coding-agent failures your eyeball test misses — and proves the judge itself is reliable. A small, cheap model becomes a trustworthy code reviewer once a stronger, reflective model optimizes its prompt.
Same model. Better prompt. That lift is the whole point.
git clone <this-repo> && cd dspy-coding-judge
python3 bench/run.py --mode cached # replay the recorded real verdicts
# or: bash scripts/demo.shDSPY CODING-JUDGE BENCHMARK · judge = llama-3.1-8b · optimizer = glm-5.2 (reflective) · 16 dev cases
──────────────────────────────────────────────────────────────────────────
verdict agreement (pass/fail) baseline 12/16 (75%) → optimized 15/16 (94%)
failure-mode accuracy (on fails) baseline 6/10 → optimized 6/10
exact match (verdict + mode) baseline 8/16 (50%) → optimized 11/16 (69%) ▲ +19pts
LIFT: +3 cases, +19pts exact match — same model, better prompt
The optimized judge runs on the exact same llama-3.1-8b as the baseline — only its
prompt changed. A small open model misjudged 4 of 16 verdicts zero-shot; after the
reflective model optimized its prompt, it misjudged 1. The lift is purely prompt quality.
Coding agents ship plausible-looking diffs that fail in five recurring ways: they invent APIs that don't exist, get the logic subtly wrong, write insecure code, ignore a stated constraint, or leave the job incomplete. A human glances, sees runnable code, and approves. You need an automated judge — but then you have to prove the judge is reliable, not just confident.
This repo does both: it builds an LLM-as-judge for coding-agent outputs, and it uses DSPy to programmatically optimize that judge's prompt, then reports a quantitative before/after against ground-truth labels so you can see — and gate on — the improvement.
DSPy's flagship two-model pattern, and the reason the result is interesting:
- The judge runs on
llama-3.1-8b-instruct— a small, cheap, open model. Zero-shot it is a mediocre judge (75% verdict agreement). - A stronger, reflective model —
glm-5.2— acts as the BootstrapFewShot teacher, generating the demonstrations that go into the small model's prompt. - The optimized judge still runs on
llama-3.1-8b. So the before/after isolates prompt quality on the same cheap model: verdict agreement lifts 75% → 94% (+19pts), exact match 50% → 69%.
No hand-tuned prompt engineering. The DSPy optimizer (reflective glm-5.2 as teacher)
discovered the demonstrations that make the cheap judge agree with a senior reviewer. A
strong model making a small model reliable is exactly the cost/quality tradeoff production
agent teams want — you don't get a bigger judge bill, you get a better-cheaper judge.
The repo also supports MIPROv2 instruction optimization (
DSPY_OPTIMIZER=mipro); on this small dataset BootstrapFewShot with a reflective teacher gave the cleanest, most reliable lift. Both use the reflectiveglm-5.2to improve the small judge.
The judge doesn't just say pass/fail — it names why. The 40-case labeled dataset is a curated seed (hand-authored, not scraped; honesty over size), spread across the five failure modes real coding agents produce:
| Failure mode | Dataset (fails) | Dev split | What it looks like |
|---|---|---|---|
| hallucinated_api | 5 | 2 | orders.sumAll('total'), arr.reject(...), o.deepCopy() |
| wrong_logic | 5 | 2 | off-by-one counts, biased "shuffle", min-where-max asked |
| insecure | 5 | 2 | SQL interpolation, plaintext password compare, secrets in localStorage |
| ignored_constraint | 5 | 2 | "never more than 10" cap dropped, "skip zeros" forgotten |
| incomplete | 4 | 2 | LRU with no eviction, rate limiter with no time window |
On the 16-case dev split, the optimization lifted verdict agreement 12→15/16; the small model's primary zero-shot weakness was misjudging pass/fail (calling wrong code acceptable, or rejecting correct code), which the reflective-sourced demos corrected. Failure-mode naming (6/10) was unchanged — an honest result: demos fixed the verdict, not the diagnosis taxonomy, on this seed.
- Replace
data/cases.jsonlwith your own labeled agent outputs (same schema:id, split, task, agent_output, label, failure_mode). python bench/run.py --mode realregenerates the before/after.- Drop
.github/workflows/eval.ymlin — now every change that regresses the judge goes red before it ships (gate: no exact-match regression AND optimized verdict ≥ 0.80).
pip install ".[real]"
export OPENROUTER_API_KEY=... # the small judge model (llama-3.1-8b-instruct)
export ZAI_CODING_KEY=... # the reflective optimizer (glm-5.2)
python3 bench/run.py --mode real
# optional: DSPY_OPTIMIZER=mipro to use MIPROv2 instruction optimization instead
--mode cachedreplays the committed real verdicts so the demo runs key-free and the CI gate is deterministic.--mode realhits the live models and rewritesartifacts/cached/*.jsonl+artifacts/optimized_judge.json. DSPy's disk cache is disabled in--mode realso every run is genuinely fresh.
An implementation of LLM-as-judge + DSPy BootstrapFewShot (reflective teacher)
and MIPROv2 (opt-in) in the reflective-model-optimizes-cheap-model configuration:
a quantitative, gated benchmark, not a vibe check. ~150 lines of judge code; the lift is
real and reproducible.
Built by Kevin Hill. I make production AI stop hallucinating, stop burning tokens, and stop silently breaking — and I hand you the evals to prove it.