GroundednessJudge silently scores infra/API errors (negative max_tokens crash) as hard fails

[haroldship]

Describe the bug

GroundednessJudge.judge() (and ExactMatchJudge's equivalent block) in evaluator/judge.py wrap the judge LLM call in a bare except Exception as e: return JudgeOutput(score=0.0, explanation=f"Judge Error {e}.") (confirmed on main at lines 95-96 and 307-308). Any API-level failure to invoke the judge — not just a genuine "this answer is ungrounded" verdict — is silently converted into a hard score=0.0.

Concretely: GroundednessJudge.judge() concatenates the last N_TOOL_CALLS_PER_TURN (20) tool responses plus the full predicted answer and sends that to the judge model with no truncation/length check against the judge model's context window. When the predicted-answer/tool-response transcript is large, the underlying langchain_openai client auto-computes max_tokens = context_window - prompt_tokens, which can go deeply negative, and the provider returns a 400. That exception is caught and scored as a hard groundedness FAIL, indistinguishable from an actual ungrounded answer.

We hit this while scoring a 300-task capability_4 (multiturn) run: 24/300 dialogues (8%) came back with groundedness_score: 0.0 and a score_explanation.groundedness string starting with "Judge Error Error code: 400 - {'error': {'message': 'max_tokens must be at least 1, got -36646.', ...}}." — i.e. the judge never actually rendered a groundedness verdict for these, but they count as full dialogue failures in the final scored output.

Separately, the second occurrence of this pattern (around line 307-308) has its own bug: explanation="Judge Error {e}." is missing the f prefix, so {e} is a literal string rather than the actual exception — that occurrence never surfaces the real error message.

To Reproduce

1. Run a capability_multiturn (capability_4) evaluation where at least one predicted answer + its last-20-tool-responses is large enough to overflow the groundedness judge model's context window (happens naturally on agents that produce very long tool-call transcripts, e.g. a step-cap/runaway condition on the agent side).
2. Score it with the standard scoring pipeline (evaluate_domain → TurnScorer → GroundednessJudge.judge()).
3. Inspect the output dialogues for details.per_turn[*].metadata.score_explanation.groundedness containing "Judge Error".

Expected behavior

An API/infra-level failure to invoke the judge should not be indistinguishable from a genuine "ungrounded" verdict. Either:

• surface it as an inconclusive/skipped judge result (excluded from the dialogue aggregate), or
• defensively bound the judge's input (truncate the concatenated tool-responses/answer) before invoking, so this class of error doesn't occur in the first place, with a bounded retry as a fallback.

Also, fix the missing f prefix on the second except block's explanation="Judge Error {e}." so the real exception text is actually captured.

Screenshots / logs

Judge Error Error code: 400 - {'error': {'message': 'max_tokens must be at least 1, got -36646.', 'type': 'BadRequestError', 'param': None, 'code': 400}}.
Judge Error Error code: 400 - {'error': {'message': 'max_tokens must be at least 1, got -44982.', 'type': 'BadRequestError', 'param': None, 'code': 400}}.
Judge Error Error code: 400 - {'error': {'message': 'max_tokens must be at least 1, got -14856.', 'type': 'BadRequestError', 'param': None, 'code': 400}}.

Additional context

Found via cuga-project/cuga-eval, which vendors a copy of this evaluator package under benchmarks/m3/evaluator/ (kept in sync with this repo per that project's own sync tooling); companion issue filed there: cuga-project/cuga-eval#143. The bare-except pattern is present in both GroundednessJudge.judge() and the analogous block later in the same file (~line 307) — worth fixing both together.