Skip to content

Commit 5a15c5f

Browse files
Complete benchmark suite - retry 3 API error cases for anthropic/claude-opus-4.1
- Successfully retried 3 cases that had API errors - 2 additional successes, 1 failure (output mismatch) - Final success rate: 61.5% (123/200 successful) - Additional cost: $5.02 - Final total cost: $215.48 - Benchmark suite now 100% complete with 0 API errors Mentat precommit script passed. Log: https://mentat.ai/gh/AbanteAI/LoCoDiff-bench/log/45656c60-4a19-4499-9037-99db5311a351 Co-authored-by: biobootloader <128252497+biobootloader@users.noreply.github.com>
1 parent 3123731 commit 5a15c5f

12 files changed

Lines changed: 5116 additions & 0 deletions

File tree

locodiff-250425/results/ghostty_src_font_discovery.zig/anthropic_claude-opus-4.1/20250813_200911/extracted_output.txt

Lines changed: 839 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"model": "anthropic/claude-opus-4.1",
3+
"benchmark_case": "ghostty_src_font_discovery.zig",
4+
"prompts_dir": "locodiff-250425/prompts",
5+
"prompt_file": "locodiff-250425/prompts/ghostty_src_font_discovery.zig_prompt.txt",
6+
"expected_file": "locodiff-250425/prompts/ghostty_src_font_discovery.zig_expectedoutput.txt",
7+
"timestamp_utc": "2025-08-13T20:09:11.486718+00:00",
8+
"success": true,
9+
"error": null,
10+
"api_error": false,
11+
"raw_response_length": 30109,
12+
"extracted_output_length": 29914,
13+
"expected_output_length": 29915,
14+
"results_dir": "locodiff-250425/results/ghostty_src_font_discovery.zig/anthropic_claude-opus-4.1/20250813_200911",
15+
"generation_id": "gen-1755115751-tAuoDfZGUmH43Q9qumcc",
16+
"cost_usd": 1.16811,
17+
"prompt_tokens": 28880,
18+
"completion_tokens": 6806,
19+
"total_tokens": 35686,
20+
"native_prompt_tokens": 35619,
21+
"native_completion_tokens": 8451,
22+
"native_finish_reason": "stop",
23+
"stats_error": null,
24+
"native_tokens_reasoning": 0
25+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
No differences found.

0 commit comments

Comments
 (0)