Skip to content

Commit e7bcc51

Browse files
biobootloaderbiobootloader
authored
Merge pull request #334 from AbanteAI/mentat-258
Add benchmark results for anthropic/claude-opus-4.1
2 parents 41e38ea + fbbcbce commit e7bcc51

1,606 files changed

Lines changed: 1305426 additions & 6159 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎benchmark_pipeline/benchmark_config.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ model_display_names:
2727
"anthropic/claude-3.5-sonnet-20240620": "Sonnet 3.5"
2828
"anthropic/claude-3.7-sonnet": "Sonnet 3.7"
2929
"anthropic/claude-3.7-sonnetthinking": "Sonnet 3.7 Thinking"
30+
"anthropic/claude-opus-4.1": "Claude Opus 4.1"
3031
"anthropic/claude-sonnet-4": "Sonnet 4"
3132
"anthropic/claude-sonnet-4thinking": "Sonnet 4 Thinking"
3233
"openai/gpt-4.1": "GPT-4.1"

‎docs/cases.html‎

Lines changed: 1209 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/cases/anthropic_claude-opus-4.1/aider_aider___init__.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/__init__.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/__init__.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="success">Success</span></p>
21+
<p><strong>Prompt Tokens:</strong> 59517</p>
22+
<p><strong>Native Prompt Tokens:</strong> 67626</p>
23+
<p><strong>Native Completion Tokens:</strong> 327</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $1.038915</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider___init__.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider___init__.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider___init__.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<div class="success-message"><p>✓ No differences found (successful run)</p><p>Expected output matches the model output exactly.</p></div>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_analytics.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/analytics.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/analytics.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="failure">Failure</span></p>
21+
<p><strong>Prompt Tokens:</strong> 24542</p>
22+
<p><strong>Native Prompt Tokens:</strong> 30657</p>
23+
<p><strong>Native Completion Tokens:</strong> 2152</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $0.621255</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_analytics.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_analytics.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_analytics.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<pre class="diff"><div></div><div>index bcb21745a..cb655cb85 100644</div><div class="diff-header">--- a/aider_aider_analytics.py_expectedoutput.txt (expected):tmp/tmp1jdkcki9_expected.txt </div><div class="diff-header">+++ b/aider_aider_analytics.py_extracted.txt (actual):tmp/tmp5cxdkvna_actual.txt </div><div class="diff-info">@@ -72,6 +72,7 @@ class Analytics:</div><div> </div><div> def __init__(self, logfile=None, permanently_disable=False):</div><div> self.logfile = logfile</div><div class="diff-added">+ self.asked_opt_in = False</div><div> self.get_or_create_uuid()</div><div> </div><div> if self.permanently_disable or permanently_disable or not self.asked_opt_in:</div><div></div></pre>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_args.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/args.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/args.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="success">Success</span></p>
21+
<p><strong>Prompt Tokens:</strong> 61543</p>
22+
<p><strong>Native Prompt Tokens:</strong> 76671</p>
23+
<p><strong>Native Completion Tokens:</strong> 7464</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $1.709865</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_args.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_args.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_args.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<div class="success-message"><p>✓ No differences found (successful run)</p><p>Expected output matches the model output exactly.</p></div>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_coders_editblock_coder.py.html‎

Lines changed: 51 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_coders_editblock_prompts.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/coders/editblock_prompts.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/coders/editblock_prompts.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="success">Success</span></p>
21+
<p><strong>Prompt Tokens:</strong> 35371</p>
22+
<p><strong>Native Prompt Tokens:</strong> 42180</p>
23+
<p><strong>Native Completion Tokens:</strong> 2072</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $0.7881</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_editblock_prompts.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_editblock_prompts.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_editblock_prompts.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<div class="success-message"><p>✓ No differences found (successful run)</p><p>Expected output matches the model output exactly.</p></div>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_coders_patch_coder.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/coders/patch_coder.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/coders/patch_coder.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="failure">Failure</span></p>
21+
<p><strong>Prompt Tokens:</strong> 22441</p>
22+
<p><strong>Native Prompt Tokens:</strong> 28880</p>
23+
<p><strong>Native Completion Tokens:</strong> 7800</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $1.0182</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_patch_coder.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_patch_coder.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_patch_coder.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<pre class="diff"><div></div><div>index 1992834ec..ae4f0e21e 100644</div><div class="diff-header">--- a/aider_aider_coders_patch_coder.py_expectedoutput.txt (expected):tmp/tmp4qhok082_expected.txt </div><div class="diff-header">+++ b/aider_aider_coders_patch_coder.py_extracted.txt (actual):tmp/tmp06tdzz40_actual.txt </div><div class="diff-info">@@ -546,7 +546,7 @@ class PatchCoder(Coder):</div><div> action = PatchAction(type=ActionType.ADD, path="", new_content="\n".join(added_lines))</div><div> return action, index</div><div> </div><div class="diff-removed">- def apply_edits(self, edits: List[PatchAction]):</div><div class="diff-added">+ def apply_edits(self, edits: List[EditResult]):</div><div> """</div><div> Applies the parsed PatchActions to the corresponding files.</div><div> """</div><div></div></pre>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_coders_wholefile_coder.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/coders/wholefile_coder.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/coders/wholefile_coder.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="success">Success</span></p>
21+
<p><strong>Prompt Tokens:</strong> 20299</p>
22+
<p><strong>Native Prompt Tokens:</strong> 26267</p>
23+
<p><strong>Native Completion Tokens:</strong> 1389</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $0.49818</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_wholefile_coder.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_wholefile_coder.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_coders_wholefile_coder.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<div class="success-message"><p>✓ No differences found (successful run)</p><p>Expected output matches the model output exactly.</p></div>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

‎docs/cases/anthropic_claude-opus-4.1/aider_aider_exceptions.py.html‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>Case: aider/exceptions.py - Claude Opus 4.1</title>
7+
<link rel="stylesheet" href="../../styles.css">
8+
</head>
9+
<body>
10+
<header>
11+
<h1>Case: aider/exceptions.py</h1>
12+
<h2>Model: Claude Opus 4.1</h2>
13+
<p><a href="../../models/anthropic_claude-opus-4.1.html">All Claude Opus 4.1 Cases</a> | <a href="../../cases.html">All Cases</a> | <a href="../../index.html">Home</a></p>
14+
</header>
15+
<main>
16+
<section class="case-details">
17+
<div class="case-info">
18+
<h2>Benchmark Case Information</h2>
19+
<p><strong>Model:</strong> Claude Opus 4.1</p>
20+
<p><strong>Status:</strong> <span class="success">Success</span></p>
21+
<p><strong>Prompt Tokens:</strong> 7265</p>
22+
<p><strong>Native Prompt Tokens:</strong> 8871</p>
23+
<p><strong>Native Completion Tokens:</strong> 1313</p>
24+
<p><strong>Native Tokens Reasoning:</strong> 0</p>
25+
<p><strong>Native Finish Reason:</strong> stop</p>
26+
<p><strong>Cost:</strong> $0.23154</p>
27+
</div>
28+
29+
<div class="content-links">
30+
<h2>View Content</h2>
31+
<ul>
32+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_exceptions.py/prompt.html" class="content-link">View Prompt</a></li>
33+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_exceptions.py/expected.html" class="content-link">View Expected Output</a></li>
34+
<li><a href="../../content/anthropic_claude-opus-4.1/aider_aider_exceptions.py/actual.html" class="content-link">View Actual Output</a></li>
35+
</ul>
36+
</div>
37+
38+
<div class="diff-section">
39+
<h2>Diff (Expected vs Actual)</h2>
40+
<div id="diff-output">
41+
<div class="success-message"><p>✓ No differences found (successful run)</p><p>Expected output matches the model output exactly.</p></div>
42+
</div>
43+
</div>
44+
</section>
45+
</main>
46+
<footer>
47+
<p>LoCoDiff-bench - <a href="https://github.com/AbanteAI/LoCoDiff-bench">GitHub Repository</a></p>
48+
</footer>
49+
</body>
50+
</html>
51+

0 commit comments

Comments
 (0)