-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathrules.html
More file actions
311 lines (296 loc) · 25.8 KB
/
Copy pathrules.html
File metadata and controls
311 lines (296 loc) · 25.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Official participant rules for team registration, Agent submissions, evaluation integrity, and result verification in the Build-Bench Challenge." />
<title>Competition Rules | Build-Bench Challenge</title>
<link rel="canonical" href="https://aiops-lab-nku.github.io/BuildBench/rules.html" />
<link rel="icon" type="image/svg+xml" href="assets/logo.svg" />
<link rel="alternate icon" type="image/png" href="assets/favicon.png" />
<link rel="stylesheet" href="styles.css?v=20260811-7" />
<script src="assets/vendor/lucide.min.js?v=20260811-7" defer></script>
<script src="i18n/rules.js?v=20260811-6" defer></script>
<script src="i18n.js?v=20260811-7" defer></script>
<script src="auth-client.js?v=20260811-7" defer></script>
<script src="app.js?v=20260811-7" defer></script>
</head>
<body class="rules-page" data-page="rules">
<a class="skip-link" href="#main-content">Skip to main content</a>
<header class="site-header">
<div class="header-inner">
<a class="brand" href="index.html" aria-label="Build-Bench Challenge home"><span class="brand-mark" aria-hidden="true"><img src="assets/logo.svg" alt="" /></span><span>Build-Bench</span></a>
<button class="icon-button menu-button" type="button" aria-label="Open navigation" aria-expanded="false" aria-controls="site-nav" title="Open navigation" data-menu-button><i data-lucide="menu" aria-hidden="true"></i></button>
<nav class="site-nav" id="site-nav" aria-label="Primary navigation" data-nav>
<a href="index.html">Overview</a>
<a href="task.html">Challenge</a>
<a href="submission.html">Participate</a>
<a class="nav-emphasis" href="leaderboard.html"><i data-lucide="trophy" aria-hidden="true"></i>Leaderboard</a>
<a class="active" href="rules.html" aria-current="page">Rules</a>
<div class="nav-more" data-nav-more>
<button class="nav-more-toggle" type="button" aria-expanded="false" aria-controls="nav-more-menu" data-nav-more-toggle>
<span>More</span><span class="nav-more-caret" aria-hidden="true">▾</span>
</button>
<div class="nav-more-menu" id="nav-more-menu" data-nav-more-menu>
<a href="timeline.html"><span>Timeline</span><small>Competition dates and milestones</small></a>
<a href="faq.html"><span>FAQ</span><small>Common questions and participant support</small></a>
<a href="contact.html"><span>Contact</span><small>Competition contact and organizing team</small></a>
</div>
</div>
</nav>
</div>
</header>
<main id="main-content">
<section class="page-masthead rules-masthead">
<div class="content-wrap">
<div>
<span class="page-kicker">Participant rulebook</span>
<h1>Competition Rules</h1>
<p>These rules govern team registration, Agent submissions, evaluation integrity, and result verification for the Build-Bench Challenge.</p>
</div>
</div>
</section>
<div class="content-wrap page-layout rules-rulebook-layout">
<article class="page-document rules-document">
<header class="rules-introduction" id="acceptance">
<p class="rules-acceptance">REGISTERING A TEAM OR SUBMITTING AN AGENT CONSTITUTES ACCEPTANCE OF THESE COMPETITION RULES.</p>
<p class="rules-lead">Build-Bench is a skills-based software engineering competition. Teams develop repair Agents; organizers run qualified Agent versions on competition Cases and validate each proposed repair through an executable target-architecture build.</p>
<p>The Rules incorporate the versioned evaluation and scoring protocol and apply together with the Submission Guide, Timeline, and published data notices. Participants should read these materials before registering or starting an official evaluation.</p>
</header>
<section class="rules-section" id="scope">
<h2>1. Competition scope</h2>
<div class="rule-clause">
<h3>1.1 Competition title</h3>
<p>The competition is the Build-Bench Challenge, organized for the ICSE 2027 Competition Track.</p>
</div>
<div class="rule-clause">
<h3>1.2 Competition task</h3>
<p>A Team submits a runnable software repair Agent. For each Case, the Agent receives the released package materials and build-failure evidence, diagnoses the failure, and modifies the permitted package worktree. The organizer evaluates the resulting change in the official target environment.</p>
</div>
<div class="rule-clause">
<h3>1.3 Participant documentation</h3>
<p>The Challenge page defines the competition task and released Case sets. The Participate page defines the Agent package and runtime interface. Section 5 of these Rules defines Case execution, validation, outcomes, and scoring. The Timeline gives the official competition dates.</p>
<p class="rules-related-links"><a href="task.html">Challenge</a><a href="submission.html">Participate</a><a href="#evaluation-scoring">Evaluation and scoring</a><a href="timeline.html">Timeline</a></p>
</div>
</section>
<section class="rules-section" id="teams">
<h2>2. Team registration and eligibility</h2>
<div class="rule-clause">
<h3>2.1 Team registration</h3>
<ol class="rules-clauses" type="a">
<li>Each Team is registered by one Team leader, who provides the Team name and member information through the competition website.</li>
<li>A Team may contain no more than five members, including the Team leader.</li>
<li>An email address is required for every member and may not appear in more than one Team in this competition.</li>
<li>A person may participate through only one Team. Duplicate, misleading, or false registration information may be rejected or investigated.</li>
</ol>
</div>
<div class="rule-clause">
<h3>2.2 Team responsibilities</h3>
<ol class="rules-clauses" type="a">
<li>The Team leader manages the roster, uploaded Agent versions, and selection of versions for official evaluation.</li>
<li>The Team must keep its registration information accurate and complete before the applicable deadline.</li>
<li>Participants are responsible for confirming that their participation and use of third-party materials are permitted by their institution, employer, and applicable licenses.</li>
<li>Any actual or potential conflict of interest involving the organizers must be disclosed through the official support channel.</li>
</ol>
</div>
</section>
<section class="rules-section" id="submissions">
<h2>3. Agent submission and version control</h2>
<div class="rule-clause">
<h3>3.1 Conforming Agent bundle</h3>
<ol class="rules-clauses" type="a">
<li>The submitted ZIP must contain a runnable Agent source bundle that conforms to the current Submission Guide, manifest schema, and permitted runtime profile.</li>
<li>The entrypoint must be non-interactive and must complete without organizer or participant intervention.</li>
<li>The bundle must contain only the Agent code, declared dependencies, configuration, and supporting documentation needed to run it.</li>
<li>Secrets, personal credentials, hidden Case information, pre-generated Case repairs, caches, and previous run artifacts must not be included.</li>
</ol>
</div>
<div class="rule-clause">
<h3>3.2 Immutable versions</h3>
<ol class="rules-clauses" type="a">
<li>Every accepted upload creates an immutable Agent version identified by its submission record and content digest.</li>
<li>Any code, dependency, manifest, or configuration change requires a new upload and creates a new version.</li>
<li>A version must pass the platform checks and Hosted Smoke Test before it can be selected for Full Evaluation.</li>
<li>Uploading a version does not automatically start Full Evaluation. The Team must explicitly select a qualified version.</li>
<li>A later upload does not alter an active or completed evaluation snapshot.</li>
</ol>
</div>
<div class="rule-clause">
<h3>3.3 Organizer-run execution</h3>
<ol class="rules-clauses" type="a">
<li>The organizer starts an independent Agent instance for each Case under the published interface and resource policy.</li>
<li>The Agent may read the provided inputs and modify only the permitted worktree and output locations.</li>
<li>The Agent may not directly access the Docker socket, the final Validator, hidden evaluator files, or organizer infrastructure.</li>
<li>Stdout, stderr, declared result files, and platform events may be retained for diagnosis, auditing, and reproducibility.</li>
</ol>
</div>
</section>
<section class="rules-section" id="data-tools">
<h2>4. Competition data, models, and tools</h2>
<div class="rule-clause">
<h3>4.1 Competition materials</h3>
<ol class="rules-clauses" type="a">
<li>Released Cases, schemas, examples, logs, and Starter Kit materials may be used for participation subject to their published licenses and data notices.</li>
<li>Participants must preserve applicable copyright, attribution, redistribution, and open-source license obligations.</li>
<li>Unreleased Cases, hidden evaluator assets, and confidential organizer materials may not be obtained, used, shared, or redistributed.</li>
</ol>
</div>
<div class="rule-clause">
<h3>4.2 Models, external data, and tools</h3>
<ol class="rules-clauses" type="a">
<li>Teams may develop Agents using language models, prompting, retrieval, code search, static analysis, log processing, and other repair tools that comply with the published runtime and network policy.</li>
<li>External models, datasets, knowledge bases, APIs, and tools must be lawfully accessible to the Team and disclosed as required by Section 8.</li>
<li>Use of a third-party service does not transfer responsibility for compliance, reproducibility, cost, availability, or licensing to the organizers.</li>
<li>Participants must not embed API keys or personal credentials in an Agent bundle. Any organizer-managed credential mechanism will be described separately if external calls are permitted.</li>
</ol>
</div>
</section>
<section class="rules-section" id="evaluation-scoring">
<h2>5. Evaluation and scoring</h2>
<div class="rule-clause">
<h3>5.1 Successful repair and Per-Case validation</h3>
<ol class="rules-clauses" type="a">
<li>Each Case is an independent run. One slow or unsuccessful Case does not stop other scheduled Cases, and the selected Agent archive remains immutable throughout the evaluation.</li>
<li>The platform freezes the run inputs by recording the Agent archive and its checksum together with the Case-set, runtime, Validator, and rules versions.</li>
<li>The platform verifies the Case checksum and creates an isolated writable worktree from the original package materials.</li>
<li>The declared entrypoint receives the standard workspace and runs under the published runtime, time, resource, and network policy.</li>
<li>After the Agent exits, the platform validates <code>agent-result.json</code> and derives the canonical <code>repair.diff</code> from the original and modified worktrees.</li>
<li>The canonical patch is audited for forbidden paths, policy violations, required output structure, and prohibited build-bypass behavior.</li>
<li>The platform applies the audited patch to a clean copy of the Case and invokes the official Docker Validator in the target-architecture environment.</li>
<li>Agent status, Validator status, duration, patch statistics, and permitted logs are stored as structured evidence.</li>
<li>A repair is not compared textually with a reference patch. It succeeds only when the clean target build and expected artifact validation succeed.</li>
</ol>
</div>
<div class="rule-clause">
<h3>5.2 Terminal outcomes</h3>
<p>The platform records Agent execution and final build validation separately. An evaluation can complete normally even when Cases are unsuccessful; an evaluation-level System Error is reserved for organizer-controlled failures.</p>
<ol class="rules-clauses" type="a">
<li><code>succeeded</code> means the audited patch produces the expected package artifacts in the clean target build and is recorded as a successful Case.</li>
<li><code>failed</code>, <code>unresolvable</code>, and <code>no_fix</code> mean that the build fails, dependencies cannot be resolved, or the Agent proposes no allowed change; each is an unsuccessful Case.</li>
<li><code>agent_error</code> means the Agent crashes, exits abnormally, or omits a valid <code>agent-result.json</code>; it is an unsuccessful Case.</li>
<li><code>timeout</code> means the Agent or its permitted build work exceeds the applicable Case-level limit; it is an unsuccessful Case.</li>
<li><code>invalid_patch</code> means the canonical repair cannot be applied or violates path, output, or repair policy; it is an unsuccessful Case.</li>
<li><code>infrastructure_error</code> means an organizer-controlled Worker, storage service, runtime, or Validator fails independently of the Agent repair. It is not treated as a participant repair failure; the affected evaluation is reviewed or rerun.</li>
<li>No partial official score is published while unresolved infrastructure errors prevent the aggregate result from being finalized.</li>
</ol>
</div>
<div class="rule-clause">
<h3>5.3 Ranking and diagnostics</h3>
<p><strong>Verified Build Success Rate</strong> is the primary ranking metric: the number of successfully repaired Cases divided by the official evaluation denominator. The platform freezes an aggregate result only after all Cases have terminal outcomes and any organizer-controlled infrastructure errors have been resolved.</p>
<p><strong>Execution Time</strong> and <strong>Token Usage</strong> are reported as secondary efficiency metrics. Diagnostic information explains performance and failures but is not combined into a weighted score.</p>
<p>Published diagnostics may include:</p>
<ol class="rules-clauses" type="a">
<li>Case outcomes grouped by result category;</li>
<li>Agent execution time and final build duration;</li>
<li>token, model, and tool usage when collected;</li>
<li>patch size, modified-file count, and policy-validation status;</li>
<li>build requests, retries, and repair iterations; and</li>
<li>the Case-set, runtime, Validator, and rules versions used for the run.</li>
</ol>
<p>The final denominator definition and any tie-breaker will be published before public evaluation opens.</p>
</div>
<div class="rule-clause">
<h3>5.4 Evaluation stages and feedback</h3>
<p>The same Agent interface is used across stages, but the Case set, purpose, and visible feedback differ.</p>
<ol class="rules-clauses" type="a">
<li><strong>Hosted Smoke Test.</strong> Checks the uploaded bundle, entrypoint, dependencies, workspace behavior, and output contract on a small lightweight set. It runs the uploaded immutable version, returns detailed qualification diagnostics, and does not produce an official score.</li>
<li><strong>Full Evaluation during the public phase.</strong> Measures repair performance on the versioned validation set. It starts only after the Team selects a qualified immutable version; progress and the completed aggregate result appear in My Submissions.</li>
<li><strong>Hidden final evaluation.</strong> Runs the Team's frozen final Agent on organizer-controlled held-out Cases using frozen Agent, Case-set, runtime, Validator, and rules versions. Aggregate results and permitted diagnostics are released after the deadline.</li>
</ol>
</div>
<div class="rule-clause">
<h3>5.5 Versioned parameters</h3>
<p>Before public evaluation opens, the organizers will publish the exact score-denominator and cancellation semantics; any tie-breaker; Agent and build timeouts; build-feedback, iteration, and tool-call budgets; CPU, memory, storage, concurrency, network, and credential policies; submission frequency and rerun rules; and the feedback visible during public and hidden evaluation.</p>
<p>Current pilot settings do not define these competition limits. Publication dates appear on the <a href="timeline.html">Timeline</a>.</p>
</div>
</section>
<section class="rules-section" id="prohibited">
<h2>6. Prohibited conduct</h2>
<p>Participants must not obtain an advantage by bypassing the repair task, evaluator, resource policy, or Team-registration rules.</p>
<ol class="rules-clauses" type="a">
<li>Do not embed hidden solutions, leaked labels, Case-specific lookup tables, pre-generated repair patches, or other undisclosed answers in the Agent or its dependencies.</li>
<li>Do not exploit or attempt to escape the Agent runtime, Build Gateway, Validator, sandbox, authentication system, storage, scheduler, or network controls.</li>
<li>Do not suppress failures, delete essential tests, replace the package with a dummy artifact, or disable meaningful functionality merely to obtain a successful build result.</li>
<li>Do not alter immutable Case inputs, manifests, checksums, hidden metadata, evaluator files, result records, or any path outside the permitted worktree.</li>
<li>Do not use undeclared manual, remote, or third-party intervention to make Case-by-Case decisions during organizer-run evaluation.</li>
<li>Do not use multiple Teams, accounts, identities, or automated requests to evade registration, submission, evaluation, or resource limits.</li>
<li>Do not run unrelated workloads, mine cryptocurrency, interfere with other participants, or attempt to degrade competition services.</li>
</ol>
</section>
<section class="rules-section" id="hidden-evaluation">
<h2>7. Hidden evaluation and confidentiality</h2>
<ol class="rules-clauses" type="a">
<li>Hidden evaluation Cases remain under organizer control and are not distributed before final evaluation.</li>
<li>Official hidden evaluation uses a frozen Agent version, Case-set version, runtime image, Validator image, and protocol version.</li>
<li>Teams may use only the feedback intentionally released by the platform and must not probe the service to infer hidden Case contents, labels, reference repairs, or evaluator internals.</li>
<li>Organizers may inspect successful, unusually small, suspicious, or policy-sensitive repairs and may rerun an entry to verify its result.</li>
<li>Any accidental exposure of hidden or confidential material must be reported promptly and must not be used or shared.</li>
</ol>
</section>
<section class="rules-section" id="disclosure">
<h2>8. Method disclosure and reproducibility</h2>
<div class="rule-clause">
<h3>8.1 Required disclosure</h3>
<p>A final entry must describe the system that produced its result. The disclosure must identify, as applicable:</p>
<ol class="rules-clauses" type="a">
<li>base model names, providers, and versions when available;</li>
<li>major prompt templates or prompting strategy, repair-loop design, retrieval sources, static analyzers, and external tools;</li>
<li>training, fine-tuning, external datasets, private knowledge bases, and pre-processing used by the entry;</li>
<li>third-party APIs or services contacted during execution; and</li>
<li>known nondeterminism, caching, dependency, and reproducibility constraints.</li>
</ol>
</div>
<div class="rule-clause">
<h3>8.2 Verification materials</h3>
<ol class="rules-clauses" type="a">
<li>Teams must retain the submitted source, lockfiles, configuration, and method documentation needed to explain and rerun the evaluated version.</li>
<li>Organizers may request logs, configuration details, or a reproducibility demonstration for result review.</li>
<li>Any public solution description or solution paper must correspond to the Agent version that produced the reported official result.</li>
</ol>
</div>
</section>
<section class="rules-section" id="enforcement">
<h2>9. Review, correction, and enforcement</h2>
<ol class="rules-clauses" type="a">
<li>A malformed or nonconforming Agent, output, or patch may be rejected or recorded as unsuccessful under these Rules.</li>
<li>When an organizer-controlled infrastructure defect affects an evaluation, organizers may correct the defect and rerun the affected work under a documented procedure.</li>
<li>Organizers may request clarification or verification when an entry is unreproducible, incomplete, anomalous, or potentially noncompliant.</li>
<li>A result may be corrected, withheld, or removed when it cannot be verified or when the Team fails to provide required disclosure or reproducibility materials.</li>
<li>Deliberate hidden-answer use, build bypass, evaluator exploitation, credential abuse, false registration, or other serious misconduct may result in disqualification and removal from the leaderboard.</li>
</ol>
</section>
<section class="rules-section rules-publication" id="publication">
<h2>10. Versioned rules and pending parameters</h2>
<p>Several operational values require pilot measurements and organizer approval. Before public evaluation opens, the organizers will publish and version the following:</p>
<ol class="rules-clauses" type="a">
<li>upload, Hosted Smoke Test, and Full Evaluation frequency limits;</li>
<li>Case-level Agent and build timeouts, CPU, memory, storage, and build-feedback budgets;</li>
<li>network access, external API, and organizer-managed credential policies;</li>
<li>the official scoring denominator, tie-breaker, feedback visibility, cancellation, rerun, and correction procedures; and</li>
<li>any remaining eligibility, conflict-of-interest, or final-result verification procedures required by the ICSE Competition Track.</li>
</ol>
<p>Material updates will carry a version and publication date. The Timeline page governs competition dates, and the frozen version of these Rules governs official scoring.</p>
</section>
<nav class="next-page" aria-label="Next page"><span>Next</span><a href="timeline.html"><strong>Timeline</strong><small>Review the competition schedule</small><i data-lucide="arrow-right" aria-hidden="true"></i></a></nav>
</article>
<aside class="page-rail rules-outline" aria-label="Rules contents">
<strong>Rules</strong>
<nav>
<a href="#acceptance">Acceptance</a>
<a href="#scope">1. Competition scope</a>
<a href="#teams">2. Teams</a>
<a href="#submissions">3. Agent submissions</a>
<a href="#data-tools">4. Data, models, and tools</a>
<a href="#evaluation-scoring">5. Evaluation and scoring</a>
<a href="#prohibited">6. Prohibited conduct</a>
<a href="#hidden-evaluation">7. Hidden evaluation</a>
<a href="#disclosure">8. Disclosure</a>
<a href="#enforcement">9. Enforcement</a>
<a href="#publication">10. Versioned rules</a>
</nav>
</aside>
</div>
</main>
<footer class="site-footer"><div class="footer-inner"><div class="footer-brand"><span class="brand-mark" aria-hidden="true"><img src="assets/logo.svg" alt="" /></span><div><strong>Build-Bench Challenge</strong><small>ICSE 2027 Competition Track</small></div></div><nav class="footer-links" aria-label="Footer navigation"><a href="submission.html">Participate</a><a href="rules.html#evaluation-scoring">Evaluation & scoring</a><a href="timeline.html">Timeline</a><a href="faq.html">FAQ</a></nav><a href="#main-content">Back to top<i data-lucide="arrow-up" aria-hidden="true"></i></a></div></footer>
</body>
</html>