ci: onboard leaderboard backfill replay caller (ENG-5992)

[nsportsman]

Summary

Onboards the operator-dispatched leaderboard backfill replay caller (ENG-5992, part of the ENG-5973 fleet-audit follow-up):

• .github/workflows/leaderboard-backfill-caller.yml (new) — dispatch-only (workflow_dispatch) caller of the leaderboard-backfill.yml reusable, replicated from the template already merged in nerva and caeruleus. Merging this enables replay; it never runs automatically, and no dispatch happens as part of this PR.

This repo has no .github/dependabot.yml, so no dependabot change is needed; the workflow header documents the ignore requirement should one ever be added.

Why

The 2026-08-09 fleet audit found 2 replay-eligible PRs in this repo with no delivery run at all (#136, #137 — merged before onboarding). Replaying them requires this caller.

Do not bump the pin

uses: …leaderboard-backfill.yml@6c715ccf351a09bfcb916aede4c542e437e5f31d # v2.17.0 is the exact SHA on the IAM trust list. Bumping it alone fails closed at AssumeRole with no local signal — a bump follows the ENG-4164 runbook (trust updated and deployed first).

Gates

• Premise gate: SKIP recorded — template replication of an already-merged mechanism; no non-trivial machinery (size-and-behavior condition).
• Shell review-after gate: passed pre-PR — shell-reviewer verdict REVIEW_APPROVED; everything from name: down verified byte-identical (sha256) to the live nerva/caeruleus canonical; actionlint clean; this repo's verify-pins.yml passes on the SHA-pinned uses:.
• Provenance: the workflow file is entirely new.

Ticket: ENG-5992 (blocks the ENG-5690 replay work).

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 25 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7e932b72-65b6-496b-9133-aebdff88f760

📥 Commits

Reviewing files that changed from the base of the PR and between 339b868 and 9fa0cc0.

📒 Files selected for processing (1)

• .github/workflows/leaderboard-backfill-caller.yml

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Codex Review

No critical issues — LGTM pending human review.

---

Reviewed by Codex (gpt-5.6-sol)

[github-actions]

Gemini Review

No critical issues — LGTM pending human review.

---

Reviewed by Gemini (gemini-3.1-pro-preview)

[github-actions]

Claude Review

Critical issues

• capability_map omission may break the contract / mis-attribute. The live sibling leaderboard-metrics.yml in this repo passes capability_map: '[{"path": ".", "name": "trajan"}]' explicitly. This new caller omits it, justified by the comment "unset means derive it from this repo's own live leaderboard-metrics.yml." But the live caller does not derive anything — it hardcodes the value — so "attributes capabilities exactly as live runs do" is unverifiable from this repo and inconsistent with the only sibling we can inspect. If leaderboard-backfill.yml@v2.17.0 does not actually derive-from-live (no default, or a different derivation), replaying #136/#137 either fails at runtime or attributes capabilities differently than the live run. Please confirm against the reusable's actual input contract before merge; if in doubt, pass the same explicit capability_map the live caller uses.
• The v2.17.0 version comment on the pin is self-guarded by verify-pins.yml (fails CI on a dishonest tag-to-SHA mapping). Merge should be gated on that check going green rather than on the PR description's sha256 assertion.

Security

• This extends id-token: write (OIDC AssumeRole) to a new external reusable (leaderboard-backfill.yml, v2.17.0) whose source is not in this repo and can't be reviewed here. It is the same praetorian-inc/public-workflows org already trusted by the merged metrics caller, but a different reusable and a newer version — the whole trust boundary rests on the SHA pin + org control of that ref. Worth an explicit sign-off that v2.17.0 of the backfill reusable itself was reviewed, not merely that its SHA sits on the IAM trust list.
• Trigger is workflow_dispatch-only (requires write access) and pr_numbers flows only into a reusable with: input, not a run: step here — no injection surface in this file, and no secrets in it.

Test coverage

Not applicable — pure GitHub Actions YAML, no *.go/*.ts/*.py production code.

[nsportsman]

Adjudication of automated review findings

Claude finding 1 — "capability_map omission may break the contract / mis-attribute": REFUTED.
"The live caller does not derive anything — it hardcodes the value" is true but is not in tension with the omission: the reusable's derivation, at the pinned SHA (6c715ccf), checks out this repo's default branch and yq-parses .github/workflows/leaderboard-metrics.yml, reading the live caller's with.capability_map literal verbatim (reusable lines ~540–579). For trajan that returns the byte-exact [{"path": ".", "name": "trajan"}]. Unparseable/ambiguous/non-string cases hard-fail the run instead of falling through to [], so replaying #136/#137 either attributes exactly as live runs do or fails loudly — it cannot silently diverge. Passing the explicit map instead is what the template header warns against: it freezes today's value and drifts silently if the live one ever changes.

Claude finding 2 — "merge should be gated on verify-pins going green, not the PR description": CONFIRMED, and satisfied.
verify / Verify Action Pins ran on this PR's head (9fa0cc08) and completed success.

Claude security note — "explicit sign-off that v2.17.0 of the backfill reusable itself was reviewed": answered.
The backfill reusable is org-controlled (praetorian-inc/public-workflows), merged through that repo's own PR review gates under ENG-5688, and is the same artifact already dispatched successfully in production by caeruleus (two green workflow_dispatch runs on 2026-08-04 at exactly this SHA — which also demonstrates the IAM trust-list membership empirically, since AssumeRole succeeded). This PR extends no trust beyond that precedent.

No fix commits needed — no finding survived confirmation as a defect. Merge remains a human decision.