Skip to content

Add reproducible scorer reports for RCA benchmarks#6

Draft
RitwijParmar wants to merge 1 commit into
foam-ai:mainfrom
RitwijParmar:codex/foam-scorer-reports
Draft

Add reproducible scorer reports for RCA benchmarks#6
RitwijParmar wants to merge 1 commit into
foam-ai:mainfrom
RitwijParmar:codex/foam-scorer-reports

Conversation

@RitwijParmar

Copy link
Copy Markdown

Summary

This updates the 04-04-2026 RCA scorer so benchmark runs can produce an auditable run artifact instead of only per-output score files.

Changes:

  • add --model and RCA_SCORER_MODEL support so the judge model can be changed without editing code
  • add --dry-run to validate output discovery and answer-key coverage without making model calls
  • add --report-json with per-eval records, statuses, scores, errors, and aggregate summaries
  • add --strict so CI/manual runs can fail when an output cannot be scored
  • print aggregate summaries after each run

Why

Right now the scorer writes per-eval score files, while the aggregate benchmark tables are maintained separately. That makes it harder to reproduce a scoring run, inspect missing/empty/error cases, or attach a single machine-readable artifact to a benchmark result.

Verification

I intentionally used dry-run mode so no model calls were made:

  • /Users/ritwij/.bun/bin/bunx tsx scorer.ts --help
  • /Users/ritwij/.bun/bin/bunx tsx scorer.ts foam --dry-run --report-json=/tmp/foam-dry-run-report.json

Dry-run result:

  • 22 Foam outputs discovered
  • 22 answer keys found
  • 0 missing answers
  • 0 empty outputs
  • 0 errors
  • JSON report written successfully

Notes

No generated reports, scores, lockfiles, or install artifacts are included in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant