Project Benchmark Harness

[jabrena]

User Story: US-1012 Project Benchmark Harness

As a Plinth maintainer
I want to run a reproducible project benchmark across agent tools, models, and Plinth configuration levels
So that I can compare time, token usage, and cost when solving the same God Analysis API implementation problem with different levels of requirements, raw OpenSpec (official skill), and Plinth command-driven refinement

Problem

Today there is no shared way to measure how Plinth commands, agents, and skills affect agent work across IDEs/CLIs and models. This issue defines a matrix-style evaluation for one fixed domain problem (God Analysis API), not the existing JVM /benchmark command (JMeter/Gatling/JMH).

Location

Benchmark harness lives under plinth-benchmark/.

plinth-benchmark/
├── README.md
├── scenario1/
│   ├── README.md
│   ├── specs/
│   └── gherkin/
├── scenario2/
│   ├── README.md
│   ├── specs/
│   └── gherkin/
├── scenario3/
│   ├── README.md
│   ├── specs/
│   └── gherkin/
└── scenario4/
    ├── README.md
    ├── specs/
    └── gherkin/

Each scenario folder contains:

• specs/ — scenario requirements and protocol
• gherkin/ — acceptance criteria for that scenario

Scope

Fixed problem domain

God Analysis API (GET /api/v1/gods/stats/sum) using the repository example under examples/openspec/god-analysis-api/.

Benchmark scenarios

Scenario	Intent	Harness path	Seed / input (repo-relative)
Scenario 1	Minimal notes only	plinth-benchmark/scenario1/	examples/openspec/god-analysis-api/requirements/problem1/README.md
Scenario 2	All requirements notes	plinth-benchmark/scenario2/	examples/openspec/god-analysis-api/requirements/problem1/
Scenario 3	Raw OpenSpec + official OpenSpec skill	plinth-benchmark/scenario3/	examples/openspec/god-analysis-api/openspec/changes/
Scenario 4	Plinth commands + refinement → generated OpenSpec → implement	plinth-benchmark/scenario4/	Seed: examples/openspec/god-analysis-api/requirements/problem1/README.md; do not use examples/openspec/god-analysis-api/openspec/changes/ as input

Acceptance criteria for each scenario are defined in:

Scenario	Specs	Gherkin
Scenario 1	plinth-benchmark/scenario1/specs/scenario1.md	plinth-benchmark/scenario1/gherkin/scenario1.feature
Scenario 2	plinth-benchmark/scenario2/specs/scenario2.md	plinth-benchmark/scenario2/gherkin/scenario2.feature
Scenario 3	plinth-benchmark/scenario3/specs/scenario3.md	plinth-benchmark/scenario3/gherkin/scenario3.feature
Scenario 4	plinth-benchmark/scenario4/specs/scenario4.md	plinth-benchmark/scenario4/gherkin/scenario4.feature

Scenario 2 includes the full notes set under the problem1 directory (user story, Gherkin feature, ADRs, OpenAPI, and related requirement assets). Scenario 3 uses the checked-in OpenSpec change tree as a raw OpenSpec input and applies the official OpenSpec skill (Fission-AI OpenSpec / OPSX), without Plinth analysis-design commands. Scenario 4 starts from the same README seed as Scenario 1, then uses Plinth commands to refine, create OpenSpec via /create-spec / @042-planning-openspec, and implement with /implement-spec.

Plinth configuration levels

1. Full Plinth — all commands, agents, and skills available (required for Scenario 4)
2. No OpenSpec — without any OpenSpec change/spec workflow (especially relevant for Scenario 1 and Scenario 2; not applicable to Scenario 3 or Scenario 4)
3. Bare agent — no Plinth assets; delegate only to the agent (not applicable to Scenario 4)

For Scenario 3, Plinth analysis-design commands and @042-planning-openspec are out of band: the run uses the official OpenSpec skill against the raw change under examples/openspec/god-analysis-api/openspec/changes/.

Agent tools

• Cursor
• Codex
• GitHub Copilot
• Claude Code

Models

• Multiple models (exact catalog TBD in implementation / follow-up ADR)

Metrics (required)

Measure three layers per scenario execution. Rank cost (or tokens) among runs with acceptance_pass = true; keep failures in a separate table. Canonical scorecard: plinth-benchmark/README.md.

Efficiency

Metric	Definition
wall_clock_s	Wall-clock seconds from start until done (human stop or acceptance gate)
active_agent_s	Sum of model/tool session time, excluding idle waits
tokens_in / tokens_out / tokens_total	Prompt, completion, and total tokens (include tool-call tokens when billed)
cost_usd	Estimated cost from provider pricing for the selected model

For Scenario 4, also record efficiency metrics per phase (/update-issue, /create-spec, /implement-spec).

Outcome quality

Metric	Definition
acceptance_pass	Boolean: pass against the scenario product acceptance criteria
acceptance_coverage	Fraction of scenario Gherkin / OpenSpec tasks satisfied (0–1)
rework_turns	Extra human correction turns before pass
artifact_completeness	Checklist score (0–1) for required outputs (API, tests, config, OpenAPI, and so on)

Process / protocol (comparability)

Metric	Definition
scenario	scenario1 … scenario4
case_id	case-1-readme-only … case-4-plinth-commands-openspec-refinement
tool	Cursor, Codex, GitHub Copilot, or Claude Code
model	Model identifier
plinth_config	Full Plinth, No OpenSpec, or Bare agent
commit	Git SHA of harness / Plinth revision used for the run
retry_count	Aborted or restarted attempts for this cell
human_intervention_min	Minutes of human steering

Minimal v1 scorecard (required)

Per execution, record at least:

1. wall_clock_s, tokens_total, cost_usd
2. acceptance_pass
3. rework_turns
4. Labels: scenario, case_id, tool, model, plinth_config, commit

Scenario-specific extras

Scenario	Extra metrics / flags
Scenario 1 / Scenario 2	Same core scorecard; compare rework_turns when notes richness changes
Scenario 3	openspec_validate_ok; official_openspec_skill_only (must be true)
Scenario 4	Phase-split time/tokens/cost; generated_openspec_present (must be true)

Do not treat JVM latency percentiles or JMH-style microbenchmarks as part of this harness (that belongs to the /benchmark command).

Acceptance Criteria

Execute and assess each scenario using its harness Gherkin feature. Paths are repository-relative.

Feature: Plinth project benchmark — God Analysis API harness

Background:
  Given the benchmark harness lives under plinth-benchmark/
  And each scenario provides specs/ and gherkin/ acceptance artifacts
  And the fixed problem domain is the God Analysis API example
  And metrics capture for time, tokens, and cost
  And results are attributable to tool, model, configuration level, and case id

@acceptance-test
Scenario: Scenario 1 harness is ready
  Given plinth-benchmark/scenario1/specs/scenario1.md exists
  And plinth-benchmark/scenario1/gherkin/scenario1.feature exists
  When the scenario1 acceptance feature is executed for a benchmark run
  Then the run follows case-id "case-1-readme-only"
  And the only requirements input is examples/openspec/god-analysis-api/requirements/problem1/README.md

@integration-test
Scenario: Scenario 2 harness is ready
  Given plinth-benchmark/scenario2/specs/scenario2.md exists
  And plinth-benchmark/scenario2/gherkin/scenario2.feature exists
  When the scenario2 acceptance feature is executed for a benchmark run
  Then the run follows case-id "case-2-all-requirements-notes"
  And the requirements input is examples/openspec/god-analysis-api/requirements/problem1/

@integration-test
Scenario: Scenario 3 harness is ready
  Given plinth-benchmark/scenario3/specs/scenario3.md exists
  And plinth-benchmark/scenario3/gherkin/scenario3.feature exists
  When the scenario3 acceptance feature is executed for a benchmark run
  Then the run follows case-id "case-3-raw-openspec-official-skill"
  And the primary input is examples/openspec/god-analysis-api/openspec/changes/
  And Plinth /create-spec, /implement-spec, and @042-planning-openspec are not used

@integration-test
Scenario: Scenario 4 harness is ready
  Given plinth-benchmark/scenario4/specs/scenario4.md exists
  And plinth-benchmark/scenario4/gherkin/scenario4.feature exists
  When the scenario4 acceptance feature is executed for a benchmark run
  Then the run follows case-id "case-4-plinth-commands-openspec-refinement"
  And Full Plinth is required
  And the seed input is examples/openspec/god-analysis-api/requirements/problem1/README.md
  And examples/openspec/god-analysis-api/openspec/changes/ is not used as input

Detailed Given/When/Then for each run live in the scenario gherkin/*.feature files listed above.

Scenario input checklist

Scenario 1 — README only

• Harness: plinth-benchmark/scenario1/
• Input: examples/openspec/god-analysis-api/requirements/problem1/README.md
• Intent: measure behavior with minimal structured notes

Scenario 2 — all requirements notes

• Harness: plinth-benchmark/scenario2/
• Input root: examples/openspec/god-analysis-api/requirements/problem1/
• Intent: measure behavior with full requirements packaging, without OpenSpec

Scenario 3 — raw OpenSpec + official OpenSpec skill

• Harness: plinth-benchmark/scenario3/
• Input root: examples/openspec/god-analysis-api/openspec/changes/
• Workflow: official OpenSpec skill / OPSX apply path, not Plinth /create-spec, /implement-spec, or @042-planning-openspec
• Intent: measure implementing from a checked-in raw OpenSpec change using the official OpenSpec skill alone

Scenario 4 — Plinth commands + refinement → OpenSpec → implement

• Harness: plinth-benchmark/scenario4/
• Seed input: examples/openspec/god-analysis-api/requirements/problem1/README.md
• Forbidden input: examples/openspec/god-analysis-api/openspec/changes/
• Required configuration: Full Plinth
• Command path: /update-issue → optional /explore-design / /create-adr → /create-spec → optional /review-alignment → /implement-spec
• Intent: measure the full Plinth refine-and-specify path versus Scenario 1 and Scenario 3

Out of scope

• JVM application performance testing via /benchmark (JMeter, Gatling, JMH)
• Claiming statistical significance without a documented sample size and protocol
• Expanding beyond the God Analysis API example until these four scenarios have a baseline campaign

Notes

• Distinguish this work from benchmark command ownership under @robot-java-performance.
• Scenario 1 and Scenario 2 isolate requirements richness.
• Scenario 3 isolates raw OpenSpec + official OpenSpec skill delivery.
• Scenario 4 isolates Plinth command-driven refinement and OpenSpec generation before implementation.
• Likely follow-ups: protocol ADR, result schema, and where reports live (docs vs CI artifact).

INVEST Validation

• Independent: Scenarios can be run and compared independently per tool/model/config cell
• Negotiable: Model catalog, optional design/ADR steps in Scenario 4, official OpenSpec skill install details per tool, automation depth, and report location remain open
• Valuable: Enables evidence-based comparison of minimal notes vs full notes vs official OpenSpec skill vs Plinth-refined OpenSpec
• Estimable: Fixed problem domain and four harness folders bound the work
• Small: First campaign can use one tool and one model across the four scenarios
• Testable: Each scenario has specs/ and gherkin/ plus recorded metrics and outcome assessment

Traceability

• GitHub issue: Project Benchmark Harness #1012
• Harness root: plinth-benchmark/

[jabrena]

• https://arxiv.org/abs/2602.11988