perf lab: retain and replay content-addressed experiment evidence

[wallstop]

Part of #500. Integrates with the frozen profile from #506.

Question

Can a reviewer reproduce every reported effect, interval, and decision after ordinary CI artifacts expire, and detect any missing or changed input?

Mechanism

The checked-in CSV is replaced and current large CI artifacts expire. Screenshots and copied summaries cannot establish provenance or reproduce a decision.

Contract

Evidence is append-only and content-addressed. Every derived value traces to raw inputs, source, tree, player, profile, workload, environment, and analysis hashes.

• Commit compact manifests, results, decisions, and retrieval instructions under progress/perf-experiments/<experiment-id>/.
• Publish each sealed large bundle as an immutable GitHub prerelease named perf-evidence-<experiment-id>-<manifest-sha8>.
• Never replace an asset. A correction creates a new revision with a new manifest hash.
• Never retain credentials, Unity license material, access tokens, or unreviewed personal/device identifiers.

Factors and workloads

Validate bundles for canonical paired throughput, allocation-heavy SubUnsub, cold latency, frame/queue latency, WPR/PMU/native mapping, and ARM64 energy captures. Exercise missing, duplicated, corrupted, reordered, schema-upgraded, inaccessible, and cross-platform-path cases.

Primary response

A clean restore reproduces the normalized machine-readable effect, interval, validity checks, and decision from the retained bundle.

Independent unit

One complete experiment bundle.

Effect threshold

Acceptance is exact: every declared artifact verifies against SHA-256, every required input is present, and deterministic reducers reproduce normalized output exactly. One changed byte or undeclared input must fail verification.

RED proof

• Modify one raw cycle, manifest, binary hash, or analysis file and prove replay fails.
• Remove a required artifact and prove replay names it.
• Make a GitHub prerelease asset inaccessible and prove the bundle is incomplete.
• Attempt to overwrite an experiment ID with different content and prove validation fails.
• Seed a credential fixture and prove publication refuses it.

GREEN suites

Pass schema, hash, replay, path-portability, secret-scanning, reducer, renderer, and workflow fixture tests on supported CI operating systems. Independently restore at least one real bundle from every artifact class.

Stop rule

Do not cite a result as campaign evidence if its bundle cannot replay. Mark a conclusion evidence-blocked when required large artifacts cannot be retrieved.

Immutable evidence

Each bundle contains the versioned manifest, raw rows, schedules/seeds, source/tree/profile/player hashes, commands/logs, diagnostics, binaries, PDBs, generated C++, assembly, ETL, reducer inputs/outputs, analysis lockfile, licenses, decision, retrieval metadata, and SHA-256 inventory.

Dependencies

All other performance-lab tasks depend on this task before declaring completion.

Completion checklist

• Define the versioned evidence schema.
• Define GitHub prerelease ownership and retention policy.
• Add hash, completeness, append-only, and secret validation.
• Add deterministic one-command replay.
• Add corruption, omission, inaccessibility, overwrite, and portability tests.
• Publish one representative bundle from every artifact class.
• Complete an independent clean-checkout restore drill.
• Link the evidence index from perf: calibrate the IL2CPP measurement system and attribute dispatch cost #500 and PLAN.md.

Complete copied-row validation in PR #552

PR #552, source 3e67087da13cf3d191a739d4342d204141de24c5, checks every field copied from a shipping cell into its matrix row. The previous reducer checked only selected metrics and accepted mismatched profile identity, missing or extra fields, and changed nested values.

Eight new altered-row cases fail against the original reducer. The new reducer passes all 220 bundle tests and preserves object-key-order independence and the producer's intentional cellId override. All 1,269 script tests pass locally.

A fresh download of real shipping artifact 9890522458 from run 33739376165 matched archive SHA-256 a2dffe392a0509638cb8a4cc86bd202ad7c85abb4c47701c24817da2946c6d88. Both reducers accept all 20 cells and produce identical normalized output, SHA-256 23d7e44d5929561c2eec2c52af0dfd0c0e250b0431ce579c0e5e8df7c813dcf3. The comparison reads original bytes; it does not publish or replace any artifact. The downloaded evidence remains ordinary expiring CI evidence.

All PR checks pass at the same source, including script and repository CI, all four Unity versions, IL2CPP performance, and both devcontainer architectures. No unresolved review threads remain. This closes a copied-row integrity gap, not the six campaign artifact classes, immutable publication, or independent sealed-bundle restoration acceptance.

Paired screen retention in PR #554

PR #554, source 5afd1aabc2470d8bcf721f66a9a9f28a10d2e818, adds paired-throughput-screen-v1 for the explicitly exploratory paired-throughput-screen artifact class. It calls the existing production bracket reducer over the exact declaration bytes and three retained summaries with raw cycle ratios. First/center/last positions, declaration digests, source relationships, execution profiles, and raw-cycle consistency are validated. The envelope source commit must match the first measured run during sealing and replay.

Accepted, rejected, and uninterpretable decisions all replay without filtering unfavorable results. Tests reject missing files, malformed inputs, altered cycles, swapped positions, changed bytes, and forged source identity. All 294 focused tests and 1,273 script tests pass. Twenty-five existing mutation cases were consolidated without changing their names, inputs, or assertion patterns, keeping the JavaScript budget unchanged.

A real historical seven-row session-247 bracket reproduces its original uninterpretable result exactly. The local four-file bundle digest is c835487bcde331429aa7f6f429124b3d6f4dc8d5296db835f218d33fc47d76ef. This is local verification, not a remote immutable artifact. The restore instructions now include pinned npm dependencies required by the XML/YAML credential scanner.

Native binaries, complete campaign classes, confirmatory intervals, immutable publication, and independent remote restoration remain open. Floating-point replay requires exact normalized output and fails closed on arithmetic differences. All checks pass at this source, including script CI on Windows, macOS, and Linux, the four-version Unity matrix, and IL2CPP performance. Independent implementation and artifact reviews are clear. The native jobs preserve their full prior outcome multisets; their green state does not turn the exploratory replay into a complete campaign artifact class.

Checklist scope audit

The checked mechanisms cover the schema-1 envelope and implemented shipping-fidelity matrix and paired-throughput screen classes. Current-source script CI passes on all three operating systems. The real local seven-row paired screen reproduces its original uninterpretable result, and the 294 focused tests exercise the production validators/reducers. These checks do not complete every raw/native campaign class, a denied-access drill against the required remote release asset, immutable publication, independent remote restoration, or a durable evidence index. The combined campaign corruption/omission/inaccessibility/portability requirement therefore remains unchecked.

The durable publication contract defines maintainer ownership, indefinite revision retention, and withdrawal handling. That definition is complete. Approval and implementation of publication, the durable index, and independent remote restoration remain behind #521; no publication approval is inferred.

Raw allocation observations in PR #555

PR #555, source fc3927cf63f7061b01b8808cf3aa2a6b1fdcc930, adds the partial allocation-subunsub-observations class and allocation-subunsub-observations-v1 reducer. Sealing and replay require the original comparison output plus canonical eight-column CSV. The production extractor must reproduce the CSV exactly. The reducer validates source, platform, execution scope, unique runs, safe integer measurements, positive timing windows, and independent -1 unmeasured sentinels.

All 24 new cases pass, including rejection of missing, changed, duplicated, malformed, and contradictory inputs. The full local script suite passes all 1,298 cases. Independent production verify/replay also accepts a real local Unity observation and its byte-identical restored copy.

That observation records 20,000 GC allocation calls and 3,920,000 informational bytes, with a 5002.930 ms timing window. Its recorded source is 857c293c2e03945383652b5c269e00bbf698a11a, on Unity 6000.4.6f1 Editor PlayMode Mono. The 446-byte raw log hashes to bbd6f5dcbfc4c8cadd077ad38090e7282b434942d9dbad42552faa5dff145f70; the 263-byte CSV hashes to b517be391ea8ec8db4c3822570bf0495d95494647d6551499e5cf3eda890f257. Local bundle digest: a1638bafeddbcba88869424cf9fcfd8a3124e1d14e2cd539c05f6ba4d5bee7ac.

The native result passes with matching run/cleanup identity and a clean restored scene. This is local observation retention, not per-operation allocation proof, IL2CPP parity, full campaign-class completion, independent host replication, or immutable remote restoration. The CSV does not carry a probe denominator. Final-source script CI and native acceptance remain pending.

[wallstop]

PR #524 fixes a reproduced classification defect in e262c00: the sealer accepted real shipping evidence labeled arm64-energy because artifactClass was not bound to its reducer. Seal, manifest writes, verify, and replay now require the registered class/reducer pair, even after an attacker recomputes the manifest digest. Six unsupported-class cases and inherited object-property reducer names are covered.

The existing 441-file shipping bundle still verifies and replays all 20 cells unchanged. Binary refusal remains intact. The local artifact audit found no measured player/PDB/ETL payloads. All six campaign classes still need complete raw-input, provenance, format-aware privacy, reducer, and independent-restore acceptance. This fix must not be counted as native evidence support.

[wallstop]

PR #528 fixes shipping reducer integrity in 68a460f. Missing raw cells, duplicate summaries, wrong schemas/types/counts, omitted fields and contradictory timing previously passed or lost information. The reducer now requires complete one-to-one successful-cell evidence and exact summaries. Explicit failed/unreadable cells can retain their actual malformed raw output for diagnosis; unknown cells still fail closed.

Red-green fixtures cover the corrupt-input class and real partial-run behavior. All 169 focused bundle tests and 1,086 frozen-tree Node 22.18.0 tests pass without skips. The JavaScript line budget remains unchanged and passing.

Recovered real shipping data still verifies/replays all 441 files to the exact original 20-cell normalized result and bundle digest 214a6c81360d04185aa89715c78490cb9a294fac9c289a21dc4127fe6a779b59. This proves the tighter reducer accepts valid historical evidence.

Six campaign artifact classes, native binary capture/inspection, their reducers, immutable publication and independent remote restore remain open. The shipping text bundle is not a substitute for those requirements.

[wallstop]

Independent clean-checkout restore drill

The "Complete an independent clean-checkout restore drill" checklist item is now executed against a real retained bundle.

Drill inputs: the real allocation-subunsub-observations bundle from session 271 (session271-subunsub-measured, bundle digest a1638bafeddbcba88869424cf9fcfd8a3124e1d14e2cd539c05f6ba4d5bee7ac, source 857c293c2e03945383652b5c269e00bbf698a11a), restored from a fresh git clone --no-hardlinks of the repository at 73627f7e with a fresh npm ci dependency install (287 packages). The bundle files live outside the checkout and were copied in as retained evidence, mirroring a real restore.

Results, all inside the clean checkout with the production scripts/unity/perf-evidence-bundle.js:

• Replay of the unmodified bundle reproduces the normalized row exactly (20000 GC allocation calls, 3,920,000 informational bytes, 451735.301 emits/s, 5002.930 ms window) and exits 0.
• One appended byte in comparison-baseline.csv: exit 1 with comparison-baseline.csv is 264 bytes but the manifest declares 263.
• Removed comparison-output.log: exit 1.
• Restored copy: exit 0 again.

Scope caveat kept explicit: the drill runs on the same host and identity as the sealing session, so it proves checkout, dependency, and replay-path independence, not a second machine or a remote immutable release. Immutable publication and independent remote restoration remain behind the #521 approval gate, and the inaccessibility drill against a real release asset stays gated with it.

[wallstop]

The retained session 271 SubUnsub bundle also replays with #559's current production reducer and updated tooling dependencies (ajv 8.20.0). The normalized result exactly matches the sealed manifest, and replay exits successfully.

This is an additional compatibility check of the existing same-host, fresh-checkout drill described above. It does not complete independent remote restoration, native binary retention, the remaining campaign artifact classes, or immutable publication. Those requirements and the #521 approval gate remain unchanged.

[wallstop]

PR #562 at source cf417d186206043e39e4489242ac288ffd9d263a supplies two prerequisite identities for future sealed evidence: the exact measured source and the canonical internal launcher digest.

The final Performance Numbers run passes. Internal artifact 10082627168 has archive SHA-256 9a34552bad7b5f7a354da59de472fdeebaff543ab8c065b297887dd8f38ebde3. Its player-size.json binds the source to DxmTestPlayer.exe, 667,648 bytes, with SHA-256 EAD801447B5CA90778F6CF2F35AAB805BCBC57F404523399430CDB8474015A02.

This is identity input only. The launcher bytes are not retained, so a restored artifact cannot re-hash them. The manifest is an ordinary expiring workflow artifact. This change adds no reducer, sealed bundle, immutable release, corruption or inaccessibility drill, or independent restore. No #508 checklist item is complete.