Skip to content

world_class_evidence_plan.md

Latest commit

 

History

History
184 lines (138 loc) · 9.56 KB

world_class_evidence_plan.md

File metadata and controls

184 lines (138 loc) · 9.56 KB
Raw

World-Class Evidence Plan

Generated at: 2026-08-17

Summary

  • decision: collect-external-evidence
  • audit decision: continue-iteration
  • ready to claim world-class: false
  • ledger completion required: true
  • evidence requirements: 4
  • tasks: 4
  • human tasks: 1
  • external tasks: 3

This report is an execution plan for the remaining world-class evidence gaps. It does not count a plan or source-report pass as completion; the ledger must still validate accepted submissions.

Task Table

Task Status Category Owner Current
provider-holdout pass external operator with provider credentials phase1 model-executed 40/40; calls 40/40; status completed
human-adjudication pass human human reviewer phase1 reviewers 3/3; pairs 20/20; promotion eligible
native-permission-enforcement external_required external target client or installer integrator native-enforced targets 0; installer-enforced targets 4
native-client-telemetry external_required external Browser/Chrome/IDE/provider client integrator external source events 0; adoption samples 0

Provider Holdout

  • objective: Complete the fixed 10-case DeepSeek Flash+Pro matrix with 40 real calls and governed budget evidence.
  • audit next action: Run evidence-build with DEEPSEEK_API_KEY and keep raw outputs in the isolated run directory.

Runbook

  • Set DEEPSEEK_API_KEY in the operator shell; never commit or print the value.
  • python3 scripts/yao.py evidence-build . --run-id <PROVIDER_RUN_ID> --self
  • Keep the generated private answer key and role-neutral review materials inside .yao/runs/<PROVIDER_RUN_ID>.
  • python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
  • Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
  • python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

Success Checks

  • reports/provider_output_evaluation.json summary.call_count == 40
  • reports/provider_output_evaluation.json summary.model_executed_count == 40
  • reports/provider_output_evaluation.json summary.failure_count == 0
  • reports/provider_output_evaluation.json summary.total_tokens <= 250000
  • reports/skill_os2_audit.json item provider-holdout status becomes pass

Evidence Artifacts

  • evals/output/provider_matrix.json
  • reports/provider_output_evaluation.json
  • reports/provider_output_blind_pack.json
  • reports/provider_output_answer_commitment.json
  • reports/skill_os2_audit.json
  • evidence/world_class/intake.schema.json
  • evidence/world_class/templates/provider-holdout.intake.json
  • reports/world_class_evidence_intake.json
  • reports/world_class_evidence_intake.md

Privacy Contract

  • Do not commit provider credentials or environment dumps.
  • The output execution report records output hashes and aggregate run metadata, not raw provider prompts.

Human Adjudication

  • objective: Collect three controlled, independent reviews of the same 20-pair provider blind pack.
  • audit next action: Collect three controlled reviewer packets and adjudicate them against the private run answer key.

Runbook

  • Give each registered reviewer an independent copy of the matching provider_review_reviewer-*.json template and the role-neutral blind pack.
  • Collect all 20 A/B choices, reasons, controlled submission ids, timestamps, and truthful independent-review attestations.
  • Export an access-controlled reviewer registry that binds each reviewer id to the exact packet SHA256.
  • python3 scripts/yao.py evidence-finalize-review . --source-run <PROVIDER_RUN_ID> --decisions <reviewer-a.json> --decisions <reviewer-b.json> --decisions <reviewer-c.json> --reviewer-registry <registry.json> --run-id <FINAL_RUN_ID> --self
  • python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
  • Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
  • python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

Success Checks

  • reports/provider_output_adjudication.json summary.reviewer_count == 3
  • reports/provider_output_adjudication.json summary.pair_count == 20
  • reports/provider_output_adjudication.json summary.failure_count == 0
  • reports/provider_output_adjudication.json evidence_binding.blind_pack_sha256 matches the source run
  • reports/skill_os2_audit.json item human-adjudication status becomes pass

Evidence Artifacts

  • reports/provider_output_blind_pack.json
  • reports/provider_reviewer_registry.json
  • reports/provider_output_adjudication.json
  • reports/provider_review_lineage.json
  • scripts/adjudicate_multi_reviewer.py
  • scripts/finalize_provider_review.py
  • evidence/world_class/intake.schema.json
  • evidence/world_class/templates/human-adjudication.intake.json
  • reports/world_class_evidence_intake.json
  • reports/world_class_evidence_intake.md

Privacy Contract

  • Reviewer packets contain choices, reasons, hashes, and controlled submission metadata without raw prompts or answer-key roles.
  • The private answer key remains under .yao/runs and is opened by the finalizer after all controlled packets are fixed.
  • The adjudication and lineage artifacts preserve blind_pack_sha256 and answer_key_sha256 commitments.

Native Permission Enforcement

  • objective: Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.
  • audit next action: Integrate a real target-client or external installer runtime guard before claiming native permission enforcement.

Runbook

  • Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
  • Update the generated target adapter only when the guard is actually enforced by that target.
  • python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip --self
  • python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation --self
  • python3 scripts/yao.py runtime-permissions . --package-dir dist --self
  • python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
  • Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
  • python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

Success Checks

  • reports/runtime_permission_probes.json summary.native_enforcement_count > 0
  • reports/runtime_permission_probes.json summary.failure_count == 0
  • reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
  • reports/skill_os2_audit.json item native-permission-enforcement status becomes pass

Evidence Artifacts

  • dist/targets/*/adapter.json
  • reports/runtime_permission_probes.json
  • reports/runtime_permission_probes.md
  • reports/install_simulation.json
  • reports/install_simulation.md
  • security/permission_policy.json
  • evidence/world_class/intake.schema.json
  • evidence/world_class/templates/native-permission-enforcement.intake.json
  • reports/world_class_evidence_intake.json
  • reports/world_class_evidence_intake.md

Privacy Contract

  • Do not mark native_enforcement true for metadata-only fallbacks.
  • Keep residual risks visible for targets that still rely on operator enforcement.

Native Client Telemetry

  • objective: Import production metadata-only events from a real external client into the local drift loop.
  • audit next action: Install a real client against the native host and import production metadata-only events.

Runbook

  • python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
  • Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
  • python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --self
  • python3 scripts/yao.py skill-atlas --workspace-root . --self
  • python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD> --self
  • Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
  • python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions --self

Success Checks

  • reports/adoption_drift_report.json summary.source_types.external > 0
  • reports/adoption_drift_report.json summary.adoption_sample_count > 0
  • reports/skill_os2_audit.json item native-client-telemetry status becomes pass

Evidence Artifacts

  • reports/adoption_drift_report.json
  • reports/adoption_drift_report.md
  • reports/telemetry_hook_recipes.json
  • scripts/telemetry_native_host.py
  • evidence/world_class/intake.schema.json
  • evidence/world_class/templates/native-client-telemetry.intake.json
  • reports/world_class_evidence_intake.json
  • reports/world_class_evidence_intake.md

Privacy Contract

  • Telemetry must remain metadata-only and local-first.
  • Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.