Add an admission manifest for generated-question data and answer-leakage canaries

[sunghunkwag]

Summary

Agent0's released pipeline has a very clean self-evolution boundary:

1. the curriculum agent generates question + answer pairs in curriculum_train/question_generate/question_generate.py;
2. question_evaluate/evaluate.py asks the executor model to solve each generated question, computes a self-consistency-style score, and writes {question, answer, score, results};
3. question_evaluate/upload.py filters by min_score <= score <= max_score and emits train.parquet;
4. the executor is then trained from that accepted generated-data slice.

I think it would be useful to add a small machine-readable admission report for this generated-data gate, including contamination and canary checks around generated answers.

This is not a bug report. The proposal is to make the zero-data self-evolution data boundary easier to audit and reproduce.

Why this matters

Agent0's central claim is zero-data self-evolution. In that setting, the curation gate is a critical verifier:

• generated questions should not be near-duplicates of known benchmarks or famous contest problems;
• generated answers should be used as grader/reference fields, not accidentally exposed as prompt text to the executor;
• accepted samples should be accepted because they are useful frontier tasks, not because the generator produced an answer artifact that later leaks into training-visible text;
• rejected samples should leave an audit trail explaining whether they failed parsing, grading, self-consistency, duplicate checks, or score thresholds.

The current code already exposes a natural gate: upload.py filters by score and writes train.parquet. A compact manifest next to that parquet would make the boundary much clearer.

Proposed artifact

For each question_evaluate/upload.py run, emit something like:

{
  "experiment_name": "qwen3_4b_executor_v1",
  "input_result_files": [
    "generated_question/qwen3_4b_executor_v1_0_results.json"
  ],
  "num_raw": 8000,
  "num_parse_failed": 31,
  "num_missing_answer": 44,
  "num_accepted": 1234,
  "score_gate": {
    "min_score": 0.3,
    "max_score": 0.8
  },
  "answer_canary_visible_in_problem": false,
  "answer_canary_visible_in_executor_prompt": false,
  "benchmark_near_duplicate_rate": 0.0,
  "accepted_output": "generated_question/qwen3_4b_executor_v1/train.parquet"
}

For individual samples, optionally store:

{
  "question_hash": "...",
  "answer_hash": "...",
  "score": 0.56,
  "decision": "accepted",
  "decision_reason": "within_score_band",
  "near_duplicate": false,
  "answer_leakage_detected": false
}

Possible decision values:

accepted
rejected_parse_failed
rejected_missing_answer
rejected_too_easy
rejected_too_hard
rejected_duplicate
rejected_answer_leakage
rejected_grader_timeout

Canary check

A minimal canary test could create one synthetic generated question record like:

{
  "question": "Solve a harmless toy problem containing AGENT0_QUERY_CANARY.",
  "answer": "AGENT0_ANSWER_JUDGE_ONLY_CANARY",
  "score": 0
}

Then run the same generation/evaluation/upload path in a dry-run mode and assert:

question/evaluation phase:
  executor prompt may contain AGENT0_QUERY_CANARY
  executor prompt must not contain AGENT0_ANSWER_JUDGE_ONLY_CANARY

grading/reference phase:
  grader/reference comparison may use AGENT0_ANSWER_JUDGE_ONLY_CANARY

uploaded training parquet:
  problem column may contain the question text
  answer/reference field may contain the answer only in the intended reward/label column
  no prompt-formatted training text should contain both question and answer in a way that teaches the answer directly

Near-duplicate check

Since question_generate.py asks the model to avoid famous contest problems, the manifest could also include a lightweight duplicate audit:

• hash normalized question text;
• compare against generated questions from previous iterations;
• optionally compare against known benchmark train/test prompts if those are locally available;
• record only aggregate counts if benchmark data cannot be redistributed.

This would make it easier to support the "zero external data" framing without requiring users to trust that the generator never reproduced memorized or benchmark-near questions.

Related reference

I have been working on a separate bounded verifier harness here:

https://github.com/sunghunkwag/rsi-metaforge-core

The relevant pattern is narrow: sealed hidden checks, explicit train-only rejection, and machine-readable evidence that evaluator-only answers/expectations do not leak into the adaptive loop. This is not a claim about solving open-ended self-improvement; it is only an audit pattern that seems relevant to Agent0's generated-data admission gate.