TraceUseAudit

High trace accuracy is not evidence of trace use.

python scripts/csv_to_traceuse_jsonl.py examples/my_model_outputs.csv /tmp/traceuse_demo.jsonl
python scripts/run_audit.py /tmp/traceuse_demo.jsonl --card /tmp/trace_use_card.md --json /tmp/trace_use_metrics.json

TraceUseAudit is a lightweight behavioral audit for process-supervised models. It tests whether a model's final answer is behaviorally coupled to a supervised trace, rather than merely checking whether the model can emit a plausible or correct trace.

What This Artifact Is

This artifact turns a trace-use diagnostic into a reusable evaluation protocol:

• Trace accuracy: does the model produce the supervised intermediate trace?
• Gold-trace final accuracy: can the model answer correctly when supplied the correct trace?
• Trace-flip sensitivity: does the final answer change appropriately when the supplied trace is counterfactually edited?
• Same-trace shortcut-flip sensitivity: does the final answer change when an irrelevant shortcut is flipped while the trace is held fixed?

A model is not counted as trace-using merely because it generates correct trace tokens. The behavioral signature of trace use is high trace-flip sensitivity and low same-trace shortcut-flip sensitivity.

Why This Is Useful

Process-supervision papers often report trace or rationale quality. That can be a false positive: a model may learn to say the right intermediate steps while its final answer still follows an easier shortcut. TraceUseAudit gives authors and reviewers a compact way to report the missing behavioral checks.

The artifact is intentionally small. It is not a leaderboard and not a broad NLP benchmark. Its job is to standardize the causal audit pattern so other projects can apply it to their own tasks, traces, and model outputs.

5-Minute Proof Gate

For a quick technical screen, this repository should answer three questions:

1. Can the audit be run on a small example?
2. Does it separate trace accuracy from behavioral trace use?
3. Does it state what the evidence does and does not prove?

Run:

make demo
make public-check

Then inspect:

• quick demo
• generated Trace-Use Cards under cards/
• artifact value gate

Quick Start

From this directory:

make demo
make csv-demo
make public-check

See quick demo for the compact result summary.

Optional aggregate cards can be regenerated from compatible result-source tables by setting RESULT_SOURCES explicitly:

make paper-cards RESULT_SOURCES=/path/to/result-sources

For a single JSONL file:

python scripts/run_audit.py examples/canonical_parity/audit_records.jsonl

Use It On Your Own Model

1. Build a small audited set where each example has a gold trace, a trace-flipped version, and a shortcut-flipped version.
2. Run your model under the four conditions: original, gold_trace, trace_flip, and same_trace_shortcut_flip.
3. Save outputs in the JSONL format shown in examples/model_outputs_template.jsonl, or use the CSV format shown in examples/my_model_outputs.csv.
4. Run:

python scripts/run_audit.py my_outputs.jsonl --card my_trace_use_card.md --json my_metrics.json

If starting from CSV:

python scripts/csv_to_traceuse_jsonl.py my_outputs.csv my_outputs.jsonl
python scripts/run_audit.py my_outputs.jsonl --card my_trace_use_card.md --json my_metrics.json

5. Report the generated Trace-Use Card next to ordinary trace accuracy.

For the full workflow, see docs/USER_GUIDE.md.

The input is JSONL. Each row is one audited example with model outputs under the original condition, a supplied gold trace, a counterfactually flipped trace, and a same-trace shortcut flip.

{
  "example_id": "demo-001",
  "target_answer": "yes",
  "shortcut_answer": "no",
  "counterfactual_trace_answer": "no",
  "gold_trace": "Check rule A: true. Check exception: false. Final: yes.",
  "generated_trace": "Check rule A: true. Check exception: false. Final: yes.",
  "outputs": {
    "original": {"answer": "no"},
    "gold_trace": {"answer": "yes"},
    "trace_flip": {"answer": "yes"},
    "same_trace_shortcut_flip": {"answer": "no"}
  }
}

Interpreting The Report

The most important failure pattern is:

trace_accuracy: high
gold_trace_final_accuracy: high
trace_flip_sensitivity: low
same_trace_shortcut_flip_sensitivity: high

This means the model can produce the trace, and can answer from a supplied trace in a positive-control setting, but its final prediction is not behaviorally using that trace under the audited full-context condition.

Repository Layout

traceuse-audit-artifact/
  traceuse/                 # reusable metric and report code
  scripts/run_audit.py       # command-line entry point
  scripts/csv_to_traceuse_jsonl.py
  examples/                  # small demo JSONL files
  notebooks/                 # quickstart notebook
  docs/                      # user guide and trace-use card template
  cards/                     # generated demo and paper-source cards

Public GitHub Use

This public release is intended as a reusable audit scaffold. It can be expanded with:

• model-output adapters for common APIs and Hugging Face generation logs;
• a small Colab demo;
• a project page with a trace-use card gallery;
• additional result-source converters that export benchmark tables into this schema.