feat: policy backtest — simulate a policy against 30 days of historical sessions before enforcing

[Siddhant-K-code]

Problem

Writing an agent policy is currently blind. You define rules in .agent-scope.json, but you have no way to know what those rules would have blocked or allowed against your actual session history before you enforce them in production.

The result: teams either don't enforce policies (fear of breaking things) or enforce them and discover they're too strict/too loose only after the fact.

---

Proposed solution

agent-strace policy backtest — simulate a policy against stored historical sessions and show exactly what would have been blocked, allowed, or flagged.

# Backtest current policy against last 30 days
agent-strace policy backtest --policy .agent-scope.json --days 30

# Compare two policy versions against the same history
agent-strace policy diff old-policy.json new-policy.json --days 30

# Show coverage: what fraction of tool calls have explicit rules
agent-strace policy coverage --policy .agent-scope.json

Backtest output

Policy backtest — last 30 days (306 sessions, 4,821 tool calls)
──────────────────────────────────────────────────────────────────
Rule                          Matched   Would block   Would allow
──────────────────────────────────────────────────────────────────
deny: write /etc/**              12          12             0
deny: read ~/.ssh/**              3           3             0
allow: read src/**             2,841           0         2,841
allow: bash git *               441           0           441
deny: bash rm -rf *               1           1             0
default (no rule matched)       1,523         —           1,523  ← uncovered
──────────────────────────────────────────────────────────────────
Coverage: 68.4% of tool calls matched an explicit rule
Would block: 16 tool calls across 9 sessions

Policy diff output

Policy diff — old vs new (last 30 days)
──────────────────────────────────────────────────────────────────
Change                          Old policy   New policy   Delta
──────────────────────────────────────────────────────────────────
Total blocked                       16           43        +27
Sessions affected                    9           18         +9
New blocks introduced               —            27        +27
  → deny: write /tmp/**             —            14        new
  → deny: bash curl *               —            13        new
Blocks removed                       0            0          0
──────────────────────────────────────────────────────────────────
Recommendation: review 27 new blocks before enforcing

Coverage report

Surfaces tool calls that fall through to the default action — these are the gaps where policy intent is unclear.

agent-strace policy coverage --policy .agent-scope.json --show-uncovered

Implementation notes

• Reads from .agent-traces/ store — no network calls
• Policy evaluation logic is the same as audit --policy (already exists) — backtest just runs it across all stored sessions in batch
• policy generate (already exists) can be used to bootstrap a policy from observed traces; backtest validates it before enforcement

---

Acceptance criteria

• agent-strace policy backtest runs policy against all sessions in --days window
• Output shows per-rule match counts, would-block, would-allow
• --show-sessions flag lists specific session IDs that would be affected
• agent-strace policy diff compares two policy files against the same history
• agent-strace policy coverage reports uncovered tool call fraction
• --format json for machine-readable output (CI integration)
• Performance: completes in <5s for 30 days of typical session history