|
| 1 | +# sesh audit — Repo Legibility Scoring |
| 2 | + |
| 3 | +## What this achieves |
| 4 | + |
| 5 | +sesh grades agent **behavior** (sessions). `sesh audit` grades the **environment** (repos). Together: "did the agent fail because it's bad, or because the repo was hostile?" |
| 6 | + |
| 7 | +## Feature list (metrics) |
| 8 | + |
| 9 | +Each metric scores 0-10 and produces findings (what exists, what's missing). |
| 10 | + |
| 11 | +| # | Metric | What it checks | |
| 12 | +|---|--------|---------------| |
| 13 | +| 1 | **Bootstrap** | Can an agent set up from scratch? Setup script, dependency file, README with install steps | |
| 14 | +| 2 | **Task entry points** | Are build/test/lint/run discoverable? package.json scripts, Makefile, pyproject.toml | |
| 15 | +| 3 | **Validation harness** | Can the agent verify changes? Test files exist, test config, CI config | |
| 16 | +| 4 | **Linting** | Can the agent self-check quality? Linter/formatter configs (.eslintrc, ruff, prettier) | |
| 17 | +| 5 | **Codebase map** | Is there a navigation doc? ARCHITECTURE.md, AGENTS.md, CLAUDE.md, directory-level READMEs | |
| 18 | +| 6 | **Doc structure** | Is documentation organized? README substantive (>50 lines), docs/ directory, inline comments | |
| 19 | +| 7 | **Decision records** | Are choices documented? ADRs, CHANGELOG, meaningful commit messages | |
| 20 | +| 8 | **Agent instructions** | Is there agent-specific guidance? CLAUDE.md, .cursorrules, AGENTS.md, .github/copilot | |
| 21 | +| 9 | **File discipline** | Are files kept navigable? Max LOC, avg LOC, files over threshold | |
| 22 | + |
| 23 | +## Integration logic |
| 24 | + |
| 25 | +``` |
| 26 | +repo path |
| 27 | + → run each metric detector against the filesystem |
| 28 | + → each returns: MetricResult(name, score 0-10, findings[], recommendations[]) |
| 29 | + → scoring engine weights + combines → AuditResult(score 0-100, grade, metrics[], recommendations[]) |
| 30 | + → formatter renders report |
| 31 | +``` |
| 32 | + |
| 33 | +**Scoring:** weighted average of metric scores, scaled to 0-100. Default weights equal. Configurable in `.sesh/config.json` under `audit.weights`. |
| 34 | + |
| 35 | +**Grade mapping:** same scale as sessions — A+ (95+), A (90+), B (75+), C (60+), D (45+), F (<45). |
| 36 | + |
| 37 | +## How metrics work together |
| 38 | + |
| 39 | +Metrics are **independent detectors** — same pattern as `sesh/analyzers/patterns.py`. Each is a function: `(repo_path, config) -> MetricResult`. A registry runs all enabled metrics. New metrics plug in without changing the engine. |
| 40 | + |
| 41 | +The findings from each metric feed into recommendations — same pattern as `sesh/analyzers/remediation.py`. "Missing test config" → "Add pytest.ini or [tool.pytest] to pyproject.toml". |
| 42 | + |
| 43 | +## MiniMax harness connection |
| 44 | + |
| 45 | +The eval loop becomes: |
| 46 | + |
| 47 | +``` |
| 48 | +1. sesh audit <repo> ← NEW: score the environment |
| 49 | +2. Call MiniMax with bad CLAUDE.md (existing) |
| 50 | +3. Grade with sesh (existing) |
| 51 | +4. sesh fix --patch (existing) |
| 52 | +5. Re-run with patched CLAUDE.md (existing) |
| 53 | +6. Compare grades (existing) |
| 54 | +``` |
| 55 | + |
| 56 | +Audit score is included in eval results. Enables: |
| 57 | +- Correlation: "agent scored C, but repo scored D on validation — environment is the bottleneck" |
| 58 | +- The harness could generate AGENTS.md / CLAUDE.md from audit findings as a pre-step |
| 59 | + |
| 60 | +## CLI |
| 61 | + |
| 62 | +``` |
| 63 | +sesh audit [path] # Audit repo at path (default: cwd) |
| 64 | +sesh audit --json # JSON output |
| 65 | +sesh audit --metric bootstrap # Run only one metric |
| 66 | +``` |
| 67 | + |
| 68 | +## What this does NOT do |
| 69 | + |
| 70 | +- Does not read file contents for quality assessment (that's subjective) |
| 71 | +- Does not execute anything (no running tests, no building) |
| 72 | +- Does not require a .sesh/ database (filesystem-only analysis) |
| 73 | +- Does not replace session grading — complements it |
0 commit comments