Skip to content

Commit 763530f

Browse files
committed
feat: v0.14.0 — accurate collaboration mirror
Three fixes that clean up the collaboration analysis: - Strip skill expansions from word counts: /tdd, /ship etc inflated human turn word counts from ~10 to 500+, misclassifying Partnership sessions as Spec Dumps. Now only the user's actual arguments are counted. - TUI details panel shows outcome vs process grades side by side, commits, tests, and collaboration archetype. The contrast tells the story — a D-process session that shipped 11 commits is visible. - Removed bad nudges: "quality drops after 100 tool calls" (based on process grades, not outcomes) and Spec Dump warning (based on contaminated word counts). After the fix, Spec Dump drops from ~15% to 2% of sessions. Partnership rises to 45% with 57% ship rate.
1 parent d76b2e7 commit 763530f

3 files changed

Lines changed: 75 additions & 2 deletions

File tree

audit-prd.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# sesh audit — Repo Legibility Scoring
2+
3+
## What this achieves
4+
5+
sesh grades agent **behavior** (sessions). `sesh audit` grades the **environment** (repos). Together: "did the agent fail because it's bad, or because the repo was hostile?"
6+
7+
## Feature list (metrics)
8+
9+
Each metric scores 0-10 and produces findings (what exists, what's missing).
10+
11+
| # | Metric | What it checks |
12+
|---|--------|---------------|
13+
| 1 | **Bootstrap** | Can an agent set up from scratch? Setup script, dependency file, README with install steps |
14+
| 2 | **Task entry points** | Are build/test/lint/run discoverable? package.json scripts, Makefile, pyproject.toml |
15+
| 3 | **Validation harness** | Can the agent verify changes? Test files exist, test config, CI config |
16+
| 4 | **Linting** | Can the agent self-check quality? Linter/formatter configs (.eslintrc, ruff, prettier) |
17+
| 5 | **Codebase map** | Is there a navigation doc? ARCHITECTURE.md, AGENTS.md, CLAUDE.md, directory-level READMEs |
18+
| 6 | **Doc structure** | Is documentation organized? README substantive (>50 lines), docs/ directory, inline comments |
19+
| 7 | **Decision records** | Are choices documented? ADRs, CHANGELOG, meaningful commit messages |
20+
| 8 | **Agent instructions** | Is there agent-specific guidance? CLAUDE.md, .cursorrules, AGENTS.md, .github/copilot |
21+
| 9 | **File discipline** | Are files kept navigable? Max LOC, avg LOC, files over threshold |
22+
23+
## Integration logic
24+
25+
```
26+
repo path
27+
→ run each metric detector against the filesystem
28+
→ each returns: MetricResult(name, score 0-10, findings[], recommendations[])
29+
→ scoring engine weights + combines → AuditResult(score 0-100, grade, metrics[], recommendations[])
30+
→ formatter renders report
31+
```
32+
33+
**Scoring:** weighted average of metric scores, scaled to 0-100. Default weights equal. Configurable in `.sesh/config.json` under `audit.weights`.
34+
35+
**Grade mapping:** same scale as sessions — A+ (95+), A (90+), B (75+), C (60+), D (45+), F (<45).
36+
37+
## How metrics work together
38+
39+
Metrics are **independent detectors** — same pattern as `sesh/analyzers/patterns.py`. Each is a function: `(repo_path, config) -> MetricResult`. A registry runs all enabled metrics. New metrics plug in without changing the engine.
40+
41+
The findings from each metric feed into recommendations — same pattern as `sesh/analyzers/remediation.py`. "Missing test config" → "Add pytest.ini or [tool.pytest] to pyproject.toml".
42+
43+
## MiniMax harness connection
44+
45+
The eval loop becomes:
46+
47+
```
48+
1. sesh audit <repo> ← NEW: score the environment
49+
2. Call MiniMax with bad CLAUDE.md (existing)
50+
3. Grade with sesh (existing)
51+
4. sesh fix --patch (existing)
52+
5. Re-run with patched CLAUDE.md (existing)
53+
6. Compare grades (existing)
54+
```
55+
56+
Audit score is included in eval results. Enables:
57+
- Correlation: "agent scored C, but repo scored D on validation — environment is the bottleneck"
58+
- The harness could generate AGENTS.md / CLAUDE.md from audit findings as a pre-step
59+
60+
## CLI
61+
62+
```
63+
sesh audit [path] # Audit repo at path (default: cwd)
64+
sesh audit --json # JSON output
65+
sesh audit --metric bootstrap # Run only one metric
66+
```
67+
68+
## What this does NOT do
69+
70+
- Does not read file contents for quality assessment (that's subjective)
71+
- Does not execute anything (no running tests, no building)
72+
- Does not require a .sesh/ database (filesystem-only analysis)
73+
- Does not replace session grading — complements it

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "agentsesh"
7-
version = "0.13.0"
7+
version = "0.14.0"
88
description = "Agent session intelligence — behavioral analysis, grading, remediation, replay, outcome testing, and prompt debugging for AI agent sessions"
99
readme = "README.md"
1010
license = {text = "MIT"}

sesh/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
"""sesh — Agent Session Intelligence CLI."""
22

3-
__version__ = "0.13.0"
3+
__version__ = "0.14.0"

0 commit comments

Comments
 (0)