Strop

Put an edge on your prompts.

Train a context pack the way you'd train a model — today one instruction file, next a whole skill. The gradient is text.

中文说明 →

---

Strop is a learning harness built on Claude Code. Give it a task — even a fuzzy one-line brief. It interviews you for the few decisions only you can make (or, in auto mode, decides them itself), then evolves your context pack — today a single instruction file, on the roadmap a full skill package — round by round — forward → judge → optimize → validate — until the result is validated on a held-out set it has never seen. No fine-tuning is involved: a strop is the leather strap used to finish a razor's edge, and the loop does the same job — it refines an existing edge, drawing out what the model already knows without touching weights.

Note

Claude Code only (for now). The harness is several parts working together — command skills, the subagent brains (judge / optimizer / init), the task-contract conventions, and (coming) loop checks and lints. Its orchestration core is a Claude Code Workflow (that link explains what a Workflow is and how it differs from subagents) — and that primitive is what ties Strop to Claude Code today; when another runtime ships one, support can follow. The output has no such dependency: θ* is plain text and infer.py speaks to any OpenAI-compatible API, so training happens in Claude Code and the result deploys anywhere.

Contents

• Why Strop
• Quick start
  • 🤖 The fast path
  • The deliberate path
• Commands
• Example — news-labeling
• How the loop works
• Leakage prevention
• Layout
• Status & roadmap
• License

Why Strop

• Prompts as weights. θ is a single instruction file with named sections. The optimizer does rule-level credit assignment: find the smallest section that owns a failure, edit only that. (Roadmap: θ grows into a full skill package — SKILL.md plus reference files as named perspectives, with credit assignment extended to the file level.)
• Text gradients. The judge scores predictions against frozen gold and writes per-item "why it's wrong" feedback — that critique is the gradient.
• Interaction is a dial. Start from a fuzzy brief: the guided interview asks only the decisions it cannot infer, one question at a time, each with a recommended answer — or turn the dial to /strop-auto and the harness adopts its own recommendations end to end. Both ends run the same disciplined loop; only who answers and who signs changes.
• ML discipline, structurally enforced. Frozen human-signed gold, a signed rubric, train/dev/held-out splits, and sealed validation: the dev judge returns a bare scalar only, the held-out set is opened exactly once.
• Humans touch three gates only — provide data, sign the rubric, sign the gold. Everything else is a command. (In auto mode the two signatures can be delegated to the machine — watermarked, so the numbers stay honest.)
• Tune on your subscription, deploy on a cheap model. All the expensive reasoning (judge / optimizer / init) runs on the Claude Code subscription you already pay for — no per-token teacher bill; the student is any cheap OpenAI-compatible API. Training distills the teacher's judgment into text the student carries at inference (in-context learning — weights untouched), and the result ships to production with no train/serve skew (same one-shot call shape).

Quick start

Usage model: this repo is the harness and your workspace. Clone it to try Strop locally, or fork / hit Use this template if you want to keep and push your own tasks. Work inside the repo; each task is fully self-contained in tasks/<name>/. The harness layer never references your tasks, so as long as you keep your work in tasks/ (and secrets in .env), upstream harness updates stay clean.

git clone https://github.com/keli-wen/strop.git
cd strop
cp .env.example .env

If you are working from a fork or template repo, keep origin pointed at your copy and add this repo as upstream:

git remote add upstream https://github.com/keli-wen/strop.git
# later: git fetch upstream && git merge upstream/master

Tip

Minimal scaffold — uv + OpenRouter + Claude Code and you're training:

npm install -g @anthropic-ai/claude-code            # the runtime the harness runs on
curl -LsSf https://astral.sh/uv/install.sh | sh     # runs infer.py (the student forward)

Then edit .env with one OpenRouter key as the student API — every cheap OpenAI-compatible model behind the same three variables:

STROP_BASE_URL=https://openrouter.ai/api/v1
STROP_API_KEY=sk-or-...
STROP_MODEL=deepseek/deepseek-v4-flash  # use your production student model

Open this directory in Claude Code (claude from the repo root) before running the slash commands below.

🤖 The fast path

Prepare data first; Strop will not invent it:

TASK=my-task
mkdir -p "tasks/$TASK/data"/{train,dev,heldout}
# put representative train/dev/heldout files into those directories

Then, in Claude Code:

/strop-auto my-task "one-line task brief" rounds=3

Strop infers the schema and rubric from the brief + data, asks one compact round of questions only if something is still missing, then closes the loop itself: init → gold → check → smoke → short train → report. By default it machine-signs the rubric and gold; add sign=human if you want it to pause at the signature gates.

The deliberate path

1. /strop-init <name> <brief> — guided interview (asks only what it can't infer), then scaffolds tasks/<name>/, drafts θ₀ + the rubric, and records every decision in tasks/<name>/design.md.
2. Drop data into tasks/<name>/data/{train,dev,heldout}/ (entity-split; heldout most unlike train).
3. /strop-gold — then review the flagged disagreements and sign. (A high disagreement rate means the spec is ambiguous; fix the spec before re-running.)
4. Sign the rubric; set .env if not done above.
5. /strop-train tasks/<name> smoke → /strop-train tasks/<name> rounds=5 → read .runs/<task>/<run-id>/report.md.

Important

The signatures are what the numbers mean. With human-signed gold, J-Score measures correctness. /strop-auto freezes machine-signed gold by default — J-Score then measures agreement with the teacher (distillation semantics), and the report is watermarked to say so. Pass sign=human to keep the gates: auto pauses at the signature and resumes after you sign.

One run yields four durable assets: frozen gold + rubric (a permanent eval), the trained instruction (deployable, transferable), an experiment report, and general rules worth harvesting back into your upstream spec.

Commands

The heavy commands (gold / train / eval) are thin launchers over background Workflows — the primitive that makes Strop Claude-Code-only today (see the note above); init and check-loop run interactively in your session.

command	what it does
/strop-auto <name> <brief>	new — one command, whole pipeline: minimal interview → init → gold (machine-signed by default; sign=human pauses at the gate) → check → smoke → train → report
/strop-init <name> <brief>	scaffold a task + guided interview + bootstrap θ₀ (seed instruction) + rubric draft + design.md decision record
/strop-gold	draft gold (N strong-reference passes + reconcile) → stops for human sign-off (sign=machine to auto-freeze, watermarked)
/strop-check-loop	read-only audit: is the task loop closed and trainable?
/strop-train [rounds=N] [smoke] [heldout]	the training loop + baseline lift + experiment report
/strop-eval arms=…	score any prediction files against frozen gold → J-Score comparison table

The user journey, end to end (diamonds are the human gates):

flowchart LR
  AUTO["/strop-auto<br/>one command"] -.wraps.-> A
  A["/strop-init<br/>brief + guided interview"] --> B["your data →<br/>data/{train,dev,heldout}"]
  B --> C["/strop-gold"]
  C --> G{"sign-off gate<br/>rubric + gold<br/>(auto: machine-signed)"}
  G --> D["/strop-check-loop"]
  D --> E["/strop-train"]
  E --> F["report + θ*"]
  F --> H["/strop-eval<br/>transfer & A/B"]
  AUTO -.-> F

Loading

Example — news-labeling

Task: one SEC 8-K press release → {title, abstract, summary, events[], topics[]}. The metric is J-Score: agreement with a frozen, human-signed gold. Task definition, data, and how to run it: tasks/news-labeling/. Experiment details: docs/news-labeling/README.md.

Training curve — over 5 rounds (run 2026-07-05-c, student deepseek-v4-flash), dev J-Score went from 0.872 (θ₀) to 0.926 (θ*); the bare baseline is 0.731:

Held-out lift — evaluated once (n=15), the trained instruction improves both student models: flash 0.875 → 0.910, pro 0.958 → 0.982. flash with θ* does not reach pro's baseline; the lift covers about 40% of the flash→pro gap, at roughly 1/3.5 of the cost:

Where the compute goes — ~40M tokens total, harness : student = 21.5 : 1 in tokens and ~350 : 1 in notional cost. The expensive side runs on the Claude subscription; the student stays cheap — that asymmetry is the hybrid cost model working as intended:

The trained instruction: tasks/news-labeling/skill.trained.md; its diff against skill.seed.md is what training changed.

How the loop works

forward    infer.py(item, θ)      -> prediction      # one-shot, one call/item, temp=0
loss       judge(prediction, gold) -> score + why    # agreement vs frozen gold
gradient   optimizer(why, θ)       -> edited θ       # bounded, rule-level edits

Who runs each step, and on which model (as in the example run 2026-07-05-c above):

step	runs on	model in the example run
forward (student)	any OpenAI-compatible API, via infer.py	deepseek-v4-flash
judge	Claude Code subagent — subscription	Sonnet 5
optimizer	Claude Code subagent — subscription	Opus
init / gold labeling	Claude Code subagent — subscription	Opus / Sonnet 5 (max effort)

Repeat N rounds on train; pick the best round on dev (scalar only); report once on held-out. If any step fails, the run stops; nothing is retried silently, and rerunning is a human decision.

Leakage prevention

The optimizer is an agent that reads text. If per-item detail from dev/held-out ever reaches it, it will write those specifics into the instruction — the score rises without the skill improving, and the validation numbers stop meaning anything. Strop prevents this with structure, not with instructions to behave:

1. Dev/held-out scores return as a single number. The judge scoring sealed splits is schema-forced to return a bare scalar; per-item detail cannot flow back into training.
2. Separate agents. The judge (reads sealed data) and the optimizer (edits weights) are different subagents with no shared memory.
3. Separate directories. Train artifacts live in round*/, sealed evals in _sealed/; the optimizer is only ever handed train paths.
4. Tool restriction. The optimizer has Read/Edit only; without Bash/Grep/Glob it cannot locate sealed data on the filesystem.
5. Held-out is evaluated once. It stays sealed through every round and is scored at the end by a fresh agent; the result is never fed back.

This blocks accidental leakage, which is the realistic failure mode. Defending against an agent that deliberately works around the isolation would need OS-level sandboxing, which is out of scope.

Warning

Keep your own task's held-out set private — an answer key that has been published can never serve as a blind test again. (The example numbers above were measured before this repo went public, so they stand.)

Layout

tasks/<name>/           # task layer — author-owned, self-contained, any student model
  README.md             #   task front page — what it is, how to run, results
  task.md               #   the contract (forward command, output schema, splits)
  design.md             #   decision record from the init interview (committed)
  skill.seed.md         #   θ₀ — THE instruction, the single learnable weight
  skill.trained.md      #   θ* — shipped example result (news-labeling run 2026-07-05-c)
  rubric.md             #   how the judge scores (human-signed)
  infer.py              #   fixed forward pass (architecture; θ is injected as system prompt)
  gold.workflow.mjs     #   gold pipeline: N passes + reconcile + sign-off gate
  data/{train,dev,heldout}/
.claude/skills/         # command launchers — incl. the /strop-auto orchestrator
.claude/agents/         # harness brains — init / optimizer / judge (+ gold & dev helper agents)
strop.workflow.mjs      # the orchestrator (Claude Code Workflow)
strop-eval.workflow.mjs # standalone eval / transfer entry
CLAUDE.md               # repo conventions & invariants for Claude Code
docs/                  # public diagrams and example experiment docs
contexts/              # local context/design scratch; gitignored, not published
.runs/<task>/<run-id>/  # outputs: round*/ (train) + _sealed/ (dev+heldout); gitignored

Status & roadmap

Practice project; one complete end-to-end run (news-labeling, 2026-07).

• Harden /strop-auto. The orchestrator exists as the fast path; next is repeated unattended runs on real non-example tasks to shake out task-agnostic rough edges.
• θ → skill package. Today the weight is one instruction file. Next: train a whole skill — SKILL.md + reference files as named perspectives — with infer.py learning to consume the package (and eventually an agent as the forward, today's design-only "mode B", making it native).
• A second task, with zero harness changes. Only after that lands does "framework" become an accurate description of Strop. Candidates are not limited to finance: GitHub-issue triage or support-ticket classification (domain transfer), or 10-K risk-factor tagging (a harder yardstick for the benchmark-saturation question).
• Runtime. Claude Code only; the orchestrator ports when another agent runtime ships a Workflow primitive.
• Direction. An agentic teacher — a tool-using optimizer/judge, not a single LLM call. (v1's parallel-heads inference ensemble stays retired, kept only as a sweepable A/B.)

License

MIT