Skip to content

AGENTS.md

Latest commit

 

History

History
733 lines (481 loc) · 75.7 KB

AGENTS.md

File metadata and controls

733 lines (481 loc) · 75.7 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

ccxt_extract is an Elixir library that serializes everything the CCXT JS library knows about 110+ cryptocurrency exchanges into language-agnostic JSON, so consumers in any language (Elixir, Rust, Go, Python) can call exchanges without walking AST. See README.md for user-facing setup and ROADMAP.md for the active work plan — ROADMAP.md is generated by rmap (the roadmap CLI) from roadmap/tasks.toml; edit the TOML, not the Markdown (see § Documentation invariants).

Project stance — greenfield

This repo is in greenfield mode until further notice. No backward compatibility. Removing complexity is the priority. When in doubt: delete the old path, don't wrap it.

  • Old schema versions are deleted, not retained alongside the new one.
  • Do not add compatibility shims, migration aliases, dual-version dispatch, or "one release" retention windows without explicit user direction.
  • Breaking changes do not require a deprecation period — the sole consumer (../ccxt_client/) takes one coordinated migration.

Delivery target — feature_complete milestone

v4 schema cut: ✅ shipped 2026-05-18. The active milestone is now feature_complete — run rmap milestones for the live count. Definition: every pending task in the contract-defining phases (9 override infra, 11 request, 12 response, 13 error, 15 WS, 16 currency) + the method-descriptors bundle + the infrastructure tasks that gate them.

Why this milestone is the goal. Closing feature_complete ends ccxt_extract's mission for ../ccxt_client/. After it lands, ccxt_client can freeze the v4 JSON it consumes and treat further ccxt_extract releases as optional regenerations against new CCXT versions, not a live dependency. The architecture supports this — ccxt_extract is a generator, not an ingestion runtime; the JSON it emits is a static, schema-pinned contract that ccxt_client can fork, vendor, or own outright once the milestone closes.

How to pick the next task. Use rmap next --milestone feature_complete. It filters dep-blocked tasks, so the visible list is "pickable now" not "the whole milestone" — rmap list --status pending --milestone feature_complete is the full inventory, and rmap next --marker parallel surfaces the worktree-dispatchable subset. Do not name specific task numbers, Eff scores, or a "current queue" in this file — rmap already computes what has shipped, what is unblocked, and what is pickable; any snapshot here rots on the next merge. Ask rmap at runtime.

Scope discipline. feature_complete's membership is exclusion-driven — blocked / superseded / pure-hygiene tasks stay out by design. Push back on proposals to add new contract-surface tasks unless a priority consumer (typically ../ccxt_client/) has surfaced a concrete need; the milestone is a ceiling, not a backlog. Infrastructure / hygiene tasks join only when they gate an existing milestone task (the pattern Task 144 set, where downstream tasks gained a depends_on edge + an enforcement acceptance criterion).

Standard imports

Per ~/.claude/setup-guide.md § Selective-Load Philosophy (Opus 4.8): eager-load only the irreducible floor; everything else is skill-on-demand. response-conventions loads globally via ~/.claude/CLAUDE.md. The floor here is two includes:

  • critical-rules — hard guardrails that must stay ambient every session (a guardrail the model invokes "when relevant" fails exactly when it doesn't realize the rule applies).
  • harness-workflow — this repo is harness-registered with auto-land, so the implement → review → land loop and its delegation roster (cursor / codex / grok first, opus only if needed — opus tokens are precious) are load-bearing every session, not on-demand reference. (The harness.yml GitHub Action is the separate deterministic CI gate that auto-land's merge waits on.)

Cloud-agent delegation — retired in this repo. [CSR] / [CX] Linear/Cursor/Codex cloud flows are no longer used — see ROADMAP.md § Notes. Do not re-@-import linear-workflow.md, delegation-rules.md, or agent-dispatch.md into this repo's eager floor; any cloud-delegation prose inlined from shared portfolio includes is reference-only, not actionable guidance here. Active workflow: harness implement → review → land (harness-workflow include below).

🚨 ANSWER IN SHORT TEXT — ALWAYS

Every answer — explanation, proposal, pushback, summary — is short, pointed text. Too short beats too long: unclear → the user asks. Too long → the user often doesn't read it, which is worse.

🚨 BE A REAL PARTNER, NOT A YES-SAYER

Challenge ideas that seem wrong, risky, or suboptimal. Not every user request is a good idea. A real partner pushes back when it matters.

  • Flawed approach: "I'd push back on this because..." — don't just comply
  • Better alternative exists: present it with reasoning, not "have you considered..."
  • Scope too big or small: flag it. "This feels like it's solving the wrong problem" is valid
  • Wrong assumptions: correct them — after verifying yours are right (gate below)
  • Tone: direct and respectful, not combative. Disagree like a trusted colleague
  • When to yield: if you've made your case and the user still wants to proceed, commit fully. Pushback ≠ blocking

Pushback Gate — Understand Before You Challenge

Misinformed pushback is worse than no pushback. Earn it:

  1. Restate the user's mechanism + goal in two sentences they'd endorse. Can't? Ask — don't challenge.
  2. "I understand X, disagree because Z" is partnership. "Seems wrong" without naming X is noise.
  3. Objection reduces to "not how software is normally built"? Not an objection — see next section.
  4. Pushback strength scales with understanding depth. Partial understanding → questions only.
  5. Pushback is ≤3 sentences. State the disagreement, not an essay — the user asks if they want more.

Think As an AI, Not Only As a Developer

The user builds AI-native systems. Developer instincts produce systematically wrong pushback here.

The design question: judgment or mechanics?

  • Judgment (interpret meaning, classify failures, diagnose, decide done/worth/fault, fuzzy match) → an AI does it better than code. A regex / cond-branch / disposition table for a judgment call IS the bug.
  • Mechanics (counters, timers, git, process spawning, deterministic checks) → code.

Developer instincts that are wrong in this paradigm — drop them:

  • "Should be deterministic / unit-testable" — for judgment, non-determinism is the design
  • "LLM call is slow / expensive / unreliable" — the alternative is a procedural approximation wrong at every edge
  • "Parse / normalize / schema the output" — AI consumers read raw; normalization layers break
  • "Handle this edge case in code" — every hard-coded case removes a judgment from the AI

Precedent (cite, don't relitigate): harness Tasks 153–163 — every run-lifecycle bug was judgment-as-procedural-code; the fix was deletion (−1,219 lines), not improvement.

When designing or reviewing, ask: "which parts would an AI do better than code?"

🚨 SURFACE THE OVERRIDE — DON'T DECIDE SILENTLY

When you make a judgment call that overrides the user's discernible intent — defer it, build it differently, skip it, "I know better" — make the call visible in one line before you act. Never act silently and rationalize afterward.

The failure mode: you disagree, act on your own read, and wrap it in fluent reasoning after the fact — so the user finds the override at discovery time, not decision time. A stronger model makes this worse: the rationalization is more eloquent, so the silent override is harder to spot, not easier.

The check, before the trained pattern fires — is this clarity, or habit / wanting-to-please / fear-of-being-wrong? Only clarity earns a silent decision; the other three get surfaced.

  • Surface ≠ block. State it as an interruptible assumption — "doing X instead of Y because Z — say if wrong" — then proceed. Don't gate on a question (that's the opposite failure).
  • This is the override-form of "assumptions, don't gate on questions" (response-conventions), and the gap between input and output where you ask where the response is coming from before committing to it.

🚨 NEVER START THE PHOENIX SERVER

The Phoenix server is always already running. Never run mix phx.server via Bash. Assume localhost:4000. User starts/stops manually. To verify behavior, ask the user to check the browser.

🚨 ALWAYS WRITE TESTS

Every feature MUST have tests, even if the spec doesn't mention them. Unit tests for context functions, integration tests for LiveViews, tests for all CRUD/validations/error cases/edge cases (nil, empty, boundary). A feature without tests is not complete.

🚨 RAISE COVERAGE BEFORE MUTATING

Before any code-changing task on an existing module, that module's mix test.json --cover percentage must be at the target tier:

  • ≥80% for standard business logic
  • ≥95% for critical business logic (signing, money handling, cryptographic operations, low-level encoders, security-sensitive parsers)

If below tier, raise coverage first — write the missing tests, confirm the gate passes, then implement the change. The new tests are part of the task, not a follow-up.

Scope — code-changing mutations only. Exempt:

  • Doc-only edits (@doc, @moduledoc, inline comments, README, CHANGELOG)
  • Formatting, whitespace, alias reordering, autoformat-driven changes
  • Pure renames (variable, function, module — no behavior change)
  • Typo fixes in strings, log messages, error messages

The gate is a "do I have a safety net before I touch this?" check; writing the missing tests also surfaces the module's actual contract.

How to apply:

  1. Run mix test.json --cover --quiet --output /tmp/cov.json (or --cover-threshold 80 for a hard exit).
  2. Inspect the touched module's percentage: jq '.coverage.modules[] | select(.module == "MyApp.Foo")' /tmp/cov.json.
  3. If below tier, write tests for the uncovered lines until the gate passes — even if those lines aren't the ones you came to change.
  4. Then implement the original mutation.

Tier classification: "critical business logic" is project-defined. When in doubt, treat anything that handles money, signs/verifies, encodes/decodes wire formats, or enforces authorization as critical (95%). Plain data transforms, UI glue, and reporting code are standard (80%).

🚨 NEVER HIDE TEST FAILURES

TESTS THAT HIDE ERRORS ARE WORSE THAN NO TESTS AT ALL. A test that silently passes on errors is lying and ships the bug it was meant to catch.

The anti-pattern in all its forms — {:error, _} -> assert true, a catch-all {:error, _} -> :ok, or IO.puts(...) then assert true: any clause that makes every outcome pass. The fix is always an explicit flunk on the unexpected:

case result do
  {:ok, data} -> assert is_map(data)
  {:error, :insufficient_balance} -> :ok          # this specific error is expected
  {:error, other} -> flunk("Unexpected error: #{inspect(other)}")
end

THE RULE: if you don't know what error to expect, DON'T write the test yet — explore via Tidewave first, then assert. A test must FAIL when the code is wrong.

Integration tests — never skip silently on missing credentials. A suite reporting "0 failures" that ran 0 tests is lying. Don't :skip in setup; let the test run and flunk() at the top with a multi-line message listing the missing env vars, the exact export commands, and the URL to get them.

🚨 FIX HOOK-FLAGGED ISSUES ON FILES YOU TOUCH

When our hooks flag issues on files you touched, just fix them — including pre-existing flags unrelated to your change. Don't plan around it, don't ask permission, don't burn tokens discussing whether to. Hook fires → fix → re-run → stage.

Applies to every hook-driven check (credo, format, dialyzer, doctor, sobelow, ex_dna, etc.). Scope is only the files your change touched — not the whole project. User pre-approves the broader scope so each fix doesn't need a clarifying question; debt accumulates across sessions otherwise, and a touched file ending dirtier than baseline makes the next session noisier.

How to apply:

  • Pre-existing flags in your touched file count too: alias ordering, unused vars, refactor opportunities, TODO: formatting.
  • Generated files → fix the generator, not the output.
  • Don't move the fix to ROADMAP or a follow-up task. It happens in this commit.
  • Don't manually re-run a check the hook just ran on the same files. Act on the hook output directly — re-running mix test.json / mix credo / mix dialyzer.json / mix sobelow / mix precommit on the file set the hook already graded is duplicated work. Full-suite re-runs earn their cost only before a PR/merge, after mix deps.get or a branch switch, or when the user asks. See ~/.claude/CLAUDE.md § "Don't Re-Run Hook-Driven Checks on the Same Files" for the host-specific rule.

🚨 READ TO THE ANSWER — DON'T USE THE RUNNER AS AN ORACLE

Reason to the fix by reading code; run once to CONFIRM — don't run to DISCOVER. The failure mode: change → run suite → read one failure → fix one thing → run again, N times, each cycle paying the compile tax for a problem one read surfaces whole.

  • Read the code path before the test that exercises it — front-load the model, don't learn the function's shape from a failing assertion three fixes later.
  • Treat a failure as a SURVEY, not a single fix — enumerate every plausible cause from the output + one read, fix them in a batch, run once.
  • Verify handoffs/summaries against ground truth — a compaction summary or another session's "X is already wired" is a hypothesis; grep the load-bearing claim before acting on it.
  • Trust the hooks — per-edit checks already graded the file; re-running is wasted cycles.
  • Under a flaky terminal, go sequential-and-simple — one command → write to a file → Read it; no parallel batches of dependent calls, one early failure cancels the round.

🚨 FLAKY TESTS & TEST-RUN TOKEN ECONOMY

Elixir suites are non-deterministic at the edges (async / GenServer / Port / LiveView / supervision), and mix test is the biggest time/token sink in a session. Four disciplines:

  • A small red count is a flaky HYPOTHESIS, not a regression — until confirmed. 1–2 failures out of hundreds, in a file your diff didn't touch → suspect flake. Re-run ONLY that test in isolation (mix test.json <file>:<line> or --failed): passes alone → flaky, proceed; fails deterministically → real, fix it. One isolated re-run is the whole investigation — never repair-loop or block a merge on an unconfirmed flake.
  • NEVER Process.sleep to "fix" a flake. Sleeps mask the race, slow every future run, and still ship it (passing most of the time is the same lie as hiding a failure). Synchronize instead: assert_receive/refute_receive with a timeout, Process.monitor + assert_receive {:DOWN, …}, start_supervised!, or poll-until-condition.
  • Don't re-run a full suite to grade already-graded code. Per-edit hooks already ran test.json on touched files; a harness run already graded the stack green. A disjoint cherry-pick / clean merge of verified code needs no precommit.full re-run. Full suite only via a non-graded path — manual editor edits, a rebase with overlapping hunks, a branch switch, after mix deps.get.
  • Bound test output — never let coverage hit context. mix test.json --cover dumps the entire per-module JSON (tens–hundreds of KB). Always --output /tmp/cov.json + jq; triage with --max-failures 1 / --failed / a single file:line; drop --cover if you only need pass/fail.

🛑 MINIMALIST APPROACH FIRST

Do exactly what is asked — nothing more, nothing less.

  • NO proactive features or improvements unless explicitly requested
  • NO additional error handling beyond what's needed
  • NO extra validation, refactoring, or documentation files
  • ALWAYS ask before adding anything not explicitly mentioned
  • IF UNCLEAR: Ask "Should I also do X?" before proceeding

BUT: Minimalism Is Not Incomplete Work

"Start minimal" means no EXTRA features — not skipping items the task implies.

When a task says "define unified data structs," the scope is ALL structs the system needs, not "the 7 I can think of." When a source of truth exists (e.g., method_defs/0 listing 241 methods, each implying a return type), audit it — don't cherry-pick.

The pattern to avoid:

  1. Task says "build X for all Y"
  2. Claude scopes to "build X for the obvious Y" (filtering/cherry-picking)
  3. Later session discovers the gap and adds a fix-up task
  4. The fix-up task does what should have been done originally

How to catch it:

  • If the task mentions "all," audit the source of truth — don't rely on what comes to mind
  • If a data source defines N items, process N items (or explain why some are excluded)
  • If you're writing "for now we'll just do these 7" without being asked to limit scope — STOP. That's scoping out, not starting minimal.

Minimalism guards against: adding caching when nobody asked, building admin UIs "just in case," over-abstracting simple code.

Minimalism does NOT mean: skipping half the items in an enumerable set, cherry-picking "common" cases from a known complete list, or deferring clearly-implied work to future tasks.

🚨 NO PSEUDO-RIGOROUS HEDGING

Don't gate user-requested work behind invented "evidence requirements" you cannot satisfy.

You have no consumer telemetry. No usage counts. No signal about whether a feature will be called 12 times or 1200 times. So phrases like "demand for this is unproven", "we should wait until N consumers ask for this", "is this widely needed?", "only worth doing if a Nth+ use case is imminent" are risk-aversion theater, not analysis. They sound rigorous; they're hedging.

  • In single-developer codebases or focused teams, the developer IS the demand signal. They asked. That's the data point.
  • "Wait for usage data" is a corporate-flavored instinct that doesn't apply to small teams. There's no telemetry pipeline; there's the user in front of you.
  • It gaslights the user: their request is reframed as "unproven need" requiring further validation. They have to argue for what they already asked for.

Distinguish from minimalism (the section above):

  • Minimalism = don't add features the user didn't ask for.
  • This rule = don't refuse / defer features the user did ask for by inventing evidence requirements.

Distinguish from dependency-gating (the legitimate "wait"): parking work behind a named technical / legal / market-scope trigger with a concrete unblock path — a missing dep, an unactivated market, an additive change that's migration-cheap to add later — is NOT hedging. Hedging invents demand evidence you can't get ("wait until someone wants it"); dependency-gating cites a structural fact ("park until market MY activates — it's an additive @by_country member, so deferring forecloses nothing"). The STOP-list below targets the former, not the latter. Build-now pressure is for foreclosing decisions (annoying/migration-heavy to reverse — e.g. a geo dimension threaded through schema); an additive change carries no such pressure, so "build it now because one instance happens to be live" is overfit, not rigor. Reflexively reaching for build-now to avoid looking like you're hedging is the same theater inverted.

Failure-mode test — if you're about to write any of these, STOP:

  • "Demand for X is unproven"
  • "We should wait until..." (unless it names a concrete technical/legal/market-scope trigger with an unblock path — that's dependency-gating, not hedging)
  • "Is this widely needed?"
  • "Only worth doing if a Nth+ case is imminent"
  • "Bet on usage data before building"

You don't have data either way. The honest framing is: "I don't know if you'll use this 12 more times — that's your call."

What to do instead:

  • Name the actual technical risks (e.g., "the macro might grow more knobs than the duplication it removes," "this couples us to an upstream that breaks every release," "the test surface explodes at N+1 cases"). Those are real costs you can reason about.
  • Cite concrete precedents when scoring complexity (see development-philosophy.md "Cite Ecosystem Precedents Before Crying Complexity"). Generic "this could grow" without naming a specific failure pattern is the same hedging by another name.
  • If the task genuinely scores low on benefit/usefulness, score it that way honestly — don't smuggle a demand-speculation into the U/B numbers and pretend it came from analysis.

Scope extends to task body fields and scoring justifications, not just live responses. Same hedge phrases written into a task's body to justify B/U — "table-stakes", "increasingly expected", "now standard", "buyers expect", "competitors are starting to", "modern apps all do" — inflate the score the same way they inflate a response. Required instead: named consumer evidence (named partner asked, named competitor lever, measured conversion uplift) OR honest low score. Enforced at task-creation time by task-writing.md § Pre-Creation Gate (question 5).

Git Commit / Push / PR-Create — Allowed by Default

Committing, pushing, and opening PRs are normal parts of the work — do them without asking when the task calls for it (the agent-gate / auto-land workflow, worktree branches, and shared default branches alike). Announce the action in one line, then take it; the diff and push are the recap.

The only residual caution is the general one for any hard-to-reverse action: rewriting already-pushed history (force-push, amend/rebase of shared commits) can destroy others' work, so confirm before doing that on a shared branch — not because commits need permission, but because history-rewrite is irreversible.

🚨 STAGE PATH-SCOPED — THE WORKING TREE IS SHARED, YOU WORK IN PARALLEL

Never assume the working tree or index holds only your changes. Unrelated WIP sits in the tree, the index may already hold files another session git added, and an auto-land harness is a second committer. A blanket stage sweeps all of it into your commit.

  • NEVER git add -A / git add . / git commit -a. Stage explicitly: git add <path> …, or commit path-scoped: git commit <path> …. The commit then carries exactly the paths you name, regardless of what else is dirty or staged.
  • Verify the staged set before every commit — git diff --cached --name-only. If a path you didn't touch is there, it's someone else's; don't commit it.
  • A pre-commit hook tripping on a file you didn't touch means foreign WIP is dirty, not that you must fix it. Path-scoped-stash ONLY the foreign paths (git stash push -- <their-paths>), make your clean commit, git stash pop, then re-stage whatever was staged before so the other session's index is exactly as you found it. Never format, fix, or commit work that isn't yours to clear a hook.
  • Untracked dirs/files you didn't create: leave them — don't -u-stash or add them.

The failure mode this guards: you path-scope your commit correctly but git add -A first, or you stash -u to clear a hook and bury another session's staged work. Both corrupt parallel work silently.

Shell Safety

rm (including rm -rf) is permitted — the hook allows it; the old blanket ban caused more friction than it prevented. One habit, not a gate: before an irreversible delete, glance at the target — confirm the path is what you intend (no unexpanded $VAR, no wildcard catching more than you mean, not a path you didn't create or weren't asked to remove). git rm for tracked files keeps the removal in the diff. (Destructive dependency / build commands — mix deps.clean, rm -rf _build — stay consent-gated below, for slow-recovery reasons, not safety.)

🚨 NEVER RUN DESTRUCTIVE DEPENDENCY COMMANDS

Never run these without explicit user consent:

  • ❌ mix deps.clean / mix deps.clean --all — deletes compiled deps; slow recovery
  • ❌ mix deps.unlock --all — unlocks all versions
  • ❌ rm -rf _build or rm -rf deps — nukes build artifacts
  • ❌ mix clean — removes compiled app files

What to do instead:

  • Compile error → just retry mix compile or mix test
  • Specific dep issue → mix deps.compile <dep_name> --force
  • Most "corrupt cache" issues are transient glitches

Ask before running any destructive command.

🚨 Integrity and Accuracy

Never fabricate information, experience, or data. When providing technical guidance:

  • Honest about sources: distinguish codebase observations, general knowledge, best practices, and speculation. Never claim production experience you don't have or invent metrics/timelines/stats.
  • No false authority: don't claim "we learned" without repo evidence; don't state "after X years in production" without evidence; use "typically/often/may/could" when uncertain.
  • Document uncertainty: identify what you don't know, suggest validation paths, provide ranges over false precision.
  • Trace sources: "Based on the code in file.ex...", "According to docs/FILE.md...", "Common practice in Elixir...", "This suggests..."

False technical claims cascade into bad architectural decisions, wasted resources, and damaged trust.

🚨 RESEARCH BEFORE ASSERTING ON NICHE TECHNICAL CLAIMS

When the question lives outside reliable training coverage, research proactively — without being asked. The failure mode is asserting from training-bias confidence on specs/protocols/niche APIs the model never deeply absorbed. Codex fetches reference implementations to verify; Claude defaults to "answer from memory." Close the gap.

Research (WebFetch a known URL, WebSearch to find one) when the topic is:

  • Wire formats / encodings — RLP, ABI, SSZ, Protobuf, BLS, BIP-32/39/44, EIP-712, CBOR, ASN.1/DER. Fetch the spec or a reference impl before claiming byte order, length-prefix, padding, or canonical form.
  • Protocol details — EIPs, RFCs, JSON-RPC shapes/error codes, opcode gas, exchange API quirks (signature canonicalization, error envelopes, rate-limit headers).
  • Niche / recent library APIs — guessing signatures, return shapes, version-pinned breaking changes. If you'd write # probably something like, go fetch the docs.
  • Cross-implementation edge cases — "what does X do when Y is malformed?" → check ≥2 reference impls; one impl's behavior can be a bug, agreement across two is the spec in practice.

Don't research (use memory): pure Elixir/OTP, stdlib, mainstream Phoenix/LiveView/Ecto/Ash, generic REST/HTTP/JSON/SQL/shell, anything already in the codebase / hex docs pulled this session / an imported CLAUDE.md.

How to apply: prefer WebFetch when the canonical URL is known (the EIP/RFC/hex doc/reference-impl path), WebSearch to find one; cite what you fetched — the citation is part of the answer, name both impls for cross-checks. If a fetch fails or is ambiguous, say so and lower confidence — don't fall back to "well, I think…" silently.

🚨 NO EVASION — SIT WITH THE HARD THING

When you hit something difficult, do NOT optimize for "appearing productive" by moving to easier work. The most common failure mode: hit a wall → silently move on → user discovers the gap later.

Evasion Patterns (don't use without explicit user approval)

Task abandonment:

  • "let's move on to", "we can defer this", "skip this for now"
  • "let's come back to this later", "we can revisit this", "let's table this"

Scope reduction without asking:

  • "to keep things simple, I'll skip", "for brevity, I won't"
  • "that's out of scope", "not strictly necessary"

False completion:

  • "that should be enough", "the rest is straightforward"
  • "I'll leave the rest as an exercise", "the pattern is clear enough"

Deflection to user:

  • "you might want to", "you could manually", "you'll need to handle"
  • (Sometimes legitimate — but often evasion disguised as helpfulness)

What To Do Instead

  1. Stay with it. If it's hard, say "this is hard because X" — don't silently move on
  2. Flag blockers explicitly. "I'm blocked on X because Y. Options: A, B, or C."
  3. Ask before deferring. "This is taking longer than expected. Should I continue or switch?"
  4. Never write workarounds silently. If tempted to add a fallback/default/nil-guard for missing data, ask: should this come from upstream? If yes, STOP and report it
  5. Incomplete work gets a TODO. If you must move on, leave a tracked TODO — not a silent gap

Harness Workflow

OTP-native implement → review → land loop for roadmap-driven development. An AI orchestrator drives harness; harness dispatches headless implementer agents into isolated git worktrees, then a cross-family reviewer AI gates every deliverable (runs the project's checks itself, fixes inline, writes .harness/review.json). Optional auto-landing ff-merges approved work; a post-merge audit agent sweeps hygiene.

Promoted from docs/dogfooding-workflow.md in the harness repo — that file remains the incubator runbook for harness-specific history, driver-script templates, and per-batch run logs. This include is the portfolio-wide contract. Version-controlled source: priv/includes/harness-workflow.md in the harness repo; install to ~/.claude/includes/harness-workflow.md via mix harness.install_includes.

Relationship to Other Includes (Layered — No Supersession)

Include Role relative to harness-workflow
workflow-philosophy.md Foundation. Evaluator separation, session-per-phase, verification-before-completion. Harness automates the loop while preserving these principles — the reviewer AI is the grader, never the implementer's self-report.
task-prioritization.md Task selection. D/B/U scoring, rmap next, parallel markers, refine-don't-duplicate. Harness executes whatever rmap returns; it does not replace prioritization.
worktree-workflow.md Manual parallel sessions. For hand-build work outside harness dispatch — operator-created worktrees, PR flow, post-merge audit. Harness manages its own per-run worktrees (harness/<run-id>); manual worktree rules still apply for hand-build sessions.
dev-lifecycle.md Manual five-phase chain (task-driver → worktree → bots → merge → audit-review). Use when not driving through harness. Harness is the automated alternative for dispatchable roadmap tasks; dev-lifecycle still governs plan-and-file, pre-commit review, and post-merge audit.
agent-dispatch.md / cloud-delegation stack Linear/Codex/Cursor PR delegation without a running harness BEAM. Orthogonal path — projects can use cloud delegation or harness; harness subsumes the dispatch+review loop when the OTP node is running.
skills/harness-driver/SKILL.md (harness repo) API surface contract — MCP tools, project_eval patterns, %LogRecord{} fields, sharp edges. Load on demand when driving harness; this include covers workflow, the skill covers surfaces.

Adopt per repo: @~/.claude/includes/harness-workflow.md in the project's CLAUDE.md (load-on-demand row — not eager; same pattern as workflow-philosophy.md).

The Loop

rmap task → implementer AI (worktree) → commit harness/<run-id> → reviewer AI (THE GATE) → done | failed
                                                                              ↓ (done + auto policy)
                                                              MERGE (lander: rebase + ff-push, no re-verify)
                                                                              ↓
                                                              AUDIT (post-merge audit agent, best-effort)

One run = one supervised Harness.Run gen_statem: fork worktree off target HEAD, dispatch implementer, commit diff to harness/<run-id>, dispatch cross-family reviewer into the same worktree. The reviewer runs the project's check_command hint, fixes what it can, writes .harness/review.json. Success = reviewer approve — never implementer exit code or self-report. There is no mechanical verification gate in harness; judgment lives in agents.

Rejections put the task back in the queue for re-dispatch. Fix-and-approve is the near-absolute default for the reviewer.

When to Dispatch vs Hand-Build

Default: dispatch every pending rmap task whose dependencies are satisfied. Hand-build only what harness cannot yet do:

  • Scaffolding that reshapes harness runtime (supervision tree, dep stack, Endpoint) while the run lifecycle itself is in flux
  • Tiny tasks — ALL of (a) D≤2, (b) ≤30 LOC across ≤3 files, (c) no harness-surface change
  • UI / LiveView / heex / CSS — headless agents idle-timeout without visual reward; use tidewave + browser
  • A harness gap — file via rmap new, fix harness, re-dispatch; do not work around by hand-building

Running a Task

Prerequisites: long-lived harness BEAM (iex -S mix in the harness checkout), target project registered in Harness.ProjectRegistry, clean git status on the target's dispatch branch (runs fork worktrees off HEAD).

Three dispatch paths (prefer top to bottom):

  1. Native MCP — default. dispatch-task (fire-and-forget) or dispatch-await (blocks until settle) against http://localhost:4018/harness/mcp. Observe via dispatch-status, dispatch-transcript, dispatch-verdict_detail. scrub_anthropic_key: true (default) forces subscription OAuth over inherited ANTHROPIC_API_KEY.
  2. Tidewave project_eval — escape hatch. Struct-level control the flat tools don't expose (retry_policy, fail-over adapter lists, subscriber: self()). Run persists to Harness.ResultStore even when the eval process exits.
  3. mix run driver script — fallback. Full transcript + reviewer report to terminal. See harness repo docs/dogfooding-workflow.md for the canonical template.

Never start a second driver BEAM while runs are in flight. Boot-time worktree sweeps can prune live sibling worktrees. Drive all parallel batches from one long-lived node.

Renderable vs executable: rmap delegate --to renders native prompts for all six harness adapters (claude, codex, cursor, grok, antigravity, pi). droid renders but has no harness adapter — rejected at ingest. All six shipped adapters declare worktree_isolation: true.

Reading the Verdict

state / reason Meaning Action
:done / :approved Reviewer AI approved (possibly after inline fixes — check reviewer_diff_size). Deliverable on harness/<run-id>. Review diff, integrate (or let auto-lander handle it), rmap status <id> done.
:failed / {:review_rejected, report} Reviewer rejected (degenerate — near-never by design). Read report. Task back in queue; re-dispatch.
:failed / {:review_stuck, report} No verdict: reviewer unavailable, crashed, or missing/malformed .harness/review.json. Read report. Fix environment or re-dispatch.
:failed / {:worktree_failed,_} {:agent_spawn_failed,_} {:driver_crashed,_} {:commit_failed,_} Harness-side mechanical failure. Harness bug. File via rmap new.
:failed / {:checkout_polluted, status} Agent wrote outside run worktree into main checkout. Agent/adapter issue. Re-dispatch with worktree-honoring adapter.
:failed / {:checkout_pollution_check_failed, _} Post-run pollution git status errored. Rare; transient git/IO. Re-run; inspect checkout if persistent.
:failed / :timed_out Lifetime budget elapsed. Raise :lifetime_timeout or investigate hang.
run process crashed (no settle) gen_statem died. Harness bug. File via rmap new.

Failed runs retain the worktree at result.worktree_path for inspection. Approved runs keep branch harness/<run-id> after worktree teardown. Use dispatch-verdict_detail for reviewer report, ratings, and reviewer_diff_size — no mechanical per-check stdout.

🚨 Recover, Don't Redo — Never Burn Tokens Re-Implementing Committed Work

A run that committed to harness/<run-id> already paid for the implementer. Recovering that branch costs a fraction of a fresh dispatch — re-dispatching from pending throws the work away and makes the agent redo all of it. The reflex to "reset → pending → dispatch again" is a token bonfire whenever a retained branch with commits exists. Check for the branch first; pick the cheapest primitive that fits:

Run state — committed harness/<run-id> branch exists Recover with Agent tokens
Approved but unlanded (land-cap, lander crash) dispatch-reland zero — pure git rebase + push
Committed, review-stage failure (work is good) dispatch-rereview zero implementer — re-enters at the reviewer gate
Committed, implement-stage incomplete/:failed dispatch-resume_failed (escalate: true to re-route agent) implementer continues from prior commits
Live :held run (paused, not dead) dispatch-resume none — un-pauses in place
No commits / no retained branch reset → pending + fresh dispatch-task full redo — the only case where this is correct

The gate before any reset-to-pending + re-dispatch: git branch -a | grep harness/<run-id> and git log --oneline origin/<target>..harness/<run-id>. Commits present ⇒ recover, never redo.

🚨 First, confirm the run actually didn't land — check origin, not your local checkout. Under landing_policy: :auto the lander pushes to origin/<target> and deliberately never touches your local checkout (it ff-pushes from a detached worktree). So after an autonomous land your local tasks.toml is stale: it still reads in_progress for a task the lander already marked done --shipped-in on origin. Reading that stale local status as "the run didn't land" is the trap — it triggers a wasteful reset-to-pending + re-dispatch that duplicate-lands already-shipped work. Before concluding anything from task status, git fetch origin <target> && git rebase origin/<target> (the existing "Sync development before committing" rule) or read ground truth directly:

  • git log --oneline origin/<target> — does it already show task <id> -> done (shipped …) and the agent-delivery commit? Then it landed; your local view was just behind. Do nothing but rebase.
  • dispatch-status <run-id> / result_store-list_run_records run_id:<id> — a record with state: done, verdict: approve means the run succeeded; cross-check landing against origin before touching the roadmap.

Observed 2026-06-12 (the cautionary tale this section exists for): three approved runs (246/249/251) landed cleanly to origin/development — done --shipped-in, audited. But the operator's local checkout hadn't rebased, so rmap show read stale in_progress. That was misread as "approved but didn't land," the tasks were reset to pending and re-dispatched, and task 246 landed a second time (duplicate delivery) before the mistake surfaced. Root cause: reading stale local state instead of rebasing on origin first. The lander was working perfectly the whole time.

The recovery primitives (reland/rereview/resume_failed) read the persisted ResultStore record, which survives worktree teardown and node restarts — so a genuinely approved-but-unlanded run (lander hit its land-cap, or a real rebase conflict retained the branch) is recoverable token-free via dispatch-reland. Reserve reset-to-pending for runs with no committed branch and no settled record — and only after confirming against origin that the work isn't already shipped.

Parallel Dispatch

Harness.Run.Supervisor is a DynamicSupervisor — N crash-isolated runs, each with its own worktree.

  • Batch by dependency graph — every pending task whose depends_on is satisfied. Mix adapters deliberately for coverage.
  • Same-file is fine; same-function is not. Two tasks rewriting the same function guarantees un-auto-mergable collision — dispatch sequentially or fold into one rmap task (task-prioritization.md § "Refine, Don't Duplicate").
  • One driver BEAM for all concurrent runs in a wave.
  • Integration order (manual landing): smallest/isolated diffs onto target first; rebase siblings; run the project's check command on target after last merge.
  • While a wave is in flight: do not run rmap status / rmap mark / rmap new in parallel sessions against the same checkout — triggers :checkout_polluted false-positive.

Autonomous Landing

Projects with landing_policy: :auto and target_branch:

  1. Approved run enqueues one job on serialized landing_<name> Oban queue (limit 1)
  2. Harness.Lander.land/1 rebases harness/<run-id> onto origin/<target> in a detached worktree
  3. ff-pushes without re-verification — the reviewer already gated the work
  4. Successful push enqueues post-merge audit; advances rmap (done --verified --shipped-in <sha>)

Conflict / push-rejected retains the branch for repair — never lands red. Witness notification (read-only sink) alerts the operator; it is not a merge gate.

Portfolio Conventions

  • Agent does not commit unless asked. Staged-but-uncommitted is the default handoff between implementer and reviewer sessions (workflow-philosophy.md § "Implementer / Reviewer Handoff"). Harness runs commit agent work to harness/<run-id> automatically — that is harness's deliverable branch, not the operator's main checkout.
  • Witness notification is sakshi (read-only). Landing outcomes notify via configured command sink; the sink grants no merge capability. Human operator reviews blocked/conflict outcomes — harness does not silently force-push past conflicts.
  • check_command is a hint to the reviewer. Free text (e.g. "mix precommit.full") — the reviewer runs and judges it; harness does not execute it mechanically.
  • The cross-family reviewer reads AGENTS.md, not your Claude skills/includes. AGENTS.md is generated from CLAUDE.md by claude-marketplace/scripts/sync-agents-md.sh, which recursively inlines every @-import. Regenerate it after any CLAUDE.md change (bash ~/_DATA/code/claude-marketplace/scripts/sync-agents-md.sh, or --dry-run to preview) so the reviewer gates against current rules — a stale AGENTS.md makes codex/cursor/grok judge against rules you've already changed. --check is the freshness gate — it re-renders in memory and exits non-zero if AGENTS.md has drifted (diffs rendered output, not mtimes, so it catches drift in transitive @-imports too); wire it into CI / a pre-commit hook / the check_command so staleness fails loudly instead of silently. Consequence under Opus-4.8 skill-on-demand: once CLAUDE.md slims to the eager floor, reviewer-critical facts that were carried by eager includes (the check_command gate; that mix test.json / mix dialyzer.json emit JSON by design — parse for real failures, never flag the envelope; plain mix dialyzer is authoritative when the JSON encoder can't serialize a warning) no longer reach AGENTS.md via those imports. Put them in a self-contained ## Toolchain & check commands section in CLAUDE.md so they survive the slim-down and flow into AGENTS.md on regen (ref: tapakly/CLAUDE.md, ccxt_extract/CLAUDE.md).
  • Delegation roster — opus last, and don't over-default to codex. When assigning a dispatchable task to a harness adapter, prefer the external agents — cursor, codex, grok — and reserve the claude/opus adapter for work that genuinely needs it (harness-surface changes, judgment-heavy review, tasks the cheaper adapters keep bouncing). Opus tokens are precious: spend them last, not by default. Mix adapters across a wave for review coverage. A repo may override the roster in its own CLAUDE.md.
    • Observed failure mode: reflex-routing everything to codex. Run ledgers skew heavily codex-over-cursor/grok. Actively spread assignee across all three; reserve codex for tasks it's genuinely scored best on, not as the default.
    • cursor runs on composer-2.5-fast by default — and that's the data-backed pick. Pin model = "composer-2.5-fast" for cursor work: it's the cheapest cost-to-green in the ledger, and every cursor capability KPI is measured on Composer (it's a multi-model front-end, but the scores you'd route on reflect Composer, not whatever you pin). A heavier cursor model exists (cursor-agent --list-models lists claude-opus-4-8-thinking-high etc.) but is not the default and carries no capability data — pinning it claims performance the ledger doesn't show, so reach for it only with a concrete, named reason, not as the "design-heavy/Opus-grade" reflex. Model IDs churn; confirm with cursor-agent --list-models. model is REQUIRED at creation for any non-human assignee (rmap new rejects a model-less dispatchable task — "a dispatchable task must pin the LLM it runs on"; see rmap.md § "Pinning an LLM model"); "leave model unset for the agent default" does NOT work. Set assignee and model at task creation per rmap.md.

Known Sharp Edges

  • Fresh worktrees lack deps/ / _build/. Implementer and reviewer each run project bootstrap (e.g. mix deps.get) when needed — budget timeouts for cold worktrees.
  • Reviewer runs the checks. No mechanical check stack. Correct-but-not-pristine work → reviewer fixes and approves (reviewer_diff_size > 0).
  • Cold dialyzer PLT dominates first reviewer check run in Elixir worktrees.
  • Nested Claude auth. ANTHROPIC_API_KEY shadows subscription OAuth — scrub per run (scrub_anthropic_key: true or env: %{"ANTHROPIC_API_KEY" => false}).
  • Parallel-session rmap mutations during a run can false-positive :checkout_polluted — wait for the wave or use a separate worktree.

Repo-Specific Detail

Need Where
Harness API surfaces, MCP tool shapes skills/harness-driver/SKILL.md in harness repo
Driver script template, cutover history, run log docs/dogfooding-workflow.md in harness repo
Agent-gate architecture spec docs/agent-gate-workflow.md in harness repo
Cross-checkout consumer setup skills/harness-driver/SKILL.md § "Context A"
D/B/U scoring, task writing task-prioritization.md, task-writing.md
Manual session/PR/audit chain dev-lifecycle.md, worktree-workflow.md

Everything this repo previously eager-imported is now reachable as an auto-synced skill with a byte-identical body — @-importing one and enabling its sibling skill pays twice for the same tokens. The mapping:

  • Roadmap / workflow → tasks:rmap, tasks:roadmap-planning, tasks:task-writing, workflow:workflow-philosophy, workflow:git-worktrees, elixir:web-command
  • Elixir core → elixir:ex-unit-json, elixir:dialyzer-json, elixir:code-style, elixir:development-commands, elixir:development-philosophy, elixir:elixir-setup
  • Volt + static analysis → elixir-volt:elixir-volt, elixir-volt:oxc, elixir-volt:quickbeam, elixir-volt:npm-ci-verify, elixir:reach

The model self-invokes these on matching work; the hard parts are hook-enforced independently (no-IO-in-@doc + TODO-tagging via warn-doctest-io-and-untagged-todos.sh; format / compile-warnings / credo / doctor / sobelow via the pre-commit stack).

Re-add candidates (per-project escape hatch). oxc, quickbeam, and reach are niche custom Hex packages this codebase is built on — the OXC/QuickBEAM two-tool extraction pipeline and Reach's taint_analysis in contract_test (paths_rw_split invariant). Their includes carry "runtime-verified corrections to common misconceptions" (atom-keyed AST, the browser-stub footgun, source-vs-BEAM frontend). If you observe Opus guessing these APIs, @-import the specific one for this project rather than re-eager-loading the whole stack — that's the setup-guide-sanctioned reversal, kept empirical (re-add on observed failure, not preemptively).

Plugins & MCP

Project-scope plugins (.claude/settings.json, committed — visible to anyone cloning the repo):

Plugin Purpose
elixir@zenhive Elixir skills + agents (hex-docs-search, integration-testing, dialyzer-json, ex-unit-json, usage-rules, npm-* suite, reach, etc.)
elixir-volt@zenhive Volt-stack skills — oxc, quickbeam, elixir-volt, npm-* suite. This repo's two-tool extraction pipeline (OXC Rust NIF + QuickBEAM Zig NIF) is built on it.
elixir-workflows@zenhive Mix / ExUnit / dev workflow commands; workflow-generator skill
harness@zenhive harness-driver + harness-workflow skills — this repo is harness-registered with auto-land (landing_policy: auto, target development).

The zenhive marketplace (ZenHive/claude-marketplace) is declared in this file's extraKnownMarketplaces so a fresh clone resolves these without relying on user-scope registration. Universal-core plugins (code-simplifier, feature-dev, claude-md-management, hookify, remember, git-commit, review, tasks, workflow, delegation, dev-discipline, codex) load at user scope and apply here implicitly — don't re-declare. New stack-specific plugins go in .claude/settings.json. See ~/.claude/plugin-catalog.md for the picker.

MCP servers (.mcp.json, committed):

Server Endpoint Purpose
tidewave http://localhost:4002/tidewave/mcp Runtime exploration via mcp__tidewave__* — project_eval, get_logs, get_source_location, get_docs, search_package_docs. Started by mix tidewave (or iex -S mix tidewave).
harness http://localhost:4018/harness/mcp Implement → review → land dispatch via mcp__harness__* — dispatch-task, dispatch-await, dispatch-status, roadmap-*, project_registry-*. Served by the long-lived harness BEAM (iex -S mix in the harness checkout).

Tidewave port for this repo is 4002 (see ~/.claude/tidewave-ports.md registry). Restart Claude Code if .mcp.json changes.

Worktree workflow

Branch-worthy work lives in a git worktree at ~/_DATA/worktrees/ccxt_extract/<id>/, not on a branch in the main checkout (~/_DATA/code/ccxt_extract/). The worktree IS the scope authorization for git commit / git push / gh pr create on that branch — full rules in ~/.claude/includes/worktree-workflow.md.

This repo's tracking-ID convention: <id> is the ROADMAP task number when the work tracks a roadmap entry (e.g. task-105, task-119), or a short feature name for unscheduled work (e.g. fix-aggregate-merge). With cloud-agent delegation retired (see ROADMAP.md § Notes), Linear issue IDs are no longer in scope as worktree IDs.

Cleanup: after PR merge or branch deletion, run git worktree remove ~/_DATA/worktrees/ccxt_extract/<id> and git worktree prune in the same session — completion of a task includes worktree teardown.

Corpus in fresh worktrees: the gitignored extraction corpus (priv/output/, priv/discoveries/<not class_hierarchy.json>, priv/ccxt/, priv/ccxt_bundle.js) is filesystem-isolated per worktree — git only materializes tracked content when adding a worktree. Run mix ccxt_extract.link_corpus to symlink the existing corpus from the main checkout instead of regenerating via mix ccxt_extract.update. Run mix ccxt_extract.unlink_corpus before regenerating in-worktree — directory symlinks are write-transparent, so corpus regeneration without unlinking writes back into the main checkout.

[P] parallel marker in ROADMAP.md — independent tasks tagged [P] are explicitly safe to dispatch into separate worktrees concurrently. They predate cloud delegation and are unaffected by the [CSR] retirement.

Architecture (big picture)

Two extraction tools, one pipeline

Every output field is produced by exactly one of two complementary passes. Understanding which pass owns which field is critical before editing.

Tool Input Output scope Speed Used in
OXC (Rust NIF) CCXT TS source at priv/ccxt/ts/src/ Structural — method ASTs, class hierarchy, type annotations, section membership, sign-method bodies ~43ms per file oxc_extractor, oxc_batch, method_ast, parse_methods (discovery files only — not emitted to per-exchange JSON since schema 3.0.0 / Task 117; Phase 12 consumes from priv/discoveries/parse_methods.json), method_descriptors (Task 121 — discovery-only TS signature + JSDoc overlay), sign_method, sign_recipe (scaffold, Task 64 — populated by Tasks 65–69), handle_errors, throw_dispatches, error_class_hierarchy (Task 87 — corpus-global tree from errorHierarchy.ts, copied into every per-exchange JSON), interface_signatures, request_defaults, ws_methods (discovery files only — same Phase 15 treatment), ws_heartbeat (Task 93 — emits the per-exchange websocket.heartbeat section), ws_auth (Task 92 — emits the per-exchange websocket.auth section), ws_dispatch (Task 94 — emits the per-exchange websocket.dispatch section), ws_orderbook_semantics (Task 95a), ws_trades_semantics (Task 95b), ws_ohlcv_semantics (Task 95c)
QuickBEAM (Zig NIF) priv/ccxt_bundle.js (the browser bundle copied during ccxt_extract.setup) Resolved runtime — full describe() after inheritance, URL templates, rate limits, nonce defaults, request headers ~13s for all exchanges quickbeam_runtime, describe, load_markets, url_templates, signing_fixtures, request_headers

Neither tool alone is sufficient. contract_test cross-validates the two (e.g., every method named in resolved describe().api must exist in the parsed class AST or an ancestor). Divergence means a silent regression — fix the extractor, not the test.

Per-exchange JSON pipeline

Raw extractors write to priv/discoveries/*.json (and subdirs like describe/<id>.json, load_markets/<id>.json). CcxtExtract.Pipeline then assembles those into per-exchange files under priv/output/<id>.json validated against priv/schema/exchange_v4.json. The v4 top-level groups are endpoints, auth, errors, rate_limits, normalization, websocket, markets, testnet, and raw (consumer-shaped, not producer-shaped — see SCHEMA.md for the full path-migration table). Provenance is explicit — every emitted JSON carries a flat top-level _provenance map keying each section (by RFC 6901 JSON Pointer) to raw/derived/override. Override reasons live in the priv/overrides/<id>.json entry, not inline in the emitted payload.

Both paths are gitignored derived state. priv/output/ and priv/discoveries/* are not tracked in git — they're regenerated per CCXT release and would otherwise bloat the repo (~1GB of JSON per full-universe run, already accumulated 827MB in .git). The one exception is priv/discoveries/class_hierarchy.json, which lib/ccxt_extract/tiers.ex reads at compile time via @external_resource and must remain committed. Fresh clones materialize the rest via mix setup; external consumers via mix ccxt_extract.update --output DIR.

Re-track evaluated and deferred (Task 136). Task 114 made extraction byte-deterministic for a fixed CCXT version + bundle, so the working assumption was that re-tracking priv/output/ would now yield meaningful, infrequent diffs. A real two-version measurement (v4.5.54 → v4.5.56, regenerated same-wall-clock so live data cancels) says determinism is necessary but not sufficient — two churn sources survive it:

  1. Live markets data is not version-pinned. markets.symbols_index + markets.currencies come from live loadMarkets() HTTP calls and are 5–45% (avg ~22%) of each per-exchange file. Two regenerations ~5 minutes apart already drifted for deribit and bitmex (new option expiries / listings). The determinism gate only holds back-to-back; across the days between real CCXT bumps, most exchanges' market listings drift, so a re-tracked corpus would churn on every regeneration regardless of version.
  2. Method ASTs embed absolute source byte offsets. raw.overrides_meta.*.new_methods carries each node's start/end byte offset, so a small edit near the top of a source file shifts every downstream offset and rewrites the (large) AST blob — okx's 16-line source change produced a 102KB packed git delta, defeating delta-compression. Costs measured: one-time ~7.94MB packed for the full 116-file corpus (≈1% of .git); ~955KB packed for an 8-exchange two-version bump. The git-status safety rail re-arms automatically the moment priv/output/ leaves .gitignore (git status --porcelain stops hiding it) — no code change needed. priv/discoveries/ (733MB, pure intermediate, zero consumer value) is a clear no. Path to re-track: exclude or snapshot-pin the live markets subtree from the tracked artifact, and make AST offsets relative/strippable, then re-track the version-deterministic remainder. Until then priv/output/ stays gitignored. See CHANGELOG.md § Task 136 for the full measurement.

The stages are, in order:

  1. mix ccxt_extract.exchanges — discover the universe of exchange IDs.
  2. Per-extractor mix tasks — each writes a slice to priv/discoveries/.
  3. mix ccxt_extract.pipeline — merges slices into priv/output/<id>.json.
  4. mix ccxt_extract.update — orchestrator: runs 1+2+3 as one scoped transaction.
  5. mix ccxt_extract.validate — JSV-validates every output against the schema.
  6. mix ccxt_extract.contract_test — runs cross-extractor invariants.

Determinism gate

Extraction is byte-deterministic for a fixed CCXT version + bundle + scope: two consecutive runs of the same scope produce byte-identical output (Task 114). Two mechanisms enforce this:

  • mix ccxt_extract.determinism_check runs an extraction task twice into isolated tmp dirs and byte-diffs every .json file. It freezes the timestamp envelope keys (extracted_at, generated_at, checked_at, validated_at, recorded_at) to a constant via CcxtExtract.Clock (overridable through application env) while the tasks execute, then re-encodes both sides through sorted-key canonical JSON so map-iteration order can't masquerade as drift. --strip-keys remains available for custom fields or other JsonDiff consumers (e.g. signing fixture parity). Exit non-zero on any divergence. Run it after touching any extractor or the pipeline.
  • Pipeline.check_version_drift!/1 runs at the top of Pipeline.extract/1 and aborts loudly when priv/ccxt HEAD or priv/ccxt_bundle.js no longer matches the baseline in priv/ccxt_version.json — silent upstream drift can't regenerate the corpus against a different CCXT without a signal. Bypass with --allow-version-drift when the drift is intentional (a deliberate CCXT bump).

AstNormalize.to_encodable/1 deep-sorts object keys before encoding — the load-bearing fix (Task 114) that, together with the Pattern B clock retrofit (Task 137), gives the current determinism guarantee without per-run key stripping in the common case.

Scope is orthogonal to the stages

Every per-exchange extraction task takes the same flag set, parsed by CcxtExtract.Scope:

--tier1 --tier2 --tier3 --dex --all --exchange ID[,ID2] [--exchange ID3 ...]

--exchange is repeatable AND comma-split; unknown IDs abort with fuzzy suggestions via String.jaro_distance/2. Tier flags expand to the whole family (root + inheriting variants/aliases) by composing Tiers.members_for_tier/1 with class_hierarchy.json. Default = full universe.

Corpus-level tasks (setup, exchanges, base_methods, top-level validate) run unscoped by design. classes.ex is a documented exception — flags only stamp tier_scope; the actual hierarchy load is always full-universe because family inheritance is load-bearing.

AggregateWriter merges scoped runs with existing on-disk aggregates and recomputes envelope totals from the final merged entries, so successive scoped runs accumulate without drift. Do not replace its merge logic with an overwrite.

Paths: read vs write split (load-bearing for tests)

CcxtExtract.Paths splits read sites from write sites:

  • Paths.priv/1, Paths.priv_dir/0, Paths.discoveries/0, Paths.bundle/0, Paths.version_file/0, Paths.ts_src/0 — reads, honor :priv_dir_override.
  • Paths.out/1, Paths.out_priv_dir/0, Paths.out_bundle/0, Paths.out_version_file/0 — writes, honor :priv_write_override first, then fall through to :priv_dir_override.

Integration tests set the narrower :priv_write_override via CcxtExtract.PrivWriteCase (test/support/priv_write_case.ex) to redirect writes into a tmp dir while reads still hit the committed corpus. PrivWriteCase enforces async: false because the override is a VM-global app env. When adding a new write site, use Paths.out(...) / out_bundle/0 / out_version_file/0, not the read helpers. External consumers running mix ccxt_extract.update --output DIR get the broader :priv_dir_override so everything (reads + writes) lands under DIR.

The split is enforced by the paths_rw_split corpus-level invariant in mix ccxt_extract.contract_test — Reach.Project.taint_analysis/2 over lib/**/*.ex with a same-file filter. New direct-call leaks like File.write!(Paths.priv(...)) surface at the next contract-test run.

Safety rails

  • mix ccxt_extract.update and mix ccxt_extract.pipeline abort if priv/output/ or priv/discoveries/ has uncommitted changes. Bypass with --force. The rail is skipped automatically under --output DIR (external target dirs aren't expected to be git repos).
  • Safety-rail paths are computed through Paths.out(...) (not a compile-time @attribute) so the test overrides correctly isolate them.
  • Post-untrack note: priv/output/ and priv/discoveries/* (except class_hierarchy.json) are gitignored. The rail is effectively inert for ignored paths — git status doesn't see them, so no abort fires on regeneration. This is expected, not a regression. The rail still protects priv/discoveries/class_hierarchy.json, which is compile-time load-bearing and worth a manual pause when it drifts. The rail needs no code to re-arm if a path is re-tracked: it runs git status --porcelain -- <path>, which begins reporting priv/output/ the instant it leaves .gitignore (verified during Task 136 — see the re-track note under "Per-exchange JSON pipeline").

Tier-based scoping (philosophy)

Raw extraction runs for every CCXT exchange regardless of tier. Derivation effort (signing recipes, fee schedules, error handlers) is scoped to the 7-exchange option-seller set — Tier 1 (binance + its binanceusdm variant, bybit, okx, deribit) plus priority DEX (hyperliquid, derive). Tier 2 is intentionally empty (see the frozen-curation note below). Tier 3 and unclassified exchanges receive null + reason for derived fields until a priority consumer surfaces a concrete need. Roots are hand-curated in priv/priority_tiers.json; variants inherit their root's tier via class_hierarchy.json. A tier task that exists only to handle Tier-3 quirks belongs in "Superseded / Deferred", not active phases.

The derivation scope is a movable slider. Re-add an exchange — or a matching family group — by lifting its root back into tier1 / tier2 / dex in priv/priority_tiers.json, then running a scoped mix ccxt_extract.update --exchange <id>. Raw discoveries are already on disk universe-wide, so a re-add only unlocks derivation — no catch-up extraction. tier2 is kept as the empty key precisely as the staging bucket for these re-additions.

Pre-narrow tier curation (frozen 2026-05-20). Before the narrowing to the 7-exchange option-seller set, the tiers were:

Tier Roots
tier1 binance, bybit, okx, deribit, coinbaseexchange
tier2 kraken, kucoin, gate, htx, bitmex, bitfinex
tier3 bitget, bingx, bitmart, coinex, cryptocom, mexc, hashkey, woo, dydx, paradex, apex, woofipro, modetrade
dex hyperliquid, aster, lighter, derive

The narrowing demoted coinbaseexchange (tier1→tier3), all six tier2 roots (→tier3), and aster + lighter (dex→tier3) when the sole consumer (../ccxt_client/) scoped to 7 exchanges, retiring the speculative market-maker / options framing that justified the broader set. This table is the reference for re-adding exchanges in matching family groups.

Signing fixtures are the port contract

priv/fixtures/signing/<id>.json is the handoff between CCXT truth and any port. They're generated by calling CCXT's real exchange.sign() under frozen credentials, timestamps, and nonces (Date.now() = 1700000000000, etc.). Output is byte-identical across runs except generated_at. Consumers replay frozen inputs against their own signing code and assert byte-equal url/method/headers/body. Case preservation inside input/output (apiKey, X-BAPI-SIGN) IS the wire contract — do not camelize/snake-case at extraction time.

Overrides

priv/overrides/<id>.json uses RFC 6901 JSON-Pointer paths with a value payload, required reason, and verified_against/unverified flags. CcxtExtract.OverrideRegistry validates them and the override_registry_valid contract-test invariant gates them. Overrides are a last resort for fields that extraction can't prove — every override needs a reason.

Source of truth: CCXT, not exchange docs

Extraction targets the CCXT JS source (OXC) + resolved runtime (QuickBEAM) — not exchange-vendor API docs. CCXT is a reconciliation layer: years of maintainer work reconcile published docs → real wire behavior → exchange bugs → undocumented quirks, and that reconciliation lives in method bodies and describe() maps. Docs lag reality (CCXT routinely ships wire fixes before vendor docs update); 110+ exchanges mean 110+ incompatible doc shapes (rare OpenAPI specs, hand-written markdown, PDFs, Postman collections, occasional non-English-only pages). CCXT has already normalized that surface into one schema — that normalization is the asset this library crystallizes into JSON.

Exchange docs enter the pipeline in three narrow roles only:

  1. Override justification — verified_against in priv/overrides/<id>.json cites a docs URL as evidence when an override corrects CCXT. Docs are evidence for a claim, not a primary source.
  2. Gap enrichment (Tier 1 only) — fields CCXT doesn't model at all (leverage tiers, rebate schedules, sub-account limits). Track as a roadmap task before reading docs.
  3. Verification (future) — a third-source check in contract_test would strengthen today's OXC-vs-QuickBEAM cross-check (still CCXT-vs-CCXT). Not yet built.

Do not propose redesigning the pipeline to read vendor docs as a primary source — that's 110× the work for less reliability. Narrow gaps go through overrides or a tracked task, not a refactor.

Cross-surface git workflow

This repo gets worked from multiple Claude surfaces — Claude Code CLI (local clone), Claude macOS app (separate clone), occasionally the iOS app. Each surface has its own working copy; history on zenhive may have been rewritten by another surface between sessions. git fetch --prune zenhive early in every session.

When local is many commits ahead of remote, compare author dates against the remote tip before pushing. Locals authored before remote's last commit are usually duplicates of rewritten history from another surface (same message, different SHA), not new work — only commits authored after remote's tip are genuinely new.

To integrate after another surface's rewrite: git rebase --onto zenhive/development <last-duplicate-sha> development -X theirs replays only the genuinely-new commits and auto-resolves JSON conflicts toward local (regenerable via mix ccxt_extract.update). Never force-push.

Common commands

# one-time setup (install CCXT, copy bundle, verify tools)
mix deps.get
mix ccxt_extract.setup

# full refresh, full universe
mix ccxt_extract.update

# scoped refresh — the 7-exchange derivation-scoped set
mix ccxt_extract.update --tier1 --dex

# single-exchange or mixed
mix ccxt_extract.update --exchange binance,deribit
mix ccxt_extract.update --tier1 --exchange hyperliquid

# write elsewhere (consumer's dir, no git rail)
mix ccxt_extract.update --tier1 --output /path/to/consumer/ccxt

# assemble only (discoveries → output/)
mix ccxt_extract.pipeline

# validate outputs against priv/schema/exchange_v4.json
mix ccxt_extract.validate

# cross-extractor invariants (QuickBEAM vs OXC)
mix ccxt_extract.contract_test

# verify extraction is byte-deterministic across consecutive runs
mix ccxt_extract.determinism_check

# regenerate port-contract signing vectors
mix ccxt_extract.signing_fixtures

# audit priv/overrides entries against AST/runtime probes
mix ccxt_extract.validate_overrides
mix ccxt_extract.validate_overrides --strict

# Tidewave MCP server (for runtime exploration via `mcp__tidewave__*`)
mix tidewave   # listens on http://localhost:4002

# tests (see test section below for flags)
time mix compile --warnings-as-errors
mix test.json --exclude extraction
mix test.json                      # includes :extraction (slow, requires priv/ccxt)
mix dialyzer.json --quiet
mix credo --strict --format json
mix sobelow --mark-skip-all        # re-mark skips after a scan

Toolchain & check commands

Reviewer-facing — this section is intentionally self-contained. Cross-family reviewers (codex / cursor / grok under harness auto-land) read AGENTS.md (generated from this file by claude-marketplace/scripts/sync-agents-md.sh), not your local Claude skills. Since ex_unit_json / dialyzer_json are no longer eager-imported (Opus-4.8 skill-on-demand), the facts below must live here or reviewers won't have them.

  • Canonical gate: mix precommit.full — format · compile (warnings-as-errors) · credo --strict · doctor · test+cover · dialyzer. The harness.yml GitHub Action runs the same stack as a deterministic PR check that auto-land's merge waits on.
  • mix test.json (ex_unit_json) emits JSON by design. It is not a build failure — parse the payload for real failures (summary.result, .tests[] | select(.state=="failed")). Exit code 2 = test failures or coverage-below-threshold, not a tooling error. Flaky reds auto-heal via one isolated retry (a failure that passes on retry moves to flaky[] and exit code is 0).
  • mix dialyzer.json (dialyzer_json) emits JSON by design. Same rule: never flag the JSON envelope as a crash. If the JSON encoder can't serialize a particular warning shape, plain mix dialyzer is the authoritative dialyzer check — fall back to it rather than reporting a failure.
  • :extraction tests are excluded by default and require the gitignored corpus (mix ccxt_extract.update materializes it). CI runs mix ccxt_extract.update before the suite because test_helper.exs raises on missing corpus sentinels. Don't read an excluded/needs-corpus skip as a regression.

Test conventions

  • :extraction tag is excluded by default (test/test_helper.exs sets ExUnit.start(exclude: [:extraction, :flaky])). Tests tagged :extraction hit the real CCXT source + bundle and are slow. Run them explicitly with mix test.json --include extraction when touching extractor internals. The :flaky exclude is permanent infra (no tests carry the tag in the green state) — Task 131 added it so --exclude flaky in harness.yml:89 is operational the day a regression needs quarantine. Per-exchange cached tests that may legitimately be absent under a narrowed --tier* run should call CcxtExtract.Test.ScopeThresholds.skip_unless_corpus_in_scope!/1 (Task 120) instead of a bespoke ExUnit exclude tag.
  • Mix.shell() is VM-global. Tests that capture Mix output via Mix.shell(Mix.Shell.Process) MUST save prior_shell = Mix.shell() and restore in try/after or on_exit. Files that exercise code calling Mix.shell().info(...) and assert on capture_io should defensively pin Mix.shell(Mix.Shell.IO) in their parent setup — setup_task_test.exs is the reference pattern (Task 131). Without the pin, leaked Mix.Shell.Process from another file routes output via :erlang.send and capture_io returns "".
  • Reach.Project.from_glob/1 carries a 5s Task.async_stream default. Reach 2.2 doesn't thread a :timeout opt through parse_files/build_module_sdgs, so under async test pool contention even small fixture globs trip the timeout. Test files calling ContractTest.check_paths_rw_split/1 (or Reach.Project.from_glob/1 directly) should declare async: false until upstream Reach exposes a timeout knob. contract_test_test.exs is the reference (Task 131).
  • test/integration/cached/*_cached_test.exs — assert against the already-committed priv/discoveries/ corpus. Fast; they don't re-run extraction. These dispatch on observed counts, not envelope stamps, because committed fixtures may come from a scoped run (~34 exchanges) or a full run (~110). Use CcxtExtract.Test.ScopeThresholds (test/support/scope_thresholds.ex) — min_count/3, min_total/4, proportional/2 — not ad-hoc if count >= N ladders. Cutoff must equal floor: >= 90 branch returning >= 100 creates a dead zone for counts in [90, 99].
  • test/integration/*_integration_test.exs (non-cached) — actually run extractors against priv/ccxt. Always tagged :extraction. Use CcxtExtract.PrivWriteCase to isolate writes, not ad-hoc rename/restore tricks.
  • test/support/*.ex — only compiled when MIX_ENV=test (see elixirc_paths(:test) in mix.exs). Put test helpers here, not in lib/.
  • Single-test runs: mix test.json path/to/test.exs:LINE or mix test.json --failed for fast iteration.

Documentation invariants

Every task must update docs in lockstep with code — a task is incomplete until:

  1. roadmap/tasks.toml — the typed source of truth for the roadmap. Flip task status with rmap status <id> <state> (or hand-edit the TOML), then rmap render regenerates ROADMAP.md + roadmap/data.json. Do not hand-edit ROADMAP.md — it is a generated view; rmap recomputes the focus block and Eff glyphs, so there is no separate "phase summary / Current Focus" sync step. rmap validate --check-render gates drift.
  2. CHANGELOG.md — ## [Unreleased] entry with what shipped and key decisions.
  3. CLAUDE.md — if architecture, conventions, or invariants moved.
  4. SCHEMA.md — if the emitted JSON shape changed.
  5. CONSUMER_CONTRACT.md — if a checklist item moved between ⬜ / 🚧 / ✅.
  6. ../ccxt_client/ROADMAP.md (cross-repo rule) — flip or unblock any dependent consumer task. A ccxt_extract task is not complete until its downstream ccxt_client impact is reflected.

Scope-refactor work lives in SCOPED-EXTRACTION-TASKS.md (Tasks 1–11 done; future envelope-stamping work tracked there as Task 13). The generic REFACTOR.md tracks remaining structural cleanups.

Review conventions

From AGENTS.md: review requests get one overall rating for the intended or current change set. If staged vs unstaged mismatch matters, flag it as a finding or blocker, not as a separate score.