fix(run): stop silent health-gate failures — the "sometimes it just fails" bug (v0.8.4)

A backend that flunked its `--version` health probe was dropped SILENTLY in
run.mjs's fallback loop (`if (!healthy) { fallbackCount++; continue; }`): no
lastErr, no failures.log, no status record — just a bare "backend options
exhausted; falling back to native" handoff with zero diagnostics. That made
transient probe misses look like random "the CLI refused to run" failures.

Compounded by two amplifiers in backends.mjs:
  - health() cached NEGATIVE verdicts for the full 60s TTL, so a single
    transient probe miss blackholed a working backend for a whole minute.
  - the `--version` probe ran only ONCE, so one cold-start / AV-scan / ConPTY
    hiccup scored the backend down.

Fix (3 surgical changes, routing contract untouched):
  1. run.mjs: a health skip is now LOUD — sets lastErr (→ carried into the
     handoff reason), writes a `failed` status record (kind:"health"), and
     appends a structured record to failures.log.
  2. backends.mjs health(): cache POSITIVE verdicts only; a negative verdict is
     never cached → the very next call re-probes and recovers.
  3. backends.mjs _healthUncached(): one-shot retry — probe `--version` up to
     twice before declaring a backend down (happy-path cost ~0).

Tests: +2 regression cases (loud kind:"health" skip; one-shot-retry recovers a
single flaky --version) → 108/108. Verified by codex through the live relay
(VERDICT: PASS, no findings). Doc test-counts 106→108.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: WatchAndyTW

.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
     {
       "name": "multi-model-team",
       "description": "Delegates token-heavy, self-contained tasks to local CLI backends chosen by task size and type, with credit-exhaustion fallback through the backend chain to native Claude and a glanceable statusline HUD. Routing and shipped backend behavior are configured in config/roster.json; adding a new backend kind requires invoker and health support in code. /team runs a decompose → dispatch → verify → fix → synthesize pipeline; an optional config-gated UserPromptSubmit hook can proactively nudge delegation.",
-      "version": "0.8.3",
+      "version": "0.8.4",
       "author": {
         "name": "WatchAndyTW",
         "email": "watchandytw@gmail.com"
@@ -29,5 +29,5 @@
       ]
     }
   ],
-  "version": "0.8.3"
+  "version": "0.8.4"
 }

.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "multi-model-team",
   "displayName": "Multi-Model Team",
-  "version": "0.8.3",
+  "version": "0.8.4",
   "description": "Delegates token-heavy, self-contained tasks to local CLI backends chosen by task size and type, with credit-exhaustion fallback through the backend chain to native Claude and a glanceable statusline HUD. Routing and shipped backend behavior are configured in config/roster.json; adding a new backend kind requires invoker and health support in code. /team runs a decompose → dispatch → verify → fix → synthesize pipeline; an optional config-gated UserPromptSubmit hook can proactively nudge delegation.",
   "author": {
     "name": "WatchAndyTW",

CLAUDE.md

@@ -9,7 +9,7 @@ glanceable statusline HUD.
 (the agy lane runs under a real pseudo-terminal — ConPTY on Windows, forkpty on POSIX); everything
 else is Node stdlib. Cross-platform (Windows/Linux/macOS). `package.json` `"type":"module"`.
 
-**Status:** built, adversarially reviewed, and green. `npm test` passes **106/106** offline
+**Status:** built, adversarially reviewed, and green. `npm test` passes **108/108** offline
 (no backend calls; live agy/codex behaviour is smoke-tested by hand, not via a `npm test` gate).
 Two live backends: **agy** (Gemini)
 and **codex** (OpenAI Codex CLI); opencode remains a config-only stub. codex also serves as the
@@ -352,7 +352,7 @@ fallback hop.
 ## Testing
 
 ```bash
-npm test                         # offline: 106/106 routing + unit tests (no backend calls)
+npm test                         # offline: 108/108 routing + unit tests (no backend calls)
 ```
 
 Keep the suite green. Add cases for any routing or behavior change. Tests live in `test/*.test.mjs`

README.md

@@ -8,7 +8,7 @@ Multi-model orchestration for Claude Code. Route by task, fan out in parallel, f
 
 ![Node](https://img.shields.io/badge/node-%3E%3D18-339933?logo=node.js&logoColor=white)
 ![Type](https://img.shields.io/badge/module-ESM-f7df1e)
-![Tests](https://img.shields.io/badge/tests-106%2F106%20passing-3fb950)
+![Tests](https://img.shields.io/badge/tests-108%2F108%20passing-3fb950)
 ![Platforms](https://img.shields.io/badge/platform-win%20%7C%20linux%20%7C%20macOS-555)
 ![Deps](https://img.shields.io/badge/runtime%20deps-1%20(node--pty)-blue)
 
@@ -302,7 +302,7 @@ docs/INTERFACES.md           module interface contract (Node ESM port signatures
 ## 🧪 Testing
 
 ```bash
-npm test                # offline: 106/106 routing + unit tests (no backend calls)
+npm test                # offline: 108/108 routing + unit tests (no backend calls)
 ```
 
 The suite is fully offline — no backend calls. Live agy/codex behaviour is verified by hand (run a

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "multi-model-team",
-  "version": "0.8.3",
+  "version": "0.8.4",
   "type": "module",
   "private": true,
   "description": "Delegates token-heavy, self-contained tasks to local CLI backends (agy/Gemini, codex) chosen by task size and type, with credit-exhaustion fallback to native Claude and a statusline HUD. Node ESM, zero-build, cross-platform (Windows/Linux/macOS).",

package.json

@@ -388,10 +388,22 @@ async function main() {
 
     let model = modelForTier(beCfg, D_tier) || modelForTier(beCfg, 'standard');
 
-    // Health-gate: an unhealthy backend (or a kind with no invoker) is skipped.
+    // Health-gate: an unhealthy backend (or a kind with no invoker) is skipped. This used to be a
+    // SILENT skip — no lastErr, no failures.log, no status record — so a backend that failed its
+    // `--version` probe vanished into a bare "backend options exhausted" handoff with zero
+    // diagnostics. That made transient probe misses look like random "the CLI refused to run"
+    // failures. Now a health skip is LOUD: it names the cause in lastErr (→ handoff reason), appends
+    // to failures.log, and writes a `failed` status record the orchestrator can poll.
     let healthy = false;
-    try { healthy = await health(beCfg); } catch { healthy = false; }
-    if (!healthy) { fallbackCount++; continue; }
+    let healthErr = '';
+    try { healthy = await health(beCfg); } catch (e) { healthy = false; healthErr = String(e && e.message || e); }
+    if (!healthy) {
+      const why = sanitizeErr(healthErr) || `'${be}' version probe failed (binary missing, not authed, or probe timed out)`;
+      lastErr = `'${be}' health check failed${healthErr ? ` (${sanitizeErr(healthErr)})` : ''}`;
+      writeStatus(statusFile, callId, { state: 'failed', backend: be, model, tier: D_tier, rule: D_rule, kind: 'health', code: 127, elapsed_ms: 0 });
+      logFailure({ backend: be, model, tier: D_tier, rule: D_rule, code: 127, durMs: 0, stderr: why, kind: 'health', callId });
+      fallbackCount++; continue;
+    }
 
     state.start({ id: callId, backend: be, model, rule: D_rule, inChars });
     const startMs = Date.now();

src/bin/run.mjs

@@ -412,24 +412,37 @@ async function _healthUncached(backendCfg) {
   }
 
   const healthFlag = field(backendCfg, 'health') || '--version';
-  const res = await runChild([bin, healthFlag], {
-    hardTimeout: 30_000, // parity with backends.sh `timeout 30`.
-    keepStdinOpen: false, // version probe: stdin closed (= </dev/null).
-  });
-  // Non-empty stdout (CR-stripped) + clean exit. spawnError/ENOENT -> code 127 -> false.
-  const out = res.stdout.replace(/\r/g, '').trim();
-  return res.code === 0 && out.length > 0;
+  // One-shot retry: a SINGLE `--version` probe is flaky on Windows — a cold-start, an AV scan of the
+  // binary, or a momentary ConPTY hiccup can make it time out or return empty even though a real
+  // dispatch seconds later would succeed. A false "unhealthy" verdict silently blackholes the backend
+  // (run.mjs skips it → native handoff). Probe twice before declaring a backend down; a healthy CLI
+  // answers `--version` in well under a second, so the retry cost on the happy path is zero.
+  for (let attempt = 0; attempt < 2; attempt++) {
+    const res = await runChild([bin, healthFlag], {
+      hardTimeout: 30_000, // parity with backends.sh `timeout 30`.
+      keepStdinOpen: false, // version probe: stdin closed (= </dev/null).
+    });
+    // Non-empty stdout (CR-stripped) + clean exit. spawnError/ENOENT -> code 127 -> false.
+    const out = res.stdout.replace(/\r/g, '').trim();
+    if (res.code === 0 && out.length > 0) return true;
+  }
+  return false;
 }
 
-// TTL-memoized health check. Returns the cached boolean if checked within HEALTH_TTL_MS, else
-// re-probes and caches. Keeps the per-hop fallback-chain cost down to one probe per backend per TTL.
+// TTL-memoized health check. Caches POSITIVE verdicts only: a healthy backend is trusted for
+// HEALTH_TTL_MS (keeps the per-hop fallback cost to one probe per TTL), but a NEGATIVE verdict is
+// NEVER cached. A transient probe miss (cold start, AV scan, ConPTY hiccup) therefore costs at most
+// one skipped dispatch, not a full TTL window of silent native-handoffs — the very next call
+// re-probes and recovers. Asymmetric on purpose: a stale "healthy" is cheap (one real call fails →
+// invalidateHealth re-probes), a stale "unhealthy" used to blackhole a working backend for 60s.
 export async function health(backendCfg) {
   const key = _healthKey(backendCfg);
   const cached = _healthCache.get(key);
   const now = Date.now();
-  if (cached !== undefined && (now - cached.ts) < HEALTH_TTL_MS) return cached.ok;
+  if (cached !== undefined && cached.ok && (now - cached.ts) < HEALTH_TTL_MS) return true;
   const ok = await _healthUncached(backendCfg);
-  _healthCache.set(key, { ok, ts: now });
+  if (ok) _healthCache.set(key, { ok: true, ts: now });
+  else _healthCache.delete(key); // never cache a negative verdict; re-probe next call
   return ok;
 }
 

src/lib/backends.mjs

@@ -317,6 +317,86 @@ test('run.mjs --cwd --writable: CLI runs IN the worktree cwd and gets the writab
   assert.doesNotMatch(body, /read-only/, 'writable mode dropped the read-only sandbox flag');
 });
 
+// ── a backend that FAILS its --version health probe is no longer a SILENT skip ────────────────────
+// This was the "sometimes it just fails" bug: a backend that flunked health was dropped with no
+// lastErr, no failures.log, no status record — just a bare "backend options exhausted" handoff. Now
+// a health skip is LOUD: named in the handoff reason, appended to failures.log (kind:"health"), and
+// recorded as a failed status. Fake CLI exits non-zero on --version (and everything else) twice.
+test('run.mjs: a backend that fails health is logged LOUDLY as kind:"health", not silently skipped', () => {
+  const d = tmp('ehealth-');
+  const fake = join(d, process.platform === 'win32' ? 'deadcodex.cmd' : 'deadcodex.sh');
+  if (process.platform === 'win32') {
+    // Always exit 7 with a stderr line — including on --version, so the health probe (×2) fails.
+    writeFileSync(fake, '@echo off\r\necho DEAD_BINARY not authed 1>&2\r\nexit /b 7\r\n');
+  } else {
+    writeFileSync(fake, '#!/usr/bin/env bash\necho "DEAD_BINARY not authed" >&2\nexit 7\n');
+    chmodSync(fake, 0o755);
+  }
+  const r = writeRosterVariant(d, 'r.json', (c) => {
+    c.backends.agy.enabled = false;                 // only codex in play
+    c.defaults.quota_fallback = ['codex', 'native:sonnet'];
+  });
+  const logDir = join(d, 'logs');
+  const { stdout, stderr } = runNode(BIN_RUN, {
+    args: ['--roster', r, '--decision', '{"backend":"codex","model":"","tier":"standard","rule":"team-verify","native":false}', 'review this'],
+    env: { MMT_BE_BIN: fake, MMT_LOG_DIR: logDir },
+  });
+  // 1. Loud stderr banner naming the backend + the health failure kind.
+  assert.match(stderr, /\[mmt\] ERROR: backend 'codex' \(health\) failed/, 'health skip emits a loud [mmt] ERROR banner');
+  // 2. Handoff still happens, but now CARRIES the cause instead of a bare "exhausted".
+  assert.match(stdout, /MMT_NATIVE_HANDOFF/, 'still hands off to native');
+  assert.match(stdout, /last error:.*codex.*health check failed/i, 'handoff reason names the health failure');
+  // 3. Durable failures.log record with kind:"health".
+  const logFile = join(logDir, 'failures.log');
+  assert.ok(existsSync(logFile), 'failures.log written on a health skip (was silent before)');
+  const rec = JSON.parse(readFileSync(logFile, 'utf8').trim().split('\n').pop());
+  assert.equal(rec.backend, 'codex', 'log record names the backend');
+  assert.equal(rec.kind, 'health', 'log record kind is "health"');
+  // 4. A failed status record exists (callId is random — glob the calls/ dir).
+  const callsDir = join(d, 'calls');
+  assert.ok(existsSync(callsDir), 'status calls/ dir created on a health skip');
+  const statusFiles = readdirSync(callsDir).filter((f) => f.endsWith('.status.json'));
+  const st = JSON.parse(readFileSync(join(callsDir, statusFiles[0]), 'utf8').trim());
+  assert.equal(st.state, 'failed', 'status reflects the failed (health) call');
+  assert.equal(st.kind, 'health', 'status kind is "health"');
+});
+
+// ── health(): a transient probe miss recovers on the NEXT call (negative verdict is never cached) ──
+// + one-shot retry: a SINGLE flaky --version no longer scores the backend down. Fake CLI fails its
+// FIRST --version (cold-start hiccup) then succeeds — within one health() call the retry recovers it.
+test('health(): one-shot retry survives a single flaky --version; negative verdict is not cached', async () => {
+  const { health, _clearHealthCache } = await import('../src/lib/backends.mjs');
+  _clearHealthCache();
+  const d = tmp('hretry-');
+  const counter = join(d, 'n.txt');
+  const fake = join(d, process.platform === 'win32' ? 'flaky.cmd' : 'flaky.sh');
+  if (process.platform === 'win32') {
+    // First --version invocation exits 1 (no output); every later one prints a version + exit 0.
+    writeFileSync(fake,
+      '@echo off\r\n' +
+      'if not exist "' + counter.replace(/\//g, '\\') + '" ( echo x > "' + counter.replace(/\//g, '\\') + '" & exit /b 1 )\r\n' +
+      'echo flaky 1.0\r\n' +
+      'exit /b 0\r\n');
+  } else {
+    writeFileSync(fake,
+      '#!/usr/bin/env bash\n' +
+      'if [ ! -f "' + counter + '" ]; then echo x > "' + counter + '"; exit 1; fi\n' +
+      'echo "flaky 1.0"\n');
+    chmodSync(fake, 0o755);
+  }
+  // health() resolves the binary for kind:"codex" via resolveBinary('codex', …) which honors MMT_BE_BIN.
+  process.env.MMT_BE_BIN = fake;
+  try {
+    const cfg = { kind: 'codex', enabled: true, health: '--version' };
+    // The first attempt's --version fails (exit 1), the in-probe retry succeeds -> healthy in ONE call.
+    const ok = await health(cfg);
+    assert.equal(ok, true, 'one-shot retry recovers a single flaky --version within one health() call');
+  } finally {
+    delete process.env.MMT_BE_BIN;
+    _clearHealthCache();
+  }
+});
+
 test('run.mjs (no --writable): CLI gets the read-only sandbox flags (default behaviour unchanged)', () => {
   const d = tmp('eread-');
   const wt = join(d, 'here');