fix(runtime): prevent file watcher SIGSEGV from crashing orca serve

[AmethystLiang]

Summary

• Replace the runtime file-watcher worker thread with forked @parcel/watcher child processes so a native FSEvents fault cannot terminate Electron main or headless orca serve.
• Share healthy runtime roots across at most four watcher children, then recover fused shards through a separate quarantine pool capped at four additional children.
• Preserve same-root deduplication while adding queue-aware cancellation, crawl-scoped deadlines, crash fuses, canary health checks, bounded event delivery, IPC backpressure, and bounded directory metadata work.
• Deliver conservative overflow refreshes across interruption gaps and cleanly retire terminal RPC and renderer subscriptions before callers retry.
• Physically abort pending desktop watcher setup during worktree deletion and app shutdown so native handles cannot outlive their owners.

Root cause

The runtime watcher previously ran @parcel/watcher in a worker thread. Worker threads isolate JavaScript execution but still share the host process, so a native watcher.node SIGSEGV or fail-fast during FSEvents delivery or teardown still kills the entire orca serve process and every connected agent session.

A forked child provides the required fault boundary: process death releases native watcher state without running the crash-prone teardown inside the host, while the supervisor can replace the child and resubscribe live roots.

Recovery and resource bounds

• A child crash restarts only its shard, resubscribes its live roots, and emits overflow after recovery so clients refresh state that may have changed during the gap.
• Three crashes within 30 seconds open that supervisor’s fuse.
• Roots from a fused healthy shard enter a bounded quarantine pool; a second fused quarantine ends those watch lifetimes instead of spawning replacements indefinitely.
• Queued cancellation removes only that request. Cancellation or timeout after a native crawl starts replaces the owning shard once and restores surviving roots.
• Runtime event delivery retains at most one active and one bounded pending batch per subscription, with overflow replacing lost precision.
• Directory metadata work is capped at eight concurrent stats per child.
• Healthy operation uses at most four runtime watcher children; degraded operation adds at most four quarantine children.

Compatibility

• macOS and Linux local runtime roots use the crash-isolated watcher process path.
• Windows runtime roots keep the existing recursive fs.watch overflow-refresh path.
• SSH provider watches continue to use the remote provider path and do not enter this pool.
• The packaged watcher entry remains unpacked for ELECTRON_RUN_AS_NODE resolution.

Validation

Recorded on macOS:

• 157 focused tests across supervisor recovery, pool ownership, child protocol and backpressure, desktop cancellation, same-root runtime dedupe, RPC ordering, and renderer cleanup:

  pnpm exec vitest run --config config/vitest.config.ts \
    src/main/ipc/parcel-watcher-process.test.ts \
    src/main/ipc/runtime-watcher-process-pool.test.ts \
    src/main/ipc/parcel-watcher-process-entry.test.ts \
    src/main/ipc/parcel-watcher-event-delivery.test.ts \
    src/main/ipc/filesystem-watcher-local-unsubscribe.test.ts \
    src/main/runtime/file-watcher-host.test.ts \
    src/main/runtime/rpc/methods/files.test.ts \
    src/renderer/src/runtime/runtime-file-client.test.ts

• The built-entry fault harness delivers an event, sends SIGSEGV to the real watcher child, verifies the host survives and replaces the child, then requires a post-recovery event:

  pnpm run build:electron-vite
  node config/scripts/runtime-file-watcher-fault-harness.mjs

• Ten consecutive harness runs passed without increasing the watcher-canary temp-directory count.

Known gaps

• The fault harness drives the built child and source supervisor, not a packaged paired orca serve session or app.asar path.
• Linux fault containment is source-equivalent but has not been live fault-injected.
• Windows runtime and SSH provider paths are unchanged but have not been live-verified for this PR.
• The four-healthy plus four-quarantine memory ceiling is inferred from prior one-child RSS data; direct degraded-state RSS, CPU, and event-loop soak evidence remains follow-up validation.

Fixes #8212
Related: #5308, #7547

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The change replaces worker-thread runtime file watching with supervised Parcel watcher child processes. It adds watcher IPC contracts, event batching with bounded metadata lookup and backpressure, cancellation and timeout handling, crash recovery, crash fuses, canary directories, and shared or quarantined supervisor pools. Runtime file-watch integration now supports shared root subscriptions, terminal errors, abort signals, recovery, and cleanup. RPC and renderer clients handle explicit error and end events. Tests and reliability evidence cover lifecycle, isolation, cancellation, recovery, and fault injection.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is missing required template sections like Screenshots, Testing checklist, AI Review Report, Security Audit, and Notes.	Reformat the description to match the repository template and add the missing sections, including screenshots/no visual change, testing checklist, AI review report, security audit, and notes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 7.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: preventing file-watcher SIGSEGV crashes from taking down orca serve.
Linked Issues check	✅ Passed	The changes address #8212 by isolating watcher crashes, recovering after overflow, and preserving orca serve availability.
Out of Scope Changes check	✅ Passed	The extra watcher, SSH, WSL, renderer, and workflow updates support the crash-isolation and cancellation goals rather than unrelated behavior.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

src/main/ipc/filesystem-watcher.ts (2)

561-567: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

unsubscribe() doesn't abort the in-flight install like the other two cancellation paths.

closeLocalWatcherForWorktreePath (line 618) and closeAllWatchers (line 949) both call inFlight.abortController.abort() right after setting cancelled = true. This function sets inFlight.cancelled = inFlight.listeners.size === 0 (line 566) but never aborts the controller, so when the last listener disconnects normally, the pending native/forked watcher subscription keeps running to completion instead of being cancelled early — the exact wasted work this abort mechanism was introduced to avoid.

♻️ Proposed fix

 function unsubscribe(worktreePath: string, senderId: number): void {
   const rootKey = normalizeRootPath(worktreePath)
   const inFlight = inFlightLocalInstalls.get(rootKey)
   if (inFlight) {
     inFlight.listeners.delete(senderId)
     inFlight.cancelled = inFlight.listeners.size === 0
+    if (inFlight.cancelled) {
+      inFlight.abortController.abort()
+    }
   }

---

521-527: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Pass the abort signal into the WSL watcher path
createWslWatcher doesn’t accept an AbortSignal, so cancelling a local install still lets the WSL snapshot subprocess keep starting until it settles or times out. Thread cancelToken.abortController.signal through that path too.

src/main/runtime/orca-runtime-files.ts (1)

922-955: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Thread the abort signal through the SSH watch path.
watchFileExplorer already receives signal from the RPC layer, but the SSH branch drops it (provider.watch() / registerSshFilesystemWatch()), so an aborted SSH-backed watch still waits for the remote fs.watch request to finish before tearing down. The Windows path is intentionally the synchronous fs.watch branch, so this only needs to be addressed for SSH.

🧹 Nitpick comments (5)

src/main/ipc/parcel-watcher-process-entry.test.ts (1)

276-278: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicate union member in callback type.

The callback type union lists the identical ((err, events) => void) member twice (also repeated in the tests at ~331-333 and ~380-382). Collapse to a single member.

♻️ Proposed cleanup

     let callback:
       | ((err: Error | null, events: { type: string; path: string }[]) => void)
-      | ((err: Error | null, events: { type: string; path: string }[]) => void)
       | undefined

src/main/ipc/parcel-watcher-supervisor-subscribe.ts (1)

73-75: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Document the VITEST branch.

Branching production subscription behavior on process.env.VITEST is non-obvious; add a one-line comment explaining why tests use the in-process fallback (avoids forking a real watcher child).

💬 Suggested comment

+  // Why: under Vitest we cannot fork a real watcher child, so exercise the
+  // subscription path in-process instead.
   if (process.env.VITEST) {
     return subscribeWithInProcessWatcher(dir, callback, opts, hooks)
   }

As per coding guidelines: "When code is driven by a design document or non-obvious constraint, add a brief one- or two-line comment explaining why it behaves that way."

Source: Coding guidelines

src/main/ipc/runtime-watcher-process-pool.ts (2)

113-131: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

forgetRoot has no direct test coverage.

resetForTest and the constructor/assignment paths are exercised by the test file, but forgetRoot (which clears isolatedRoots/failedQuarantineRoots fault history) isn't covered by any test. Given it mutates fault-tracking invariants that assignmentForRoot depends on (the failedQuarantineRoots fuse check on line 134), a regression here could silently let fused roots retry forever or vice versa.

---

113-122: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Only teardown path for all pooled supervisors is named for tests only.

resetForTest() (public) is the sole mechanism that disposes every slot in allSlots. If any production code path (e.g. app shutdown, hot-reinitialization of the pool in file-watcher-host.ts) relies on this method to release pooled watcher child processes, the "-ForTest" naming is misleading and risks being missed/removed by a future refactor that assumes it's test-only. If it truly is test-only, consider whether production callers have a separate, real disposal path for this pool; if not, consider exposing a properly named disposeAll()/shutdown() alongside (or instead of) resetForTest().

♻️ Possible approach

-  resetForTest(): void {
-    for (const slot of this.allSlots) {
-      this.disposeSlot(slot)
-    }
-    this.activeSlots.clear()
-    this.allSlots.clear()
-    this.assignments.clear()
-    this.isolatedRoots.clear()
-    this.failedQuarantineRoots.clear()
-  }
+  /** Disposes every pooled supervisor and clears fault state. Safe to call from production shutdown paths. */
+  disposeAll(): void {
+    for (const slot of this.allSlots) {
+      this.disposeSlot(slot)
+    }
+    this.activeSlots.clear()
+    this.allSlots.clear()
+    this.assignments.clear()
+    this.isolatedRoots.clear()
+    this.failedQuarantineRoots.clear()
+  }
+
+  /** `@deprecated` use disposeAll(); kept for existing test call sites. */
+  resetForTest(): void {
+    this.disposeAll()
+  }

src/main/ipc/runtime-watcher-process-pool.test.ts (1)

40-227: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Repeated pool-construction boilerplate across most tests.

The new RuntimeWatcherProcessPool({ maxSharedSupervisors: 1, createSupervisor: () => { const supervisor = new FakeSupervisor(); supervisors.push(supervisor); return supervisor } }) block is duplicated nearly verbatim in beforeEach (lines 40-50) and 5 more tests (lines 75-82, 108-115, 146-156, 166-173, 195-202). Extracting a small createPool(overrides) helper would reduce duplication and make future changes to the fake-supervisor wiring less error-prone.

♻️ Suggested helper

function createPool(overrides: RuntimeWatcherProcessPoolOptions = {}): RuntimeWatcherProcessPool {
  return new RuntimeWatcherProcessPool({
    maxSharedSupervisors: 4,
    createSupervisor: () => {
      const supervisor = new FakeSupervisor()
      supervisors.push(supervisor)
      return supervisor
    },
    ...overrides
  })
}

Then each test can call pool = createPool({ maxSharedSupervisors: 1 }).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 43246895-3aea-43b7-839a-ac0fd7e53fa5

📥 Commits

Reviewing files that changed from the base of the PR and between 94f6fd4 and 4ea16a2.

📒 Files selected for processing (34)

• config/reliability-gates.jsonc
• config/scripts/runtime-file-watcher-fault-harness.mjs
• electron.vite.config.ts
• src/main/ipc/filesystem-watcher-local-unsubscribe.test.ts
• src/main/ipc/filesystem-watcher.ts
• src/main/ipc/parcel-watcher-canary-directory.ts
• src/main/ipc/parcel-watcher-child-launch.ts
• src/main/ipc/parcel-watcher-crash-fuse.ts
• src/main/ipc/parcel-watcher-event-delivery.test.ts
• src/main/ipc/parcel-watcher-event-delivery.ts
• src/main/ipc/parcel-watcher-host-subscriptions.ts
• src/main/ipc/parcel-watcher-in-process-fallback.ts
• src/main/ipc/parcel-watcher-pending-subscribe.ts
• src/main/ipc/parcel-watcher-process-entry.test.ts
• src/main/ipc/parcel-watcher-process-entry.ts
• src/main/ipc/parcel-watcher-process-failure.ts
• src/main/ipc/parcel-watcher-process-protocol.ts
• src/main/ipc/parcel-watcher-process-subscription.ts
• src/main/ipc/parcel-watcher-process-supervisor.ts
• src/main/ipc/parcel-watcher-process.test.ts
• src/main/ipc/parcel-watcher-process.ts
• src/main/ipc/parcel-watcher-supervisor-subscribe.ts
• src/main/ipc/runtime-watcher-process-pool.test.ts
• src/main/ipc/runtime-watcher-process-pool.ts
• src/main/runtime/file-watcher-host.test.ts
• src/main/runtime/file-watcher-host.ts
• src/main/runtime/file-watcher-worker.ts
• src/main/runtime/orca-runtime-files-watch.test.ts
• src/main/runtime/orca-runtime-files.test.ts
• src/main/runtime/orca-runtime-files.ts
• src/main/runtime/rpc/methods/files.test.ts
• src/main/runtime/rpc/methods/files.ts
• src/renderer/src/runtime/runtime-file-client.test.ts
• src/renderer/src/runtime/runtime-file-client.ts

💤 Files with no reviewable changes (2)

• src/main/runtime/file-watcher-worker.ts
• electron.vite.config.ts

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 9cffd8e8-3f32-43f5-9a4a-46c664c5ee94

📥 Commits

Reviewing files that changed from the base of the PR and between 4ea16a2 and 9198b01.

📒 Files selected for processing (32)

• config/reliability-gates.jsonc
• config/scripts/runtime-file-watcher-fault-harness.mjs
• electron.vite.config.ts
• src/main/ipc/filesystem-watcher-local-unsubscribe.test.ts
• src/main/ipc/filesystem-watcher.ts
• src/main/ipc/parcel-watcher-canary-directory.ts
• src/main/ipc/parcel-watcher-child-launch.ts
• src/main/ipc/parcel-watcher-crash-fuse.ts
• src/main/ipc/parcel-watcher-event-delivery.test.ts
• src/main/ipc/parcel-watcher-event-delivery.ts
• src/main/ipc/parcel-watcher-host-subscriptions.ts
• src/main/ipc/parcel-watcher-in-process-fallback.ts
• src/main/ipc/parcel-watcher-pending-subscribe.ts
• src/main/ipc/parcel-watcher-process-entry.test.ts
• src/main/ipc/parcel-watcher-process-entry.ts
• src/main/ipc/parcel-watcher-process-failure.ts
• src/main/ipc/parcel-watcher-process-protocol.ts
• src/main/ipc/parcel-watcher-process-subscription.ts
• src/main/ipc/parcel-watcher-process-supervisor.ts
• src/main/ipc/parcel-watcher-process.test.ts
• src/main/ipc/parcel-watcher-process.ts
• src/main/ipc/parcel-watcher-supervisor-subscribe.ts
• src/main/ipc/runtime-watcher-process-pool.test.ts
• src/main/ipc/runtime-watcher-process-pool.ts
• src/main/runtime/file-watcher-host.test.ts
• src/main/runtime/file-watcher-host.ts
• src/main/runtime/file-watcher-worker.ts
• src/main/runtime/orca-runtime-files-watch.test.ts
• src/main/runtime/orca-runtime-files.test.ts
• src/main/runtime/orca-runtime-files.ts
• src/main/runtime/rpc/methods/files.test.ts
• src/main/runtime/rpc/methods/files.ts

💤 Files with no reviewable changes (2)

• src/main/runtime/file-watcher-worker.ts
• electron.vite.config.ts

🚧 Files skipped from review as they are similar to previous changes (22)

• src/main/ipc/parcel-watcher-process-subscription.ts
• src/main/ipc/parcel-watcher-process-failure.ts
• src/main/ipc/parcel-watcher-event-delivery.test.ts
• src/main/ipc/parcel-watcher-canary-directory.ts
• src/main/ipc/parcel-watcher-process-protocol.ts
• src/main/runtime/rpc/methods/files.ts
• src/main/ipc/parcel-watcher-crash-fuse.ts
• src/main/runtime/orca-runtime-files.ts
• src/main/runtime/orca-runtime-files.test.ts
• config/scripts/runtime-file-watcher-fault-harness.mjs
• src/main/runtime/file-watcher-host.test.ts
• src/main/ipc/parcel-watcher-process.ts
• src/main/ipc/runtime-watcher-process-pool.test.ts
• src/main/ipc/parcel-watcher-pending-subscribe.ts
• src/main/ipc/parcel-watcher-event-delivery.ts
• src/main/ipc/parcel-watcher-process-entry.test.ts
• src/main/ipc/parcel-watcher-in-process-fallback.ts
• src/main/ipc/parcel-watcher-process.test.ts
• src/main/runtime/rpc/methods/files.test.ts
• src/main/ipc/parcel-watcher-process-entry.ts
• src/main/ipc/filesystem-watcher-local-unsubscribe.test.ts
• config/reliability-gates.jsonc

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/main/ipc/parcel-watcher-host-subscriptions.ts (1)

77-91: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Snapshot records before invoking terminal hooks.

reportWatcherTerminalError runs hooks synchronously. A supervisor-level hook can retire/dispose the supervisor and clear records while this loop is active, so later subscriptions may never receive the terminal error before the map is cleared. Iterate over a snapshot.

Proposed fix

-  for (const record of records.values()) {
+  for (const record of [...records.values()]) {

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 6a0837fe-2e8d-4fe1-a3ab-b04c2110ff7e

📥 Commits

Reviewing files that changed from the base of the PR and between 911daa9 and a3989d5.

📒 Files selected for processing (17)

• .github/workflows/release-cut.yml
• .github/workflows/release-mac-build.yml
• config/reliability-gates.jsonc
• config/scripts/runtime-file-watcher-fault-harness.mjs
• src/main/ipc/filesystem-watcher-wsl.ts
• src/main/ipc/parcel-watcher-host-subscriptions.ts
• src/main/ipc/parcel-watcher-pending-subscribe.ts
• src/main/ipc/parcel-watcher-process-entry.test.ts
• src/main/ipc/parcel-watcher-process-subscription.ts
• src/main/ipc/parcel-watcher-process-supervisor.ts
• src/main/ipc/parcel-watcher-process.test.ts
• src/main/ipc/runtime-watcher-process-pool.test.ts
• src/main/ipc/runtime-watcher-process-pool.ts
• src/main/providers/ssh-filesystem-provider-watch.ts
• src/main/providers/ssh-filesystem-provider.test.ts
• src/main/providers/ssh-filesystem-provider.ts
• src/main/runtime/file-watcher-host.ts

🚧 Files skipped from review as they are similar to previous changes (8)

• src/main/ipc/parcel-watcher-process-subscription.ts
• config/scripts/runtime-file-watcher-fault-harness.mjs
• src/main/ipc/parcel-watcher-pending-subscribe.ts
• src/main/ipc/parcel-watcher-process-entry.test.ts
• src/main/ipc/runtime-watcher-process-pool.ts
• src/main/ipc/parcel-watcher-process.test.ts
• src/main/runtime/file-watcher-host.ts
• src/main/ipc/runtime-watcher-process-pool.test.ts