RFC: on-device voice dictation — set conditions, waste, labels, and notes while you trace

[QuisutDeus1]

The idea

An estimator's hands are busy: one on the mouse tracing, one on the keyboard for tool keys. Everything else — picking a condition, setting waste, naming a label, leaving a room note — steals the hands away from measuring. Voice fixes that: push-to-talk dictation that sets takeoff metadata while you keep tracing.

"carpet one, waste seven" → activates/creates condition CPT-1, sets waste 7%
"label phase two" → arms the shape label "Phase 2" for subsequent traces
"note: verify sheet vinyl in soiled utility with GC" → attaches a note

This is dictation, not conversation — v1 has no AI in the loop at all. It's a grammar over actions the app already exposes.

Hard constraint: nothing leaves the browser

OpenTakeoff's pledge is client-only. Audio must be processed on-device. Two candidate paths, and part of this RFC is picking one with benchmarks:

1. Whisper-class WASM (e.g., whisper.cpp compiled to WASM, tiny/base English models, lazy-loaded like fflate/pdf-lib already are). Pro: quality, consistency across browsers. Con: ~40–150 MB model download (must be opt-in + cached), CPU cost on low-end machines.
2. Web Speech API fallback. Pro: zero download. Con: on Chrome it ships audio to a speech service — that violates the pledge, so it could only ever be offered behind an explicit "uses your browser's cloud recognizer" consent, or not at all. Firefox/Safari support is inconsistent.

Recommendation to validate: WASM primary, no silent cloud fallback. If the model download is unacceptable, the feature simply isn't enabled — never a quiet privacy downgrade.

Scope (v1)

• Push-to-talk (hold a key — proposal: V) with a live transcript chip near the HUD; release to commit, Esc to discard.
• Grammar, not free text, parsed deterministically from the transcript:
  • <finish tag> [waste <n>] — activate (or offer to create) a condition. Tag vocabulary comes from the live conditions list + common Div-9 patterns (CPT/LVT/VCT/CT/RB/TR + number).
  • label <text> / clear label — the #111 shape-label vocabulary.
  • note <text> — freeform note attached to the active sheet (or selected shape).
  • waste <n> — active condition's waste %.
• Everything routes through existing app actions — the parser produces the same calls the UI produces. No new mutation paths.
• Explicit non-goals for v1: tool switching by voice, drawing by voice, conversational agent, always-on listening.

Why this belongs in OpenTakeoff

No takeoff tool on the market has hands-free metadata entry. It compounds with everything else here: conditions and labels are exactly the fields whose provenance the capture layer records, and a spoken transcript is honest evidence of intent. It also matters for accessibility — RSI and single-handed users get a first-class path.

Testing bar (a PR is reviewable only with ALL of this)

This feature touches microphone permissions, heavy WASM, and text that mutates takeoff data — the test bar is deliberately high:

1. Grammar parser: pure + exhaustively unit-tested. The transcript→intent parser must be a pure module (no DOM) with table-driven tests: every grammar production, ambiguous inputs ("carpet one seven" — tag+waste or tag only?), homophone traps ("waist seven"), numbers as words vs digits, rejected/garbage input (must produce no action, never a guess), and locale number formats.
2. Fixture audio corpus. A committed set of short WAV fixtures (multiple speakers/accents/noise profiles, recorded for this project — no third-party audio) with expected transcripts + expected intents. CI runs the WASM recognizer over the fixtures headlessly and asserts end-to-end intent accuracy ≥ an agreed floor; regressions fail the build.
3. Permission + lifecycle states, all tested manually and documented: mic denied, mic revoked mid-hold, no mic device, tab backgrounded mid-dictation, model still downloading when V is pressed (clear "not ready" affordance, never a silent drop), model download failure/retry, and cleanup on unmount (no orphaned audio contexts — verify with chrome://media-internals).
4. Cross-browser matrix documented in the PR: latest Chrome, Firefox, Safari (incl. one Apple-Silicon and one x86 run); WASM SIMD availability differences noted; graceful feature-absence (button hidden, not broken) where unsupported.
5. Performance budget: recognizer must not degrade canvas interaction — pan/zoom stays smooth while decoding (decode in a Worker, never the main thread); memory ceiling documented; model load lazy + cached (verify second-load path).
6. Privacy proof: a test (and a line in the PR description) demonstrating zero network requests during dictation — the existing "No telemetry" stance, extended to audio.
7. Mutation safety: voice-produced actions must be identical to their UI-produced equivalents (assert deep-equal on resulting state in tests) — no voice-only code path may touch shapes/conditions directly.

Where the seams are (for whoever picks this up)

• Conditions + waste: the condition bar's activate/create/update paths in web/src/pages/TakeoffCanvas.jsx (addCondition, updateCond, activateCondition).
• Shape labels: the #111 label vocabulary (shape_labels, assignShapeLabel, active-label state).
• Lazy heavy deps precedent: web/src/lib/ingest.js (fflate/pdf-lib load on demand).
• Design tokens for the transcript chip: web/src/styles/tokens.css.
• Keyboard handling (for the push-to-talk key, mind existing single-letter tool keys): the keydown effects in TakeoffCanvas.jsx.

Slices

This RFC is deliberately sliced so each piece lands independently against the bar above.

1. Pure grammar parser + test table — landed in Feat/59 voice intent parser #79.
2. Wiring intents to canvas actions — landed in feat(voice): wire intents into canvas actions (RFC #59 slice 2) #80. Voice-produced state is asserted deep-equal against UI-produced state.
3. Deixis ("carpet one, this room" — cursor position as the one-click seed) — landed in feat(voice): deixis — cursor-seeded one-click ("this room") per RFC #59 design #83, design record in the comments below.
4. The recognizer — WASM STT, fixture audio corpus, mic lifecycle, browser matrix, privacy proof. Claimed. This is the slice the testing bar was written for, and it does not land without all seven items.
5. Two-tier router — deterministic grammar on the fast path, falling through to the in-canvas agent loop for compound asks. Seam is in place, design not settled.

Slice 5 is open. Post a design comment before a PR.

[angguntrie3-lgtm]

Hi! I noticed this issue and would love to help fix it. I'm currently analyzing the best approach and will open a PR soon.

[karthikyeluripati]

Opened #79 — the parser-only slice this issue invited: parseVoiceIntent(transcript, ctx) as a pure module (no DOM, no deps), 79 table-driven tests covering every grammar production plus the ambiguity/homophone/word-vs-digit/locale traps called out above. "carpet one seven" rejects rather than guessing tag-only, per the never-a-guess bar. No audio/WASM/UI yet — that's next. Would appreciate a look when you get a chance.

[karthikyeluripati]

Slice 2 is up: #80 — the wiring. Parsed intents now drive the real actions: activation and creation go through the same activateCondition/mintCondition paths the buttons use, waste patches by id, labels learn the vocabulary, and notes land as text markups. Mutation safety is covered two ways — ordered call-log tests (each intent emits exactly the UI's call sequence, rejections emit nothing) and deep-equal state-model tests (voice vs. the equivalent UI script on identical seeds).

Input this slice is a typed Command box in the toolbar — the same grammar surface push-to-talk will feed, and the keyboard/accessibility path meanwhile.

Also carries the #79 review note: live tags now return the ctx literal, so a condition tagged cpt 1 is directly actionable by exact finish_tag lookup.

Recognizer slice (bullets 2–6: WASM STT, audio corpus, mic lifecycle, browser matrix, privacy proof) is next — I'll claim it once this lands.

[QuisutDeus1]

Design for the deixis slice — promised in the #80 review ("voice plus cursor is the real hands-free trace loop"). This comment is the record; code follows on it. I'm taking this slice. The recognizer slice (bullets 2–6 of the testing bar) stays claimed by @karthikyeluripati and nothing here touches it — deixis rides the same grammar surface the Command box feeds today and push-to-talk will feed later.

What it is. "carpet one, this room" — the utterance carries the condition/waste/label, the cursor carries the where. While the pointer is on the canvas, this room / here / this one resolves the current cursor position into a one-click seed: arm CPT-1 → one-click at cursor → commit. Voice supplies everything except aim, because aim is the one thing the hand was already doing.

Grammar (parser stays pure). One new terminal production: an utterance may end with a deixis token (this room, here, this one, right here). The parser emits a new intent kind:

| { kind: "trace_at_cursor"; tag?: string; known?: boolean; waste?: number; label?: string }

The parser knows nothing about pointers — it marks that the speaker aimed, never where. Bare deixis ("this room" alone) is valid iff a condition is active; with none active it rejects loudly, per the never-a-guess bar. Deixis riding note or clear label rejects (nothing to aim at). Every existing production is unchanged.

Seed resolution (caps layer). VoiceCapabilities grows one capability: getAimSeed(): { x, y, sheetId } | null. TakeoffCanvas implements it off the existing crosshair position ref — the same tracker the live-measure readout reads. No second tracker. Two guards, both spoken/visible rejects, never a silent no-op:

• pointer not over a sheet → reject ("aim at a sheet")
• stale aim → reject: the position must have updated since the utterance began, so a pointer parked off-canvas or a tab refocus can't inherit a ghost seed

The who-aimed-it rule (the provenance line). Human-deixis commits DIRECT: the human aimed the cursor, so the trace goes through the exact oneClickAt path a physical click takes and lands as human work — undo covers it, capture records the same one-click origin. Agent-inferred placement (no human aim — the #63 loop deciding where on its own) keeps going through the agentProposals accept gate, unchanged. The line is aim: whoever put the cursor there owns the work.

One deliberate call inside that: voice deixis commits the flood result in one utterance — no preview-then-Enter. The spoken imperative is the confirmation; a hands-free loop that ends in "now press Enter" isn't hands-free. The one-click reject paths (open_boundary, tiny) speak their failure instead of committing garbage, exactly as the click path surfaces them today.

Non-goals (seams left, per the #80 thread). No two-tier router yet — an unparsed transcript still dead-ends red; grammar-first-then-#63-agent is its own slice after the recognizer. No tool switching, no drawing by voice beyond the seeded one-click.

Tests. voiceIntent.test.ts: table rows for every deixis token × (tag | bare | +waste | +label), bare-deixis-with-no-active-condition reject, deixis-on-note reject. voiceActions.test.ts: seed handoff (ordered call-log — exactly the click path's sequence), who-aimed-it (direct commit, never a proposal row), off-sheet and stale-aim rejects (message asserted, zero mutation), and deep-equal state vs. the equivalent aimed click.

If the aim-is-the-line rule collides with anything you're planning on the recognizer side, flag it here — otherwise I'm building on this.