wip: handover — commit plan done, perf verified, consumer imports pending

Next steps:
- Prove performance is optimal (update bson-js to latest, re-run all benchmarks)
- Fix broken consumer BSON imports (rpc, mongo, broker, framework-debug-api)
- Full test suite verification after consumer migrations

Handover document: .claude/handover.md

Author: marcj

.claude/handover.md

@@ -1,6 +1,6 @@
 # Handover
 
-*Updated 2026-02-08 (Gen 69)*
+*Updated 2026-02-08 (Gen 70)*
 
 ## Init Checklist
 <!-- Next agent: complete these steps IN ORDER before doing anything else. -->
@@ -29,93 +29,90 @@ Test migration: Jest → node:test + @deepkit/run/expect shim
 
 ## Current State
 - **Branch**: `feat/next`
-- **Backup branch**: `feat/next-backup` (points to `5dc5f9120`, predates perf fixes)
-- **Active plan**: `.claude/commit-plan.md` — READY TO EXECUTE (all perf regressions fixed, benchmarks done)
-- **Working on**: Adding bson-js comparison numbers to PR description, then commit plan execution
+- **Backup branch**: `feat/next-backup` (points to pre-perf-fix state, 306+ messy commits)
+- **Active plan**: none — commit plan executed, PR description updated
+- **Working on**: Consumer package migration (broken BSON imports) + performance verification
 - **Dirty files**: none (all committed)
-- **Failing tests**: none — all 360 BSON tests pass, all 1975 type tests pass (4 pre-existing skips)
-- **Master worktree**: `/Users/marc/bude/deepkit-master` — fully built, has compiled benchmarks for type and bson
-- **Verification command**: `git log --oneline -5` (expect: top commit is `perf(bson): fix array serialization/deserialization`)
+- **Failing tests**: none known for bson/type/type-compiler — consumer packages (rpc, mongo, broker, framework-debug-api) have broken BSON imports that will fail at compile
+- **Master worktree**: `/Users/marc/bude/deepkit-master` — fully built, has compiled benchmarks (`packages/bson/tsconfig.bench.json` → `dist/bench/`)
+- **Verification command**: `git log --oneline -5` (expect: 21 semantic commits, top is `chore: update handover...`, below it `docs(website)...`, `fix: update remaining packages...`, etc.)
 
 ## Next Steps
-1. **Add bson-js comparison numbers to PR description** — `docs/pr-description.md` needs the vs-bson-js numbers. User explicitly requested: "these numbers must be mentioned in the pr-description so people know what they get." Use the data from Benchmark Data section below.
-2. **Execute commit plan** — Follow `.claude/commit-plan.md` to squash 306+ commits into semantic history
-3. **Verify final state** — Tests must pass, commit history must be clean
+1. **Prove performance is still optimal** — Run BSON and type benchmarks vs master and bson vs bson-js. bson-js should be updated to newest version first. Benchmark files:
+   - feat/next perf-regression: `node --import @deepkit/run --test packages/bson/tests/serialize/perf-regression.spec.ts` and `tests/deserialize/perf-regression.spec.ts`
+   - master comparison: `cd /Users/marc/bude/deepkit-master && npx tsc -p packages/bson/tsconfig.bench.json && node --expose-gc packages/bson/dist/bench/benchmarks/compare-bench.js`
+   - type benchmarks: `packages/type/benchmarks/compare-bench.ts` (feat/next), master has equivalent in its worktree
+2. **Fix consumer package BSON imports** — See Suppressed Issues for the full list. Key broken packages: rpc, mongo, broker, broker-redis, framework-debug-api. See `docs/pr-description.md` "Open TODOs" section for migration details.
+3. **Verify final state** — Full test suite must pass after consumer migrations
 
 ## Alignment Check
-- **Goal**: Add bson-js comparison numbers to PR description, then clean commit history for PR
-- **Scope boundary**: Do NOT fix consumer package imports yet. The array perf issue is FIXED — no more investigation needed.
-- **Plan file**: `.claude/commit-plan.md` — execute ONLY after benchmark numbers are added to PR description
-- **Recovery**: `git reset --hard feat/next-backup` (loses ALL perf fixes — backup predates them)
+- **Goal**: Get feat/next branch ready for PR — all tests passing, performance proven, clean commit history
+- **Scope boundary**: The commit plan is DONE (21 commits). Focus is now on (a) proving perf is optimal, (b) fixing broken consumer imports. Do NOT re-squash commits.
+- **Plan file**: none — commit plan was executed this session
+- **Recovery**: `git reset --hard feat/next-backup` restores the 306+ commit history (loses clean squash)
 
 ## Tasks
-- [x] **Fix core BSON benchmark shared-buffer bug** — `benchmarks/src/benchmarks/core/bson/bson.bench.ts` destructure+slice pattern. [status: completed]
-- [x] **Run BSON vs bson-js comparison** — Fixed array perf (60-90x improvement). Deepkit now 6-12x faster at ALL sizes. [status: completed]
-- [ ] **Add bson-js numbers to PR description** — User explicitly requested these numbers in docs/pr-description.md. [status: pending]
-- [ ] **Execute commit plan** — Follow `.claude/commit-plan.md`. [status: pending, blocked by PR description]
-- [ ] **Verify final state** — Tests pass, history clean. [status: pending, blocked by commit plan]
+- [ ] **Update bson-js to latest version** — Check if `bson` npm package in the repo is latest. Update and re-run comparison benchmarks. [status: pending]
+- [ ] **Run BSON perf benchmarks vs master** — Run both serialize and deserialize perf-regression tests on feat/next AND master compare-bench.ts. Verify all ratios hold. [status: pending]
+- [ ] **Run type perf benchmarks vs master** — Ensure all 12 type benchmarks still beat master. [status: pending]
+- [ ] **Fix consumer BSON imports (rpc)** — `packages/rpc/src/protocol.ts` uses Writer, getBSONSizer, BsonStreamReader. Needs refactored message construction. HIGH complexity. [status: pending]
+- [ ] **Fix consumer BSON imports (mongo)** — `packages/mongo/src/client/connection.ts` uses BSONBinarySerializer, Writer, getBSONSizer. Needs new serializer composition. HIGH complexity. [status: pending]
+- [ ] **Fix consumer BSON imports (broker)** — `packages/broker/src/snapshot.ts` uses old getBSONSerializer return type. `packages/broker/src/adapters/deepkit-adapter.ts` uses getBsonEncoder. LOW complexity. [status: pending]
+- [ ] **Fix consumer BSON imports (broker-redis)** — Remove AutoBuffer, fix getBsonEncoder → getBSONEncoder. MEDIUM complexity. [status: pending]
+- [ ] **Fix consumer BSON imports (framework-debug-api)** — `packages/framework-debug-api/src/stopwatch-encoding.ts` uses Writer, BaseParser, getBSONSizer, stringByteLength. MEDIUM complexity. [status: pending]
 
 ## Benchmark Data
 
-### @deepkit/type: feat/next vs master (pre-resolved API)
-
-| Benchmark | feat/next | master | Change |
-|-----------|-----------|--------|--------|
-| small deserialize | 40.9M | 21.7M | **+88%** |
-| medium deserialize | 9.6M | 4.3M | **+124%** |
-| union deserialize | 16.5M | 2.2M | **+637%** |
-| small serialize | 130.5M | 19.0M | **+587%** |
-| medium serialize | 2.0M | 1.3M | **+51%** |
-| union serialize | 34.6M | 2.4M | **+1342%** |
-| small validate | 63.8M | 20.4M | **+213%** |
-| medium validate | 20.6M | 4.9M | **+317%** |
-| union validate | 8.5M | 0.5M | **+1605%** |
-| small is | 368.5M | 21.1M | **+1649%** |
-| medium is | 51.9M | 4.9M | **+960%** |
-| union is | 8.9M | 0.5M | **+1679%** |
-
-### @deepkit/bson: feat/next vs master (pre-resolved API)
-
-| Benchmark | feat/next | master | Change |
-|-----------|-----------|--------|--------|
-| serialize int32 | 145.1M | 17.7M | **+720%** |
-| serialize string | 86.2M | 10.9M | **+691%** |
-| serialize MongoId | 33.9M | 9.9M | **+243%** |
-| serialize UUID | 22.3M | 6.1M | **+266%** |
-| serialize sensor (4 fields) | 127.1M | 7.4M | **+1617%** |
-| serialize apiMeta (3 strings) | 14.3M | 5.0M | **+186%** |
-| deserialize int32 | 117.9M | 29.0M | **+307%** |
-| deserialize string | 37.8M | 15.0M | **+152%** |
-| deserialize MongoId | 21.5M | 9.3M | **+131%** |
-| deserialize UUID | 13.2M | 4.6M | **+187%** |
-| deserialize sensor (4 fields) | 21.4M | 13.5M | **+59%** |
-| deserialize apiMeta (3 strings) | 12.2M | 3.7M | **+230%** |
-
-### @deepkit/bson vs bson-js (FINAL — after array perf fix)
-
-| Size | Operation | Deepkit | bson-js | Ratio |
-|------|-----------|---------|---------|-------|
-| 1 item | serialize | 21.7M | 1.85M | **Deepkit 11.7x faster** |
-| 1 item | deserialize | 12.4M | 1.73M | **Deepkit 7.2x faster** |
-| 10 items | serialize | 1.94M | 175K | **Deepkit 11.1x faster** |
-| 10 items | deserialize | 1.05M | 167K | **Deepkit 6.3x faster** |
-| 1K items | serialize | 19.1K | 1,885 | **Deepkit 10.1x faster** |
-| 1K items | deserialize | 10.7K | 1,757 | **Deepkit 6.1x faster** |
-| 10K items | serialize | 1,745 | 182 | **Deepkit 9.6x faster** |
-| 10K items | deserialize | 1,088 | 175 | **Deepkit 6.2x faster** |
+### @deepkit/bson: perf-regression results (Gen 70, fresh run)
+
+**Serialize (feat/next vs bson-js):**
+
+| Benchmark | Ops/sec | vs bson-js |
+|-----------|---------|------------|
+| int32 | 1,174M | 301x |
+| sensor (4 numeric) | 336M | 193x |
+| user profile (6 fields) | 16.6M | 16x |
+| API meta (3 strings) | 15.9M | 8x |
+| sensor[] 10 cursor | 8.7M | 46x |
+| sensor[] 1K cursor | 100K | 43x |
+| mixed[] 10 cursor | 1.9M | 15x |
+| mixed[] 1K cursor | 20K | 14x |
+
+**Deserialize (feat/next vs bson-js):**
+
+| Benchmark | Ops/sec | vs bson-js |
+|-----------|---------|------------|
+| int32 | 187M | 31x |
+| sensor (4 numeric) | 23M | 12x |
+| user profile (6 fields) | 11.5M | 12x |
+| API meta (3 strings) | 13.1M | 8x |
+| sensor[] 10 cursor | 1.7M | 11x |
+| sensor[] 1K cursor | 17K | 10x |
+| mixed[] 10 cursor | 1.0M | 10x |
+| mixed[] 1K cursor | 10K | 10x |
+
+**Master comparison (from same-session run):**
+
+| Benchmark | master | feat/next | Improvement |
+|-----------|--------|-----------|-------------|
+| ser sensor | 7.5M | 336M | 45x |
+| ser sensor[]10 cursor | 758K | 8.7M | 11x |
+| ser sensor[]1K cursor | 8.4K | 100K | 12x |
+| des sensor | 13.1M | 23M | 1.8x |
+| des sensor[]10 cursor | 1.5M | 1.7M | 1.1x |
+| des sensor[]1K cursor | 16.6K | 17K | ~same |
 
 ## Learnings
 - ✅ [2026-02-06] **Type compiler resolves `T | undefined` as optional T, not union** — `prop.optional = true` and `prop.type = UUID`.
 - ✅ [2026-02-07] **`isBsonTypeCompatible` must handle union types** — Returning false for ALL unions disabled shape JIT.
 - ✅ [2026-02-08] **Always use `@deepkit/bench` (`BenchSuite`)** for benchmarks.
 - ✅ [2026-02-08] **Type serializer perf regressions: 3 unbounded fn() rebuilds** — Fix: cache at JIT time.
-- ✅ [2026-02-08] **`guardObjectFast` handles BOTH `is()` AND `validate()`** — Registered for both objectLiteral and class. `guardObjectScore` is only for union scoring.
-- ✅ [2026-02-08] **Medium model perf gap was architectural — FIXED** — Master inlines ALL index signature handling. feat/next now does too via `forKey()` + `state.forKey(key).build()`.
-- ✅ [2026-02-08] **`getBSONSerializer` returns `[sharedBuffer, size]` — buffer is reused across ALL serializers globally**. Callers must copy with `buf.slice(0, size)` if they need to keep the data past the next serialize call.
-- ✅ [2026-02-08] **BSONBuildState `forIndex()` must NOT increment depth** — Array loop bodies share a single code path (no code bloat). Incrementing depth caused array element objects to be extracted to separate functions at depth >= MAX_DEPTH (3), making array serialization 60-90x slower. The MongoDB cursor response `{ cursor: { firstBatch: Item[] } }` hits depth 3 at `Item` — the most common real-world pattern.
-- ✅ [2026-02-08] **Shape JIT must handle primitive arrays (string[], number[], boolean[])** — Without inline support, any document with primitive array fields (e.g., `tags: string[]`) causes shape JIT BAILOUT, forcing ALL elements through the slow `buildDocumentBody` path with per-field name matching.
-- ⚠️ [2026-02-08] **BenchSuite executor does NOT prevent V8 dead-code elimination** — Use a `sink` variable: `suite.add('name', () => { sink = fn(data); })`.
-- ⚠️ [2026-02-08] **Master BSON benchmark needs compiled JS** — Create `tsconfig.bench.json` with `"reflection": true`, compile with `npx tsc -p packages/bson/tsconfig.bench.json`, run from `dist/bench/`.
+- ✅ [2026-02-08] **`getBSONSerializer` returns `[sharedBuffer, size]` — buffer is reused across ALL serializers globally**. Callers must copy with `buf.slice(0, size)`.
+- ✅ [2026-02-08] **BSONBuildState `forIndex()` must NOT increment depth** — Incrementing depth caused extraction at MAX_DEPTH=3, 60-90x slower for cursor responses.
+- ✅ [2026-02-08] **Shape JIT must handle primitive arrays** — Without inline, documents with `string[]`/`number[]` cause shape JIT BAILOUT.
+- ✅ [2026-02-08] **BenchSuite executor does NOT prevent V8 dead-code elimination** — Use `sink` variable.
+- ✅ [2026-02-08] **Master BSON benchmark needs compiled JS** — `tsconfig.bench.json` with `"reflection": true`, compile with tsc, run from `dist/bench/`.
+- ⚠️ [2026-02-08] **BSON array deserialize improvement limited at scale** — Single-doc sensor 1.8x faster, but sensor[]1K cursor ~same speed. Setup/overhead improvement is amortized; per-element byte-reading cost is similar between branches (~55-60ns/sensor). Serialize arrays DO scale (12x faster) due to buffer reuse.
 
 ## Dead Ends
 - [2026-02-04] **"Assumed order" fast-path** — WRONG for MongoDB which returns arbitrary field order.
@@ -128,13 +125,13 @@ Test migration: Jest → node:test + @deepkit/run/expect shim
 - [2026-02-07] `packages/mongo/src/mongo-serializer.ts:10` — Imports `BSONBinarySerializer`, `ValueWithBSONSerializer`. Will fail at compile.
 - [2026-02-07] `packages/mongo/src/client/client.ts:10` — Imports `BSONBinarySerializer`. Will fail at compile.
 - [2026-02-07] `packages/broker-redis/src/broker-redis.ts:13` — Imports `AutoBuffer`, uses `getBsonEncoder` (renamed). Will fail at compile.
-- [2026-02-07] `packages/framework-debug-api/src/stopwatch-encoding.ts:7-9` — Uses removed BSON internals. Will fail at compile.
+- [2026-02-07] `packages/framework-debug-api/src/stopwatch-encoding.ts:1-9` — Imports `Writer`, `BaseParser`, `getBSONSizer`, `stringByteLength`. Will fail at compile.
 - [2026-02-07] `packages/broker/src/snapshot.ts` — Uses `getBSONSerializer` return as `Uint8Array` (actually `[Uint8Array, number]`). Will fail at runtime.
 - [2026-02-07] `packages/broker/src/adapters/deepkit-adapter.ts` — Uses `getBsonEncoder` (renamed). Will fail at compile.
 - [2026-02-08] `packages/type/tests/serializer.spec.ts:1153,1180,1204,1363` — 4 tests skipped (`test.skip`): onLoad call (x3), extend with custom type. Pre-existing Jest skips.
 
 ## Open Questions
-*None — array perf issue resolved.*
+- **BSON array deserialize does NOT benefit from single-doc speedup** — Per-element cost ~55-60ns on both branches. Setup overhead (33ns improvement) is amortized across 1000 elements. Optimization target: investigate if shape JIT can reduce per-element overhead for array deserialization.
 
 ## Generation Log
 - [2026-02-04] Gen 1-35: BSON rewrite foundations
@@ -153,7 +150,8 @@ Test migration: Jest → node:test + @deepkit/run/expect shim
 - [2026-02-08] Gen 66: Benchmarked both branches. Fixed 9 uncached guard rebuilds. Medium validate/is still slow.
 - [2026-02-08] Gen 67: Inlined index signatures. ALL 12 type benchmarks beat master.
 - [2026-02-08] Gen 68: Ran BSON benchmarks. Found shared-buffer corruption bug in benchmarks. Fixed: all 22 BSON benchmarks beat master. Fixed bson-js comparison benchmark for new API.
-- [2026-02-08] Gen 69: Fixed BSON array perf — 60-90x improvement. Root cause: (1) `forIndex()` incremented depth causing extraction at MAX_DEPTH=3, (2) shape JIT bailed out on primitive arrays. Deepkit now 6-12x faster than bson-js at ALL sizes. Added 12 array perf regression tests (sensor[], mixed[], string[], number[] at 10 and 1000 items in cursor response). Fixed core bson benchmark shared-buffer bug.
+- [2026-02-08] Gen 69: Fixed BSON array perf — 60-90x improvement. Deepkit 6-12x faster than bson-js at ALL sizes.
+- [2026-02-08] Gen 70: Ran all 53 BSON perf-regression tests (pass). Ran master comparison benchmarks. Updated PR description with 3 numbers per benchmark (ops/sec, vs master, vs bson-js). Executed commit plan — squashed 306+ commits into 21 clean semantic commits. Discovered array deserialize does not scale with single-doc improvement.
 
 ---
-*End of handover. Next: add bson-js numbers to docs/pr-description.md, then execute commit plan.*
+*End of handover. Next: prove perf is optimal (update bson-js, re-run benchmarks), then fix consumer BSON imports.*