Skip to content

Commit 25956ae

Browse files
feat: implement CDCProcessorDO sharding for scalability
Add CDCShardCoordinatorDO to manage dynamic sharding for CDC event processing. This addresses the bottleneck of having a single CDCProcessorDO per namespace by distributing load across multiple shards based on collection name hash. Key features: - CDCShardCoordinatorDO manages active shards per namespace - getCDCProcessorShard() routes by collection hash for consistency - Fallback to hash-based routing when coordinator unavailable - Configurable min/max shards (default 1-16) Shard naming: CDCProcessor:{namespace}:shard-{id} Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 5e23d17 commit 25956ae

4 files changed

Lines changed: 668 additions & 1 deletion

File tree

.beads/issues.jsonl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,7 @@
188188
{"id":"events-o0r","title":"Add exactOptionalPropertyTypes to tsconfig","description":"tsconfig.json is missing exactOptionalPropertyTypes which provides stricter optional property handling. Add if feasible without major refactoring.","status":"closed","priority":3,"issue_type":"task","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-02T07:23:08.897803-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-02T07:41:14.488055-06:00","closed_at":"2026-02-02T07:41:14.488055-06:00","close_reason":"Closed"}
189189
{"id":"events-o6f","title":"Constrain getJson\u003cT\u003e generic type parameter","description":"sql-mapper.ts getJson\u003cT\u003e has unconstrained generic. Add constraint: getJson\u003cT extends JsonValue\u003e for type safety.","status":"closed","priority":3,"issue_type":"task","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-02T06:40:09.635389-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-02T06:55:05.658364-06:00","closed_at":"2026-02-02T06:55:05.658364-06:00","close_reason":"Closed"}
190190
{"id":"events-okm","title":"Add unit tests for metrics.ts - Analytics Engine integration","status":"closed","priority":1,"issue_type":"task","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-02T08:16:38.729691-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-02T08:21:25.903104-06:00","closed_at":"2026-02-02T08:21:25.903104-06:00","close_reason":"Closed"}
191-
{"id":"events-os7u","title":"Implement CDCProcessorDO sharding","description":"CRITICAL SCALABILITY: Single CDCProcessorDO per namespace is a bottleneck. Implement per-collection or per-shard sharding similar to EventWriterDO. Create CDCShardCoordinatorDO with getCDCProcessorShard(ns, collection) method. Route CDC events by namespace + collection hash.","status":"open","priority":0,"issue_type":"feature","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-02T08:58:13.492586-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-02T08:58:13.492586-06:00","dependencies":[{"issue_id":"events-os7u","depends_on_id":"events-f8t2","type":"blocks","created_at":"2026-02-02T09:01:57.04926-06:00","created_by":"Nathan Clevenger"}]}
191+
{"id":"events-os7u","title":"Implement CDCProcessorDO sharding","description":"CRITICAL SCALABILITY: Single CDCProcessorDO per namespace is a bottleneck. Implement per-collection or per-shard sharding similar to EventWriterDO. Create CDCShardCoordinatorDO with getCDCProcessorShard(ns, collection) method. Route CDC events by namespace + collection hash.","status":"closed","priority":0,"issue_type":"feature","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-02T08:58:13.492586-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-02T16:44:40.530632-06:00","closed_at":"2026-02-02T16:44:40.530632-06:00","close_reason":"Implemented CDCShardCoordinatorDO with getCDCProcessorShard() helper for consistent hash-based routing by collection name","dependencies":[{"issue_id":"events-os7u","depends_on_id":"events-f8t2","type":"blocks","created_at":"2026-02-02T09:01:57.04926-06:00","created_by":"Nathan Clevenger"}]}
192192
{"id":"events-ovm","title":"Restrict CORS on authenticated endpoints","description":"src/index.ts:438-453 - All responses use Access-Control-Allow-Origin: *. Any website can make cross-origin requests to authenticated endpoints. Restrict to known origins for auth endpoints.","status":"closed","priority":2,"issue_type":"task","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-01T15:55:35.387065-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-01T16:25:55.980263-06:00","closed_at":"2026-02-01T16:25:55.980263-06:00","close_reason":"Closed"}
193193
{"id":"events-p3d","title":"Persist EventWriterDO buffer to DO storage","description":"src/event-writer-do.ts - Buffer is in-memory only. DO eviction between ingest() and flush() loses all buffered events. Use ctx.storage.put() on ingest, ctx.storage.delete() after flush.","status":"closed","priority":0,"issue_type":"bug","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-01T15:54:40.039284-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-01T16:25:55.958207-06:00","closed_at":"2026-02-01T16:25:55.958207-06:00","close_reason":"Closed"}
194194
{"id":"events-pli","title":"Add unit tests for scheduler-lock.ts - race conditions, expiration","status":"closed","priority":1,"issue_type":"task","owner":"4130910+nathanclevenger@users.noreply.github.com","created_at":"2026-02-02T08:16:38.012764-06:00","created_by":"Nathan Clevenger","updated_at":"2026-02-02T08:30:50.109306-06:00","closed_at":"2026-02-02T08:30:50.109306-06:00","close_reason":"Closed"}

0 commit comments

Comments
 (0)