feat(duckdb-node): DuckDB backend for buckaroo-js-core (#930) - #936
Conversation
New package `packages/buckaroo-duckdb-node` — a DuckDB-backed buckaroo backend for pure Node/Electron hosts (no Python kernel) that produces the same wire payloads buckaroo-js-core's DFViewerInfinite already speaks. Implemented (v1, transport-agnostic, zero native bindings in core): - rename: DESCRIBE -> a,b,c space + synthesized index (base-26 matching df_util.py:to_chars); removes index-collision/dotted-name/dup-name foot-guns - query: effective-query seam + infinite_request -> sorted/windowed/renamed SQL (index assigned pre-sort so rows keep their position) - stats: SUMMARIZE -> SDType -> pivoted stat rows (dtype, null_count derived, distinct_count, mean/std/min/q25/q50/q75/max) - column config: DuckDB type -> displayer with DefaultMainStyling parity - serialization: COPY -> tmpfile parquet no-coercion path + @duckdb/node-api adapter (separate ./node-api entry) - transport: IModel-over-IPC adapter for Electron (renderer <-> main) Spike (test/spike.duckdb.test.ts, real DuckDB via @duckdb/node-api): - BIGINT >2^53 round-trips exactly as bigint; DATE/TIMESTAMP/NULL exact - DECIMAL(38,9) and HUGEINT decode as JS double (lossy) -- DECIMAL matches the plan's "v1 casts DECIMAL->DOUBLE; #934" - temp-file-per-window latency negligible (~3ms COPY + ~5ms read / 101 rows), answering the plan's open question (no ramdisk needed) 45 vitest tests pass; tsc clean. End-to-end rendering is blocked on #933 (renderer needs decodeDFData to consume the inline parquet_b64 reply); the producer side here is complete and tested against the contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📦 TestPyPI package publishedpip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.1.dev28055374933or with uv: uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.1.dev28055374933MCP server for Claude Codeclaude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.1.dev28055374933" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table📖 Docs preview🎨 Storybook preview |
examples/duckdb-ws-server: a Node http+ws server that stands the backend behind buckaroo's existing browser bundle (buckaroo/static/standalone.js) so the viewer renders live against DuckDB today -- infinite scroll, sort, summary stats over a synthetic 250k-row table, no Python kernel. It works pre-#933 by sending each row window as a binary parquet frame paired with the infinite_resp JSON frame -- the legacy wire WebSocketModel already decodes via parquetRead(buffers[0]). Same backend logic as the IPC transport; only the framing differs. Run with `pnpm demo`, open http://localhost:8780/. Also switch the package to NodeNext module resolution (relative imports carry .js extensions) so dist/ is valid Node ESM -- required for the adapter and the demo to import the built package. Add `ws` devDep + `demo` script. Verified end to end: initial_state (50k rows, 9 renamed cols + stats), a sorted window decodes correctly, BIGINT>2^53 preserved as bigint. 45 tests still pass; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: deaf9e5e0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }, | ||
| df_display_args: { | ||
| main: { | ||
| data_key: this.dataKey, |
There was a problem hiding this comment.
When a caller passes new DuckBackend(..., { dataKey: 'orders' }), this value is sent as df_display_args.main.data_key. The existing Buckaroo viewer only wires the infinite datasource when that field is exactly main; any other key is treated as a preloaded static array, and this backend initializes it to [], so the grid renders empty and never issues infinite_requests. Either keep the live data_key as main or update the viewer contract before exposing this option.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a71d73b by removing the dataKey option entirely and hardcoding 'main'. You're right: BuckarooWidgetInfinite.getDataWrapper only wires the on-demand infinite datasource when data_key === 'main'; any other key is read as a preloaded static array out of df_data_dict (which the backend leaves []), so the grid would render empty and never issue infinite_request. Making it configurable was a footgun with no upside for a single live source, so it's gone. Backend test tightened to assert the key.
| const n = Number(trimmed); | ||
| return Number.isFinite(n) ? n : v; |
There was a problem hiding this comment.
When a VARCHAR column's SUMMARIZE min/max is numeric-looking, such as ZIP codes or IDs like 00123, this helper is used for min/max and coerces the value to a number, stripping leading zeros and changing the displayed pinned stats. The conversion should be based on the DuckDB column type/stat, or min/max for string columns should remain strings.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a71d73b. min/max are now coerced to a number only for integer/float columns (decided via duckTypeToColType); VARCHAR/date/boolean min/max keep the raw SUMMARIZE string. A zero-padded value like '00123' no longer loses its leading zeros. Added a failing test (a6fcdd3) with a numeric-looking VARCHAR column before the fix.
A zero-padded VARCHAR min/max (ZIP codes, IDs like '00123') was coerced to a number by the shared maybeNumber helper, stripping leading zeros in the pinned summary stats. Add a failing fixture column to lock the contract. Codex PR #936 review (P2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new package had no CI coverage — its vitest suite never ran on CI. Add a build+test step (SKIP_DUCKDB=1 for the pure-logic tests; the native DuckDB spike stays local) so the failing-test-first workflow is meaningful. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ain' Codex PR #936 review (two P2s): - stats: min/max are now coerced to numbers only for integer/float columns (via duckTypeToColType). VARCHAR/date/boolean min/max keep their raw SUMMARIZE string, so zero-padded values ('00123') no longer lose leading zeros in the pinned stats. - backend: drop the configurable dataKey option and hardcode 'main'. The viewer (BuckarooWidgetInfinite.getDataWrapper) only wires the on-demand infinite datasource when data_key === 'main'; any other key is read as a preloaded static array (which we leave empty), so the grid would render empty and never issue infinite_request. Tighten the backend test to assert the key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#933 (unified DF transport) finalizes the consumer side: js-core's decodeDFData(msg.payload, buffers) decodes the inline parquet_b64 infinite_resp payload, so the single-JSON-message reply renders end-to-end with no binary side-channel frame. Align the producer to the landed contract: - Widen DFEnvelope.layout to 'wide' | 'row' to match js-core's final shape (DFWhole.ts:258-261); the plan doc had only 'wide'. - Drop the "BLOCKED ON #933 / do not ship" framing from transport.ts, wireTypes.ts, README, and the WS demo; cite #933 as the established decode contract instead. - Clarify the WS demo deliberately uses WebSocketModel's binary-frame path (self-contained over plain WebSocket), while production IPC uses parquet_b64 + decodeDFData. No behavior change (type widening is a superset); decode logic lives in js-core. Build clean, 44 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Manually-dispatched (workflow_dispatch) publish for the new buckaroo-duckdb-node package, modeled on release-npm.yml: version input (coupled by convention to the buckaroo release, e.g. 0.15.2), dry_run toggle, OIDC Trusted Publishing + provenance, and an already-published skip guard. Kept standalone rather than folded into release.yml's release-npm job while the package is new, so it publishes only on explicit trigger. The header documents the one-time bootstrap (manual first publish at 0.15.2, then register the Trusted Publisher) and that it can later merge into release.yml for automatic co-publishing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aligns buckaroo-duckdb-node with the buckaroo release line ahead of its initial publish via release-buckaroo-duckdb-node.yml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
New package
packages/buckaroo-duckdb-node— a DuckDB-backed buckaroo backend for pure Node/Electron hosts with no Python kernel. It produces the same wire payloads (initial_state,infinite_resp, wide summary stats) that buckaroo-js-core'sDFViewerInfinitealready speaks, so the viewer renders behind DuckDB exactly as it does behind pandas/polars. Implements #930; design indocs/plans/930-duckdb-node-backend.md.What's here (v1)
Transport-agnostic, core imports zero native bindings:
DESCRIBE→ buckaroo'sa,b,c…space + synthesizedindex(base-26 matchingdf_util.py:to_chars). Removes theindex-collision, dotted-name, and duplicate-name foot-guns.infinite_request→ sorted/windowed/renamed SQL (indexassigned pre-sort so rows keep their position).SUMMARIZE→SDType→ pivoted stat rows (dtype, null_count derived from count×null%, distinct_count, mean/std/min/q25/q50/q75/max).DefaultMainStylingparity.COPY → tmpfile parquetno-coercion path + a@duckdb/node-apiadapter (separate./node-apientry).IModel-over-IPC adapter for Electron (renderer ⇄ main).Spike findings
test/spike.duckdb.test.tsexercises a real DuckDB via@duckdb/node-api:BIGINT(incl. > 2^53)bigintDATE/TIMESTAMPDateVARCHAR,NULLstring/nullDECIMAL(38,9)number(double)HUGEINTnumberTemp-file-per-window latency is negligible (~3ms COPY + ~5ms read per 101-row window), answering the plan's open question — no ramdisk needed.
Blocked on #933 (only the renderer side)
The producer side (the
parquet_b64inline envelope) is complete and unit-tested against the contract. End-to-end rendering needs #933'sdecodeDFData(msg.payload, buffers)— today the infinite path doesparquetRead(buffers[0])and can't consume a single-JSON inline-parquet reply. Don't ship the renderer integration before #933.Tests
45 vitest tests pass;
tscclean. The DuckDB spike skips cleanly withSKIP_DUCKDB=1(optional peer dep). As a greenfield package with no CI wiring yet, tests are bundled with the implementation rather than committed failing-first.🤖 Generated with Claude Code