docs: plan unified dataframe transport layer by paddymul · Pull Request #933 · buckaroo-data/buckaroo

paddymul · 2026-06-23T17:56:29Z

Summary

Design doc for a comprehensive, transport-agnostic dataframe payload mechanism. Today the "how is a dataframe encoded/decoded" concern is smeared across resolveDFData.ts, a bare parquetRead in BuckarooWidgetInfinite.tsx, three duplicated Python infinite_resp senders, sd_to_parquet_b64, and _df_to_parquet_b64_tagged — and the two decode paths are incompatible (inline-b64-in-one-JSON crashes the infinite path because it only ever reads buffers[0]).

This plan collapses that into one place per side:

JS: a single decodeDFData(envelope, buffers?) => Promise<DFData>, awaited only at the ~3 ingestion edges. Components stay synchronous on plain DFData; the fragile sync resolveDFData is deleted.
Python: a single encode_df(df, transport, …) => (envelope, buffers), capability-driven (comm→buffer, static→b64) with an explicit fmt override.

Envelope

Tagged union on format, names self-describing, no legacy alias:

parquet_buffer — parquet bytes in a comm side-channel buffer (buffer_index)
parquet_b64 — base64 parquet inline (unchanged from today)
json — record array inline

Nested under a payload key on comm messages, kept separate from cache-protocol routing fields. This is forward-compatible with the merged PR #719 cache redesign, whose unfinished phase 7c adds populate/sort/filter message kinds — decodeDFData(msg.payload, buffers) becomes the one transport primitive shared across all of them.

Also covered

Fixes a latent bug: the infinite buffer path never runs parseParquetRow, so main-table list/dict cells currently render as raw JSON strings while the b64 static path parses them. Unifying fixes this (with a test).
TDD sequence per repo rules: failing tests committed and seen failing on CI first, then the fixes.

Doc: docs/plans/unified-df-transport.md. No code changes in this PR — plan only.

🤖 Generated with Claude Code

Design for a single encode/decode interface that handles parquet-in-buffer, parquet-b64-inline, and json-records across DFViewer, DFViewerInfinite, BuckarooInfiniteWidget, and DFViewerInfiniteDS. Collapses the two parallel decode paths (resolveDFData + the bare parquetRead in BuckarooWidgetInfinite) into one decodeDFData primitive awaited only at ingestion edges, and the three duplicated Python infinite_resp senders + summary-stats + artifact paths into one encode_df producer. Envelope nests under `payload`, forward-compatible with the PR #719 populate/sort/filter cache redesign. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ffc4859901

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-23T17:57:53Z

+hand-build `{"type":"infinite_resp", 'key':…, 'data':[], 'length':…}` +
+`[to_parquet(slice)]`. Replace each with a shared helper that calls
+`encode_df(slice, 'comm')` and sends
+`{"type":"infinite_resp", 'key':…, 'length':…, **envelope}` plus the buffers.


Use the decided payload envelope shape

This sender instruction spreads **envelope onto the top-level infinite_resp, but the later decided message shape in this same plan says the envelope must live under payload and the JS should call decodeDFData(msg.payload, buffers). If an implementer follows this section, msg.payload is undefined and the documented decoder returns [] for nullish input, so infinite responses would silently drop rows; please update this example to send "payload": envelope instead.

Useful? React with 👍 / 👎.

Good catch — fixed in eba8518. The 'Consolidate the three senders' example spread **envelope onto the top-level infinite_resp, contradicting this plan's own decided shape (envelope nested under payload, decoded via decodeDFData(msg.payload, buffers)). Following it would send no payload key, so decodeDFData returns [] for nullish input and silently drops rows. The example now sends "payload": envelope.

github-actions · 2026-06-23T17:58:52Z

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.1.dev28052835944

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.1.dev28052835944

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.1.dev28052835944" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

…note The 'Consolidate the three senders' example spread **envelope onto the top-level infinite_resp, contradicting the plan's own decided message shape (envelope nested under 'payload', decoded via decodeDFData(msg.payload)). An implementer following the example would send no 'payload' key, so decodeDFData returns [] for nullish input and silently drops rows. Also note xorq builds buffers via window_to_parquet, not to_parquet directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…de_df) Per the unified-df-transport plan, commit the failing tests first: - JS: decodeDFData over all envelope shapes (parquet_buffer, parquet_b64, json, plain DFData passthrough), incl. layout='wide' pivot, buffer_index selection, and JSON-parse of list/dict cells on the parquet_buffer path — the latent infinite-path bug where object cells render as raw JSON strings. - New fixture main_table_object_cells_parquet_b64.json (main-table parquet with list/dict object columns, via serialization_utils.to_parquet). - Python: encode_df / buffer_payload / b64_payload envelope shapes per (transport, fmt), and resolve_summary_stats_payload still decoding wide payloads routed through the new encoder. These reference decodeDFData/encode_df which don't exist yet, so they fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Collapse the smeared "how is a dataframe encoded/decoded" concern into one place per side, per docs/plans/unified-df-transport.md. JS — one decoder, awaited only at ingestion edges: - Add DFEnvelope tagged union (parquet_buffer | parquet_b64 | json) to DFWhole.ts; DFDataOrPayload = DFData | DFEnvelope. - decodeDFData(env, buffers?) in resolveDFData.ts is the single transport primitive; decodeDFDataDict decodes a whole df_data_dict. Delete the fragile sync resolveDFData / resolveDFDataAsync / preResolveDFDataDict. - Wire the ingestion edges: the infinite comm handler now does decodeDFData(msg.payload, buffers); the df_data_dict trait hooks (BuckarooView/ServerView) and the static mount (BuckarooStaticTable, packages/js static-embed/standalone/widget) all decode at the edge so the component tree stays synchronous on plain DFData. Python — one encoder, capability-driven: - encode_df(df, transport, layout=, fmt=) + buffer_payload / b64_payload build the envelopes; make_infinite_resp nests a parquet_buffer envelope under `payload` (separate from key/length routing fields). - Route sd_to_parquet_b64 and artifact._df_to_parquet_b64_tagged through b64_payload; migrate every infinite_resp sender (anywidget pandas/polars/ xorq, lazy polars, and the websocket loaders) to the payload envelope, dropping the stale data:[] placeholder. Fixes the latent infinite-path bug: the buffer path never ran parseParquetRow, so main-table list/dict cells rendered as raw JSON strings while the static b64 path parsed them. Both now flow through decodeDFData, which JSON-parses object cells on either path (covered by a test over real Python-encoded parquet bytes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

examples/duckdb-ws-server: a Node http+ws server that stands the backend behind buckaroo's existing browser bundle (buckaroo/static/standalone.js) so the viewer renders live against DuckDB today -- infinite scroll, sort, summary stats over a synthetic 250k-row table, no Python kernel. It works pre-#933 by sending each row window as a binary parquet frame paired with the infinite_resp JSON frame -- the legacy wire WebSocketModel already decodes via parquetRead(buffers[0]). Same backend logic as the IPC transport; only the framing differs. Run with `pnpm demo`, open http://localhost:8780/. Also switch the package to NodeNext module resolution (relative imports carry .js extensions) so dist/ is valid Node ESM -- required for the adapter and the demo to import the built package. Add `ws` devDep + `demo` script. Verified end to end: initial_state (50k rows, 9 renamed cols + stats), a sorted window decodes correctly, BIGINT>2^53 preserved as bigint. 45 tests still pass; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…on_utils Replace Dict[str, Any] envelope shapes with TypedDicts mirroring the JS DFEnvelope union: ParquetBufferEnvelope / ParquetB64Envelope / JsonEnvelope (+ Envelope union and InfiniteRespMessage), with Layout / Transport / Fmt Literal aliases. buffer_payload / encode_df / make_infinite_resp now carry precise return types instead of Dict[str, Any]; layout is Optional[Layout] not Any. Rename b64_payload → parquet_b64_payload at all call sites (sd_to_parquet_b64, artifact, tests) for symmetry with buffer_payload. Also tighten neighboring annotations: pd_to_obj returns List[Dict[str, Any]] (it always returned a record array, not a dict), to_parquet -> bytes, sd_to_parquet_b64 -> Union[ParquetB64Envelope, List[...]], and the JSON/ pyarrow cell helpers. No runtime behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

paddymul · 2026-06-23T19:16:54Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0bc9e3f7e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-23T19:21:53Z

+                return [];
+            }
+            const rows = await readParquetRows(dataViewToArrayBuffer(dv));
+            return parquetRowsToDFData(rows, envelope.layout);


Avoid JSON-parsing native parquet strings

When the envelope is parquet_buffer, this now sends every row-layout parquet frame through parseParquetRow. That is correct for pandas/fastparquet frames that JSON-encode object cells, but the Polars/lazy/Xorq infinite paths write native parquet strings without JSON-encoding (for example buckaroo/polars_buckaroo.py:105, buckaroo/server/data_loading.py:215, and buckaroo/xorq_buckaroo.py:311). For those backends, user string values such as null, true, 123, or {"a":1} will be JSON.parsed into null/boolean/number/object here, corrupting displayed data after this change; please only parse buffers known to use the pandas JSON-cell convention or carry an explicit encoding flag.

Useful? React with 👍 / 👎.

make_infinite_resp now returns (msg, parquet_bytes) instead of (msg, [parquet_bytes]) — one infinite_resp always carries exactly one dataframe, so the list wrapper was pure ceremony. A new send_infinite_resp helper wraps make_infinite_resp + widget.send for the anywidget/zmq comm path, so: - comm senders call send_infinite_resp(self, ...) instead of self.send(*make_infinite_resp(...)) - websocket/server senders return make_infinite_resp(...) directly instead of unpacking msg, buffers then returning (msg, buffers[0]) No behavior change; the two-frame websocket write and the buffers-list comm write produce identical wire output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…recorder - widget.tsx: route the anywidget transcript recorder's infinite_resp decode through srt.decodeDFData(msg.payload, buffers) instead of its own inline parquetRead/extractArrayBuffer path, so it honors the envelope's buffer_index/layout and JSON-parses object cells like the rest of the app. Drop the now-unused extractArrayBuffer helper; the handler's buffers param is DataView[] (what anywidget/WebSocketModel actually emit). - DFWhole.ts: remove the dead ParquetB64Payload interface — the DFEnvelope union's parquet_b64 member is the single source of truth, and nothing imports the alias after the transport unification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…on test JsonEnvelope.stories.tsx mounts BuckarooStaticTable with a json envelope as df_data, driving decodeDFData's json branch at the real component ingestion edge into AG-Grid. json-envelope.spec.ts loads the story and asserts the inline record-array rows render. Closes the json-envelope integration gap — the json branch previously had decoder unit coverage only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

#933 (unified DF transport) finalizes the consumer side: js-core's decodeDFData(msg.payload, buffers) decodes the inline parquet_b64 infinite_resp payload, so the single-JSON-message reply renders end-to-end with no binary side-channel frame. Align the producer to the landed contract: - Widen DFEnvelope.layout to 'wide' | 'row' to match js-core's final shape (DFWhole.ts:258-261); the plan doc had only 'wide'. - Drop the "BLOCKED ON #933 / do not ship" framing from transport.ts, wireTypes.ts, README, and the WS demo; cite #933 as the established decode contract instead. - Clarify the WS demo deliberately uses WebSocketModel's binary-frame path (self-contained over plain WebSocket), while production IPC uses parquet_b64 + decodeDFData. No behavior change (type widening is a superset); decode logic lives in js-core. Build clean, 44 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

paddymul temporarily deployed to testpypi June 23, 2026 17:57 — with GitHub Actions Inactive

chatgpt-codex-connector Bot reviewed Jun 23, 2026

View reviewed changes

paddymul mentioned this pull request Jun 23, 2026

docs: plan DuckDB/Node backend for buckaroo-js-core (#930) #935

Open

paddymul temporarily deployed to testpypi June 23, 2026 18:03 — with GitHub Actions Inactive

paddymul mentioned this pull request Jun 23, 2026

feat(duckdb-node): DuckDB backend for buckaroo-js-core (#930) #936

Merged

paddymul temporarily deployed to testpypi June 23, 2026 18:39 — with GitHub Actions Inactive

paddymul temporarily deployed to testpypi June 23, 2026 18:51 — with GitHub Actions Inactive

chatgpt-codex-connector Bot reviewed Jun 23, 2026

View reviewed changes

paddymul and others added 3 commits June 23, 2026 15:50

paddymul enabled auto-merge June 23, 2026 19:53

paddymul temporarily deployed to testpypi June 23, 2026 19:54 — with GitHub Actions Inactive

paddymul added this pull request to the merge queue Jun 23, 2026

Merged via the queue into main with commit 2479655 Jun 23, 2026
27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: plan unified dataframe transport layer - #933

docs: plan unified dataframe transport layer#933
paddymul merged 8 commits into
mainfrom
docs/unified-df-transport-plan

paddymul commented Jun 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 23, 2026

Uh oh!

paddymul Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026 •

edited

Loading

Uh oh!

paddymul commented Jun 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

paddymul commented Jun 23, 2026

Summary

Envelope

Also covered

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

paddymul Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 TestPyPI package published

MCP server for Claude Code

📖 Docs preview

🎨 Storybook preview

Uh oh!

paddymul commented Jun 23, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jun 23, 2026 •

edited

Loading