feat(duckdb-node): histogram summary stat (numeric .01–.99 + categorical) - #939
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c1dc5dfedf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| categoricalHistogram, | ||
| buildHistogram, | ||
| type NumericHistogramArgs, | ||
| } from '../src/histogram'; |
There was a problem hiding this comment.
These new tests import ../src/histogram, but the commit does not add packages/buckaroo-duckdb-node/src/histogram.ts (nor the histogramSql.ts module imported by the companion test/spike), and the current src tree only contains the existing backend/config/query files. In any environment with the package dev dependencies installed, Vitest will fail module resolution before executing the suite, so CI for buckaroo-duckdb-node is blocked until the histogram implementation files and the related DuckSource.queryRows/pinned-row changes these tests expect are included.
Useful? React with 👍 / 👎.
…iling) Add the histogram-stat contract tests ahead of the implementation: - histogram.test.ts: pure math ported from the Python producer's fixtures (histogram_test.py / polars_categorical_histogram_test.py) — numpy round-half-to-even, SI-prefix bucket labels, numeric tail/meat bars, and the categorical top-N/longtail/unique/NA bars on the 0–100 scale. - histogramSql.test.ts: the quantile/bin SQL builders and row parsers. - backend/columnConfig: assert the injected `histogram` pinned row (after dtype, displayer 'histogram') and per-column bar lists. - spike.duckdb.test.ts: real-DuckDB numeric histogram reproduces numpy's 1st/99th-percentile normalized populations, plus a categorical column. These reference src/histogram.ts + src/histogramSql.ts (not yet added), so the suite is red until the next commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…olumns Compute the `histogram` bar list the viewer renders through the `displayer: 'histogram'` pinned row, matching buckaroo's pandas/polars producer (customizations/histogram.py) so a DuckDB column shows the same bars. - histogram.ts: faithful TS port of the Python math — numeric path clips the meat to the 1st/99th percentile (np.quantile == quantile_cont), 10 bins, low/high tail markers and an NA bucket; categorical path takes the top-7 categories plus longtail/unique/NA. All percentages 0–100, rounded with numpy round-half-to-even, SI-prefix bucket labels. - histogramSql.ts: per-column quantile + equal-width-bin SQL (DuckDB has no width_bucket, so floor((v-min)/width) clamped to the last bin reproduces np.histogram's edges) and the top-N value-counts query, plus the dispatcher that picks numeric vs categorical exactly like histogram.py:histogram. A failure on any single column yields an empty histogram, never a broken initial_state. - DuckSource gains a `queryRows` seam (implemented in the node-api adapter) for these small scalar aggregates; the no-coercion row path is untouched. - backend injects the `histogram` row right after dtype; columnConfig pins it with the histogram displayer (DefaultMainStyling.pinned_rows parity). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
c1dc5df to
4d8b2a1
Compare
📦 TestPyPI package publishedpip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.1.dev28068689297or with uv: uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.15.1.dev28068689297MCP server for Claude Codeclaude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.15.1.dev28068689297" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table📖 Docs preview🎨 Storybook preview |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87fa4bbbc7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| least(${lastBin}, greatest(0, floor((_meat.v - _mm.meat_min) / ((_mm.meat_max - _mm.meat_min) / ${MEAT_BINS}.0))))::INTEGER AS bin, | ||
| count(*) AS c | ||
| FROM _meat, _mm | ||
| WHERE _mm.meat_max > _mm.meat_min |
There was a problem hiding this comment.
For skewed numeric columns where the 1–99% meat is non-empty but all the same value (for example, many copies of one center value plus enough repeated tail values to make distinct_count > 5), this filter drops the only aggregate row because meat_max === meat_min. computeColumnHistogram then sees no numeric args and falls back to a categorical histogram, whereas the Python/NumPy producer still emits a numeric histogram for constant meat by expanding the bin range. Special-case the equal-min/max case instead of treating it as unavailable.
Useful? React with 👍 / 👎.
…stogram computeColumnHistogram reimplemented the numeric-vs-categorical dispatch already in buildHistogram, so the two copies could drift and the production numeric path was only exercised by the SKIP_DUCKDB-gated spike (never on CI). Make buildHistogram async with lazy fetchers (fetchNumericArgs/ fetchCategorical) so it keeps the short-circuit — the categorical query runs only when the numeric path doesn't win — and have computeColumnHistogram delegate to it with SQL-backed fetchers. The CI-run histogram.test.ts now drives the exact dispatcher that ships, and asserts the categorical fetch is skipped when numeric wins. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…aframe tests A DOUBLE column containing nan/±inf crashed initialState: DuckDB's SUMMARIZE runs STDDEV_SAMP, which throws "out of range" on any non-finite value and aborts the whole stats query (ddd_library.py:df_with_infinity, which pandas renders fine). Add RenamePlan.statsRelation — renamedRelation with non-finite floats nulled (CASE WHEN isfinite(col) ...) — and summarize over it. The rule: non-finite floats are missing for summary statistics. Only the stats path is guarded; the row (COPY) and histogram paths stay no-coercion. Real IEEE-754 types only; DECIMAL is fixed-point and can't be non-finite, so it's left untouched. Also add test/ddd.duckdb.test.ts porting the ddd pathologies that map to SQL columns (column named "index", ±inf/nan, HUGEINT > 2^63 lossiness, the ENUM/INTERVAL/TIME/DECIMAL/BLOB weird-types frame), plus CI-running unit tests for statsRelation (the ddd suite is SKIP_DUCKDB and doesn't gate CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Adds the
histogramsummary stat tobuckaroo-duckdb-node, the fast-follow #936 reserved (stats.ts: "Histograms/quantile displayers are a fast-follow"). A DuckDB column now renders the same histogram bars as the pandas/polars backends, through the existingdisplayer: 'histogram'pinned row.Stacked on #936. Base is
feat/930-duckdb-node-backendbecause the package only exists there. Once #936 merges, this retargets tomain(automatic when the base branch is deleted).What's here
histogram.ts— a faithful TS port of the Python producer (customizations/histogram.py):np.quantile≡ DuckDBquantile_cont), 10 bins, low/high tail markers, NA bucket.cat_pop), thenlongtail/unique/NA.histogramSql.ts— per-column quantile + equal-width-bin SQL (DuckDB has nowidth_bucket, sofloor((v-min)/width)clamped to the last bin reproducesnp.histogram's edges) and the top-N value-counts query, plus the numeric-vs-categorical dispatcher matchinghistogram.py:histogram. A failure on any single column yields an empty histogram, never a brokeninitial_state.DuckSource.queryRowsseam (implemented in the node-api adapter) for the small scalar aggregates; the no-coercion row path is untouched.histogramrow right afterdtype; columnConfig pins it with the histogram displayer (DefaultMainStyling.pinned_rowsparity).Validation
The real-DuckDB spike reproduces numpy's exact normalized meat populations for the Python
INT_ARRfixture (1st/99th-percentile clipping verified end to end) plus a categorical column. Note CI runsSKIP_DUCKDB=1, so the spike runs locally — full suite (77 tests incl. spike) green locally; the non-DuckDB tests run on CI.TDD
First commit pushes the contract tests red (they reference the not-yet-added modules); the second commit makes them green.
🤖 Generated with Claude Code