Skip to content

feat(samples): rebuild and modernise the tpch_sample #108

Merged
rederik76 merged 11 commits into
mainfrom
feature/tpch-sample-rebuild
Jul 6, 2026
Merged

feat(samples): rebuild and modernise the tpch_sample #108
rederik76 merged 11 commits into
mainfrom
feature/tpch-sample-rebuild

Conversation

@rederik76

@rederik76 rederik76 commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Rebuilds the long-unfinished tpch_sample bundle into the framework's most comprehensive, end-to-end medallion reference. The sample was started long ago but never completed; this revisits it and brings it up to feature-samples / pattern-samples standards, turning the samples.tpch dataset into a fully streaming medallion warehouse from multi-source ingestion through a governed gold star schema — plus a governed semantic layer, an AI/BI Genie space, and AI/BI Lakeview dashboards on top.

What's included

  • Multi-source, schema-on-read bronze — Parquet staging from 8 simulated source systems (one bronze schema per system), ingested via Auto Loader with schema inference + evolution.
  • Conformed, historized silver — SCD2 dimensions, SCD1 reference data, append-only facts, and data-quality expectations with quarantine (quarantineMode: table).
  • Governed gold star schema — xxhash64 surrogate keys, point-in-time (as-of) joins, dim_date fiscal calendar, row tracking for incremental MV refresh, and two metrics approaches side by side (pre-aggregated MVs + UC metric views).
  • Realistic data distribution (staging-side, volume-preserving) — optional reshaping on the initial load (tpch_data_realism.ipynb, gated by a toggle): temporal growth/seasonality, supplier Zipf skew, and category/market-segment/region relabels. Plus always-on economics + a ship-mode fulfillment SLA in initialize.ipynb (unit-price reduction so net sales land ~$800M; receipt_date = ship_date + transit(mode) so on-time rate varies by ship mode), applied consistently across setup and Runs 2–3.
  • AI/BI Genie space (optional) — natural-language analytics over the gold schema (deployed as a DABs resource; requires a SQL warehouse).
  • AI/BI Lakeview dashboards (optional) — a Commercial Overview and a Pipeline Health / data-quality view, deployed as first-class DABs resources with currency ($) and percent formatting.
  • Template specs — one bronze ingestion template and per-archetype silver templates (SCD2 / SCD1 / append) to collapse repetitive flows.
  • Three-run incremental simulation — full refresh, then incremental Runs 2–3 covering SCD2 changes, fact growth, and a backdated out-of-order correction.
  • "Real world complexity" — schema evolution (new loyalty_tier column auto-evolved in bronze), CDC deletes/tombstones via apply_as_deletes (cdc_operation flag; Run 3 supplier delete), and late-arriving dimensions with an unknown member (-1) on the MV dims + COALESCE in the fact.
  • Orchestration + docs — dedicated deploy_tpch_and_test.sh / deploy_tpch.sh / destroy_tpch.sh (with a --warehouse_id switch that gates the Genie space + dashboards), and a restructured doc set: a skimmable README.md (quickstart), DESIGN.md (background, architecture, design choices), and GUIDE.md (detailed deploy + demo walkthrough), cross-linked.

Removed

Legacy WIP scaffolding: old per-entity bronze specs, schema-on-read schema files, snapshot-fact specs/DML, stale resource layout, and pytest fixtures.

Notes

  • tpch_sample remains separate from the main deploy.sh / deploy_and_test.sh (feature + pattern samples), with its own orchestrator that reuses common.sh.
  • The Genie space and Lakeview dashboards are optional and only deploy when a --warehouse_id is supplied.

Test plan

  • deploy_tpch_and_test.sh — deploy + setup + Run 1/2/3 all TERMINATED SUCCESS (no pipeline/expectation errors)
  • Schema evolution — loyalty_tier present in bronze_crm.customer; only Run-2 demo rows populated, baseline NULL
  • Supplier delete — supplier 3 in silver.supplier shows a closed __END_AT (tombstone) and 0 open versions
  • Late-arriving dimension — part 9000001 in fct_order_lines has a -1 row (Run 2) and a real part_sk (Run 3); unknown member present in dim_part / dim_supplier / dim_location
  • DQ quarantine — rows captured in customer_address_quarantine and orders_quarantine across runs
  • Both metric views and all pre-aggregated MVs build in Run 1
  • Data realism — gold layer reflects temporal growth/seasonality, supplier/category/segment/region skew; net sales land ~$800M; on-time rate varies by ship mode
  • Genie space — deploys with --warehouse_id; sample NL questions return sensible results over the gold schema
  • Dashboards — Commercial Overview + Pipeline Health deploy with --warehouse_id; tiles render with $ / % formatting
  • Reviewer: spot-check the GUIDE.md demo flow / validation queries against a fresh deploy

rederik76 added 3 commits June 30, 2026 20:00
@rederik76
feat(samples): rebuild tpch_sample into a complete end-to-end medalli…
d22b6c8 
…on reference

The tpch_sample bundle was started long ago but never finished. This revisits and
completes it as the framework's most comprehensive sample, built on the samples.tpch
dataset and aligned with feature-samples / pattern-samples conventions.

Highlights:
- Multi-source, schema-on-read bronze: Parquet staging from 8 simulated source systems
  (one bronze schema per system) ingested via Auto Loader with schema inference + evolution.
- Conformed, historized silver: SCD2 dimensions, SCD1 reference data, append-only facts,
  and data-quality expectations with quarantine (quarantineMode: table).
- Governed gold star schema: xxhash64 surrogate keys, point-in-time (as-of) joins,
  dim_date fiscal calendar, row tracking for incremental MV refresh, and two metrics
  approaches side by side (pre-aggregated MVs + UC metric views).
- Template specs: one bronze ingestion template and per-archetype silver templates
  (scd2 / scd1 / append) to collapse repetitive flows.
- Three-run incremental simulation: full refresh then incremental Runs 2-3 covering SCD2
  changes, fact growth, and a backdated out-of-order correction.
- Tier-3 messiness: schema evolution (new loyalty_tier column auto-evolved in bronze),
  CDC deletes/tombstones via apply_as_deletes (cdc_operation flag; Run 3 supplier delete),
  and late-arriving dimensions with an unknown member (-1) on MV dims + COALESCE in the fact.
- Orchestration + docs: dedicated deploy_and_test_tpch.sh / deploy_tpch.sh / destroy_tpch.sh,
  a full README (design choices, setup, demo flow, validation queries), and a planning doc.

Removed the legacy WIP scaffolding (old per-entity bronze specs, schema-on-read schema
files, stale resources, pytest fixtures).

Validated end-to-end on e2-demo-field-eng (serverless, catalog main, _es): deploy + setup +
Run 1/2/3 all green, with data-level checks confirming schema evolution, supplier tombstone,
late-arriving part unknown-member resolution, and DQ quarantine capture.
@rederik76
chore(samples): rename deploy_and_test_tpch.sh to deploy_tpch_and_tes…
cb515e1 
…t.sh; bump version to v0.18.0

- Rename the tpch deploy+test orchestrator for naming consistency and update all
  references (script usage header + tpch_sample README).
- Bump VERSION to v0.18.0.
- Minor docs wording fix in what_is_lakeflow_framework.rst (SDP -> Spark Declarative Pipelines).
@rederik76
remove plan
d15f113
@rederik76 rederik76 self-assigned this Jun 30, 2026
@rederik76 rederik76 requested a review from haillew June 30, 2026 10:17
@rederik76 rederik76 linked an issue Jun 30, 2026 that may be closed by this pull request
[FEATURE]: Modernise the tpch_sample bundle into a comprehensive end-to-end medallion reference #107
Closed
1 task
rederik76 added 8 commits July 4, 2026 14:03
@rederik76
feat(samples): add optional AI/BI Genie space to tpch_sample
6b44f25 
Deploy a Genie space over the tpch gold schema for natural-language analytics,
wired as a post-pipeline notebook task in Run 1 (after create_metric_views). The
space registers the gold facts, dimensions, and both UC metric views, with worked
example SQL demonstrating both star-schema joins and MEASURE()-based metric-view
queries.

Genie deployment is optional and never blocks the sample:
- new optional `warehouse_id` bundle variable (default empty = skip)
- deploy scripts prompt for it interactively (TTY only); leaving it blank skips Genie
- create_genie_space notebook no-ops with a clear message when no warehouse is
  supplied, and is idempotent (find-or-update the space by title)
- destroy_tpch.sh trashes matching Genie spaces via the CLI during teardown

Validated end-to-end (deploy + setup + runs 1-3 incl. metric views and Genie) on
serverless with the quarantine double-qualification fix deployed.

README updated with the optional-warehouse prerequisite, Day-1 demo flow, cleanup
note, and design rationale.
@rederik76
feat(samples): add optional AI/BI Lakeview dashboards to tpch_sample
d88f35c 
Two dashboards as native DABs `dashboards` resources over the gold schema:
Commercial Overview and Pipeline Health & Governance. Datasets bind to
`dataset_catalog`/`dataset_schema` so they marry to the deployed catalog,
namespace and logical_env. Warehouse-gated and optional like the Genie
space; torn down via `bundle destroy`. Updates prompt/docs; bumps v0.19.0.
@rederik76
correct version file
dc12d13
@rederik76
feat(samples): add tpch_sample data realism and restructure its docs
e0a8607 
Data realism (staging-side, volume-preserving):
- add tpch_data_realism.ipynb: optional distribution reshaping on the initial
  load (temporal growth/seasonality, supplier Zipf skew, category/market-
  segment/region relabels), gated by the data_realism_enabled toggle
- add always-on economics + ship-mode SLA in initialize.ipynb (PRICE_SCALE
  unit-price reduction so net sales land ~$800M; receipt_date = ship_date +
  transit(mode)), applied across setup and run 2/3 so magnitude and on-time
  rate stay consistent across batches

Dashboards:
- format Lakeview counters/axes as currency ($) and percent on the Commercial
  Overview and Pipeline Health dashboards

Docs:
- split tpch_sample/README.md into a quickstart README + DESIGN.md
  (architecture & design choices) + GUIDE.md (detailed deploy & demo),
  cross-linked; document the realism component in DESIGN 3.21

Deploy scripts:
- clarify the --warehouse_id prompt (now covers Genie space + dashboards) in
  common.sh / deploy_tpch.sh / deploy_tpch_and_test.sh
@rederik76
resolve merge conflict
2d91c88
@rederik76
chore: bump VERSION to v0.19.0 on feature/tpch-sample-rebuild
c35d4b8
@rederik76
resolve merge conflict
2162712
@rederik76
Merge main into feature/tpch-sample-rebuild
4aa8662 
Bring in validate-samples CI fix: JFrog pip install + requirements.lock
(#114). Keep VERSION at v0.19.0 on the tpch feature branch.
@rederik76 rederik76 merged commit 3fd7580 into main Jul 6, 2026
5 checks passed
@rederik76 rederik76 deleted the feature/tpch-sample-rebuild branch July 6, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@haillew haillew haillew approved these changes

Assignees

@rederik76 rederik76

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Modernise the tpch_sample bundle into a comprehensive end-to-end medallion reference

2 participants

@rederik76 @haillew