A collaborative repository gathering real-world data challenges, tricky migrations (like SAS to Python), and debugging exercises. This is a WIP and any help is welcome as we try to make it a useful resource for data people.
Whether you are losing rows in a Pandas merge, fighting with NaN values, or migrating legacy code, you'll find templates and interactive fixes here.
You don't need to install anything locally. Click the buttons below to open the interactive templates directly in Google Colab!
| Challenge / Exercise | Description | Open in Colab |
|---|---|---|
| SAS to Python: Missing Rows |
How to track down dropped rows and debug NaN handling using Set Differences and dataprof (beta). |
 |
| Profiling Messy CSVs |
Run profiling on diverse data types using dataprof (beta – API subject to change). |
|
| Stop Putting DBs in Columns | Example of why nesting complex structures in pandas columns is an anti-pattern. | |
| Data Leakage (Startup Case) | Exploring a complex data leakage issue similar to one experienced by a major startup. | |
| Leakage at Scale (Polars + DataFusion) | Advanced leakage detection using Polars LazyFrames, Arrow C Interface, and DataFusion SQL. | |
| Salting Skewed GroupBy (PySpark) | Fixing aggregation bottlenecks in Databricks/Spark using two-phase salted aggregation. | |
| Sensor Drift & Stuck-At Faults | Detecting silent IoT sensor failures with rolling z-scores and variance checks. | |
| Merge Pitfalls | Silent row explosion from duplicate keys, lost rows from wrong join type, and dtype mismatches. | |
| DateTime Hell | Timezone-naive vs aware mixing, ambiguous date formats, and off-by-one resampling bugs. | |
| PII Anonymization Leakage | Why hashing emails isn't enough when quasi-identifiers (ZIP, DOB, gender) allow re-identification. | |
| Spark Memory Spill & Window OOM | Unbounded window functions and forced SortMergeJoin vs proper partitioning and broadcast. | |
| SCD2 Idempotent Upsert | Non-idempotent SCD2 scripts that duplicate rows on re-run vs correct upsert with DuckDB. | |
| Late Arriving Events & Windowing | Processing-time vs event-time windowing and the watermark trade-off for streaming pipelines. | |
| Schema Evolution & Breaking Changes | Detecting breaking schema changes with PyArrow contracts before downstream pipelines fail. | |
| DAG Coupling & Retry Safety | Tightly coupled tasks and non-idempotent retries vs atomic checkpointed pipelines. | |
| Example | Description | Run |
|---|---|---|
| HashMap vs Linear Merge | When does O(n+m) actually beat O(n log n)? Benchmark of a real optimization that was correctly rejected. | cargo run --release --example merge_benchmark (README) |
If you prefer to run this locally, the repository is managed via standard Python tooling.
git clone https://github.com/AndreaBozzo/prac.git
cd prac
pip install .
# or use uv
uv sync