Automated, Metadata-Driven Data Ingestion Framework for Microsoft Fabric
Metadata Analyzer is a production-ready framework that automates end-to-end data ingestion in Microsoft Fabric—without writing pipelines for every table.
Traditional data pipelines are rigid:
- ❌ Add a new table → Edit the pipeline
- ❌ Schema changes → Silent failures
- ❌ No audit trail → Trust issues
- ❌ Manual for every table → Not scalable
Metadata-driven architecture:
- ✅ Add a new table → One SQL statement
- ✅ Schema changes → Auto-detected & logged
- ✅ Full audit trail → Every step tracked
- ✅ Scalable to 100s of tables → Zero pipeline edits
# You'll need:
- Azure Fabric workspace with F2 capacity
- Azure SQL Database (or any SQL Server)
- Azure CLI: az login
- Python 3.8+ (optional, for scripts)git clone https://github.com/prathameshsable/metadata-analyzer.git
cd metadata-analyzer
cp .env.example .env
# Edit .env with YOUR values# Create workspace & capacity (in Fabric UI or CLI)
az fabric workspace create --name "Analytics"
az fabric capacity assign --workspace-id <ID> --capacity-id fabricf2prod
# Create Lakehouse, Warehouse, and 3 Connections (must be UI)
# See SETUP-GUIDE.md for step-by-stepOption A: Fastest (Claude Code)
- Open https://claude.com/code
- Paste content from
MASTER-FRAMEWORK.md - Press Send
- Claude auto-executes all 11 steps ✅
Option B: Manual (More control)
- Follow steps 1-11 in
MASTER-FRAMEWORK.md - Execute manually in your environment
- Verify each step with queries
# Run all 10 test suites (see FRAMEWORK-TEST-CASES.md)
# Expected results:
# ✅ 48 tables discovered
# ✅ 5 tables activated
# ✅ 165,365+ rows loaded
# ✅ 15 DQ rules: 15 passed, 0 failed
# ✅ Success email delivered| Feature | Details |
|---|---|
| Source Discovery | Auto-discovers all tables & columns from source DB |
| Incremental Loading | Watermark-based delta loads (only new/changed data) |
| Schema Evolution | Auto-detects new columns, logs drift events |
| Data Quality | 8 rule types, 15+ rules evaluated per run |
| Audit Trail | 7 control tables track everything (what, when, who, how) |
| Email Notifications | Success/failure emails with full metrics |
| Zero Config Onboarding | Add new tables with one SQL UPDATE statement |
| Fault Tolerant | Graceful error handling, no data corruption |
meta.table_config -- 48 tables catalogued, activation rules
meta.column_config -- 487 columns with metadata
meta.type_mapping -- SQL ↔ Parquet type conversions
meta.drift_log -- Schema change events (auto-detected)
meta.load_audit -- Per-table load history & row counts
meta.run_audit -- Overall run statistics & summaries
meta.dq_rules -- Data quality rule evaluationsSales_Orders -- 38,172 rows
Sales_OrderLines -- 123,456 rows
Warehouse_StockItems -- 500 rows
Application_Cities -- 3,256 rows
Application_StateProvinces -- 181 rows
────────────────────────────────────────
TOTAL -- 165,365 rows loaded ✅
- pl_metadata_driven_ingestion — 11 activities, handles discovery → load → audit
- pl_ingestion_orchestrator — Invokes main pipeline, sends success/failure emails
- nb_merge_to_delta — Spark notebook for Delta MERGE + schema evolution
- nb_data_quality — Spark notebook for 8 DQ rule types
- vl_ingestion_config — 11 centralized variables (connections, paths, emails)
Want to show this on YouTube? Complete script provided:
- Segment 1: Intro (1 min)
- Segment 2: Manual setup (3-4 mins)
- Segment 3: Run framework (1 min)
- [Fade 75 mins] — Timer shows execution
- Segment 4: Results walkthrough (5-7 mins)
- Segment 5: Closing (2-3 mins)
Result: 15-20 minute video, high retention ✅
See YOUTUBE-RECORDING-SCRIPT.md for exact dialogue + timing.
All test cases included (10 suites, 24 tests):
# Run test suite
# Expected: 10/10 PASS
✅ TEST 1: Control Tables (4 tests)
✅ TEST 2: Data Load (2 tests)
✅ TEST 3: Schema Drift (2 tests)
✅ TEST 4: Data Quality (2 tests)
✅ TEST 5: Lakehouse Delta (2 tests)
✅ TEST 6: Pipeline Execution (3 tests)
✅ TEST 7: Email Notifications (2 tests)
✅ TEST 8: Watermark Tracking (2 tests)
✅ TEST 9: Variable Library (1 test)
✅ TEST 10: Edge Cases (3 tests)See FRAMEWORK-TEST-CASES.md for complete test suite with expected outputs.
| Document | Purpose |
|---|---|
MASTER-FRAMEWORK.md |
Core automation (11 steps, 14,000+ words) |
YOUTUBE-RECORDING-SCRIPT.md |
Video script + editing guide (7,000+ words) |
FRAMEWORK-TEST-CASES.md |
Test suites + validation (8,000+ words) |
SETUP-GUIDE.md |
Step-by-step onboarding |
docs/ARCHITECTURE.md |
Technical architecture deep-dive |
docs/TROUBLESHOOTING.md |
Common issues & solutions |
docs/FAQ.md |
Frequently asked questions |
✅ Zero credentials in repo
- All connection IDs are
<placeholders> - All emails are sanitized
- All tenant/workspace IDs removed
- All server names are
<your-*>
- Connection IDs (from Fabric UI)
- Your email address
- Your server/database names
- Your workspace/capacity names
Never commit .env file! (Already in .gitignore)
Instead of:
Table 1 → Custom Pipeline 1
Table 2 → Custom Pipeline 2
Table 3 → Custom Pipeline 3
...
You get:
48 Tables → 1 Generic Pipeline → Metadata Config → ✅ All loaded
-- First run: Load all data
SELECT * FROM Sales.Orders
-- Second run: Load only NEW/CHANGED data
SELECT * FROM Sales.Orders
WHERE LastEditedWhen > '2025-12-20 14:12:33' -- WatermarkSource: OrderID, CustomerID, OrderDate, Amount
Target: OrderID, CustomerID, OrderDate, Amount
After 1 week:
Source: OrderID, CustomerID, OrderDate, Amount, SpecialFlag ← NEW
Auto-detected ✅
Logged to meta.drift_log ✅
Delta table auto-evolved ✅
- Metadata-driven architecture (scalable data patterns)
- Microsoft Fabric (Lakehouse, Warehouse, Pipelines, Notebooks)
- Incremental data loading (watermarks, MERGE, delta logic)
- Schema drift detection (auto schema evolution)
- Data quality automation (8 rule types, 15+ rules)
- Email orchestration (success/failure notifications)
- Azure SQL + Spark (synapsesql connector, PySpark)
- Fabric REST APIs (programmatic automation)
After running the complete framework:
✅ 165,365 rows loaded from 5 source tables
✅ 0 schema drift events (stable schema)
✅ 15 data quality rules: 15 passed, 0 failed
✅ 5 Delta tables created in Lakehouse
✅ 7 audit tables tracking everything
✅ Success email with full metrics
✅ Watermarks set for incremental reload
✅ Zero manual edits to onboard new tables
Found a bug? Have an improvement? Please contribute!
- Fork the repository
- Create a feature branch:
git checkout -b feature/improvement - Commit changes:
git commit -m "Description" - Push:
git push origin feature/improvement - Open a Pull Request
See CONTRIBUTING.md for guidelines.
Stuck?
- Check
docs/FAQ.md - Review
docs/TROUBLESHOOTING.md - Open a GitHub Issue
- Comment on YouTube video
Questions about Fabric?
MIT License - You're free to use, modify, and distribute this code.
See LICENSE file for details.
Prathamesh Sable
- 🏢 Principal Consultant, Microsoft Fabric & Power BI
- 📚 Founder of TheBIMasters (free Fabric/Power BI education)
- 🏆 Microsoft Certified Trainer (MCT) since 2020
- 🎤 Speaker at Fabric community events
- ✏️ 18+ years in BI & Analytics
Connect:
- 🐙 GitHub: @prathameshsable
- 🎬 YouTube: @TheBIMasters
- 🔗 LinkedIn: Prathamesh Sable
- 🌐 Website: thebimasters.pro
Please star this repository! It helps others discover the project.
# Star this repo ⭐
# Follow on GitHub 🐙
# Subscribe on YouTube 🎬
# Share with your team 🤝- Add support for other data sources (Snowflake, BigQuery)
- Create Power BI dashboard for audit trail
- Add scheduling templates
- Create video tutorials
- Build CLI tool for setup automation
- Add Python SDK wrapper
- Support for CDC (Change Data Capture)
Ready to automate your data ingestion? Let's go! 🚀
Status: Production Ready
Last Updated: July 2026
License: MIT