dbt-DQX Quality Integration

Simplified Data Quality checking at scale for PySpark workloads on streaming and standard DataFrames.

Project Overview

This project shows how to use dbt together with Databricks Labs DQX to make data pipelines more reliable.

We use DQX to run data quality checks.

After the checks, the clean data is loaded into Databricks tables.

These tables are then used to build dashboards, so the reports are based on trusted data.

Why this matters

✅ Data is checked before it’s used.

📊 Dashboards show accurate information.

⚡ Pipelines stay efficient and scalable.

🔍 Easy to observe and monitor data quality.

How it works

Build models with dbt.

Run DQX checks to catch problems.

Load the verified data into Databricks tables.

Use those tables in dashboards for insights.

Running DQX with DBT Projects

This project shows how to apply data quality checks from dbt projects. The DQX quality checking is executed using dbt Python models.

Prerequisites

To get started, install dbt and the required adapters:

Project Configuration

1. Open the profiles.yml file and update the following parameters:

   • http_path: Specify the Databricks SQL Warehouse ID (<warehouse_id>) to use for executing dbt SQL models.
   • default catalog and schema (optional, default: main.default).

2. The project uses a serverless cluster by default to execute dbt Python models (DQX quality checks). To change the default cluster, open the dbt_project.yml file and update the submission_method.

Execution

Provide authentication credentials to connect to the Databricks workspace by running the following commands in your console:

export DBT_ACCESS_TOKEN=<databricks_pat_token> export DBT_HOST=<databricks_workspace_url> export DBT_WAREHOUSE_ID=<databricks_sql_warehouse_id>

Navigate to /dqx_dbt and execute the following commands:

poetry run dbt run

📖 Documentation

The complete documentation is available at: [https://databrickslabs.github.io/dqx/]

DBT Resources

• Learn more about dbt in the docs
• Check out Discourse for commonly asked questions and answers
• Join the chat on Slack for live discussions and support
• Find dbt events near you
• Check out the blog for the latest news on dbt's development and best practices
• DBT Cheatsheet -> https://datacaffee.com/dbt-data-built-tool-commands-cheat-sheet/

👨‍💻 Contributing

Feel free to fork the repository, submit pull requests, or raise issues for improvements!

---

📧 Contact

For questions or support, reach out to Bindusekhar Gorintla at (gorintla.bindusekhar@gmail.com).