CHANGELOG.md

Version changelog

0.16.0

• Added a pluggable actions and alerting subsystem (#1289). DQX now supports extensible actions that run when checked data violates an optional condition evaluated against the summary metrics produced by DQMetricsObserver. The built-in DQAlert action can send notifications to Slack, Microsoft Teams, a generic HTTPS webhook, or the log, so pipelines can react to data quality regressions without custom plumbing. You can create your own custom actions as well, and custom alerting is possible via the callback destination, which invokes an in-process Python callable for each alert.
• Added an MCP (Model Context Protocol) server for DQX (#1252). The server exposes DQX's data quality capabilities as tools that any MCP-compatible AI agent (Claude, Genie Code, Cursor, Mosaic AI) can discover and orchestrate. It runs as a Databricks App with on-behalf-of (OBO) authentication, so all data access is governed by the calling user's Unity Catalog permissions.
• Added support for summary metrics in Lakeflow Declarative Pipelines (LDP/DLT) (#1301). A new DQEngine.compute_summary_metrics(...) produces the same row counts, per-check breakdown, and custom observer metrics as a lazy aggregation over the results DataFrame, so metrics can be computed inside Spark Declarative Pipelines where the observer- and streaming-listener-based paths cannot be used.
• Added the aggr_matches_dataset dataset-level check (#1309). The check compares an aggregate metric (row count by default, or any curated/built-in aggregate) computed on the checked DataFrame against the same aggregate computed on a reference (upstream) table, enabling reconciliation-style validations against a source of truth.
• Added the has_no_gaps_per_time_window dataset-level check (#1370). It detects gaps in a time series — windows of a configurable size that contain no rows between windows that do — with optional grouping and trailing-gap handling.
• Added semantic validation of rulesets (#1203). A new ChecksSemanticValidator inspects declarative check metadata and reports duplicate rules (same function, arguments, criticality, and filter) and conflicting rules, surfacing authoring mistakes before checks run.
• Added the has_valid_string_case row-level check (#1347). It validates consistent string casing with upper, lower, title, and sentence modes, casting non-string columns to strings before comparison.
• Added the is_valid_national_id row-level check (#1346). It validates national identification numbers per country (default US), covering format, ranges, and obvious structural errors; it does not verify that a number was actually issued.
• Added the is_valid_currency_code row-level check (#1368). It validates values against ISO 4217 currency codes, supporting both the three-letter alphabetic (e.g. USD) and three-digit numeric (e.g. 840) representations via code_format.
• Added the is_valid_country_code row-level check (#1369). It validates values against ISO 3166-1 country codes in alpha-2 (default), alpha-3, or numeric form via code_format.
• Added the is_valid_language_code row-level check (#1403). It validates values against ISO 639 language codes in alpha-2 (ISO 639-1) or alpha-3 (ISO 639-3) form.
• Added the is_valid_subdivision_code row-level check (#1404). It validates values against ISO 3166-2 country subdivision codes (e.g. US-CA, GB-ENG), with optional cross-column country consistency via country_column.
• Added the is_valid_uuid row-level check (#1436). It validates values against the canonical RFC 9562 UUID string form (case-insensitive), mirroring the other pure pattern-match checks.
• Added a profiler profile builder for the has_no_outliers check (#1317). The profiler can now generate a has_no_outliers check, and the MAD-based calculations and profiler defaults were refactored into shared constants. Disabled by default to retain existing performance.
• Added Databricks runtime requirements verification (#1278). DQX now verifies the minimum Databricks environment version before executing checks (including geospatial checks with specific runtime requirements), while preserving backward compatibility.
• Migrated configs and rules to Pydantic v2 (#1259, #1302). DQX data models and the declarative checks path moved from dataclasses to Pydantic v2 for automatic validation and simpler YAML/JSON (de)serialization. Behavior is preserved: callers never see raw pydantic ValidationError — every entry point still raises DQX error types with the pre-migration message format.
• Persisted full check metadata across table storage backends (#1319). Checks saved to Delta or Lakebase now faithfully round-trip all logical fields (including message_expr and typed user_metadata), matching the File/Volume backends. See Breaking Changes for the user_metadata at-rest encoding change on the Delta backend.
• Guarded LLM-generated sql_query rules against unsafe SQL (#1275). Both LLM-assisted rule-generation paths now drop any generated sql_query rule whose query contains unsafe (DML/DDL) SQL before returning it to the caller.
• Guarded check filter and row_filter against unsafe SQL (#1303). All filter compile sites now route through a shared safe_filter_expr helper that rejects destructive SQL keywords, and a check with an unsafe filter is treated as skipped rather than failing the run.
• Decoupled AI-assisted rule generation from Spark (#1422). Rule generation no longer requires an active Spark session — column metadata is fetched via the Databricks SDK, and Spark sessions are created lazily only when required.
• Added a public parameter-order contract for built-in checks (#1408). A test now pins the intended public parameter order of every registered check in check_funcs.py, making any reordering that would break positional callers visible during review.
• DQX Studio: overhauled rule authoring with data-contract and AI generation, insights, and internationalization (#1262). The app can import ODCS v3.x data contracts and generate predefined, property-based, and natural-language rules via a new "from contract" UI, alongside richer authoring and insights.
• DQX Studio: added a French (fr) locale (#1330). French joins the existing English, Brazilian Portuguese, Italian, and Spanish translations.
• DQX Studio: simplified deployment to a natively declarative bundle (#1300). Legacy post-deploy grant scripts and one-time resource binding were removed in favor of a natively declarative Databricks Asset Bundle deploy.
• Deployed the MCP server with native DAB grants (#1430). The MCP server's Unity Catalog grants are now native bundle declarations and the post-deploy setup job was dropped, so make mcp-deploy is a single end-to-end command.
• Made telemetry best-effort so it can never fail a job (#1401). log_telemetry no longer makes a blocking per-check control-plane call that could raise a TimeoutError and terminate Structured Streaming jobs; telemetry is now non-throwing, deduplicated per process, and bounded by a short timeout.
• Stopped reconfiguring the root logger on import (#1359). Importing DQX no longer calls blueprint's install_logger(), which previously removed existing root handlers and overwrote the logging configuration of applications using DQX as a library.
• Fixed filtered is_unique violations (#1442). is_unique now requires the current row to match its filter before reporting a duplicate, so unfiltered rows sharing a key with filtered rows are no longer falsely flagged.
• Fixed aggregate checks over * combined with a row filter (#1453). Dataset-level aggregate checks that aggregate over * with a row_filter no longer raise INVALID_USAGE_OF_STAR_OR_REGEX when constructed with F.col("*"); unfiltered count/count_distinct over * continue to work, and unsupported star/aggregate combinations now raise a clear InvalidParameterError.
• Fixed row-level pattern checks accepting a trailing newline (#1454). is_valid_email, is_valid_ipv4_address, is_valid_uuid, is_valid_national_id, and the is_ipv4_address_in_cidr value path now reject values with a trailing newline, which Java/Spark rlike previously accepted because $ also matches before a final line terminator.
• Fixed null-safe result joins (#1356). Custom SQL and grouped-check results are now reattached with null-safe key matching, preserving violations for rows and groups with null key components.
• Fixed NULL group matching in time-window gap checks (#1387). has_no_gaps_per_time_window now preserves gap violations for groups with null key components.
• Fixed non-deterministic fingerprints for set-valued arguments (#1345). Set and frozenset rule arguments are canonicalized after normalization, so rule fingerprints are stable regardless of set iteration order.
• Fixed validation of columns requiring SQL identifier escaping (#1342). Bare column names that need escaping (e.g. Customer Name) now validate via a two-pass fallback.
• Preserved fractional seconds in timestamp profiles (#1399). The profiler now preserves microsecond precision when converting timestamp min/max aggregates to profile values for TimestampType and TimestampNTZType.
• DQX Studio: fixed the rule source value for rules imported through the app (#1318).
• DQX Studio: fixed installation on Windows (#1327). build_app.py now appends the .cmd suffix to Node binaries on Windows, and CI sweeps orphaned jobs.
• Reported MCP server integration test coverage (#1429). The MCP integration suite now runs on every pull request and installs DQX built from the repository (production keeps the published pin), so a change anywhere in the library is exercised end to end.
• Fixed the set_utc_timezone test fixture to actually apply UTC (#1402).
• Added documentation versioning tags (#1388). User-guide pages and subsections can now be tagged with a feature lifecycle stage and the version a feature was introduced.
• Added a Genie Space connection guide to the Quality Dashboard documentation (#1263).
• Simplified tolerance docstring examples in the check functions (#1254).
• Fixed data contract examples in the documentation (#1250).

BREAKING CHANGES!

• is_in_list, is_not_in_list, and is_not_null_and_is_in_list now resolve their allowed / forbidden string values as column expressions (consistent with the comparison checks), not string literals. A bare string is interpreted as a column reference, a numeric string (e.g. "3") is parsed as a number, and an ISO-date string (e.g. "2024-01-01") as a date. To match a string literal, single-quote the value (e.g. 'value') or wrap it in F.lit("value"). Existing checks that relied on bare strings being treated as literals must quote them. (#1419)
• user_metadata saved through the Delta table storage backend is now JSON-encoded at rest to preserve non-string types through the MAP<STRING, STRING> column. Save→load via DQX is transparent (you get the original typed value back), but the stored representation changes: direct SQL/dashboard consumers now read JSON-encoded values (decode with from_json), existing tables are not migrated, and legacy string values that look like JSON atoms ("true", "1", "null") read back as typed values (True / 1 / None) — re-save affected rule sets after upgrading to normalize. The File/Volume (YAML/JSON) and Lakebase (JSONB) backends are unaffected. (#1319)

0.15.0

• Added LLM-generated AI explanations for row-level anomaly detection (#1129). The has_no_row_anomalies check now attaches a plain-language ai_explanation to each flagged row under _dq_info[].anomaly, describing the likely cause, business impact, suggested action, the top contributing features, and the group's size and average severity. Explanations are generated vis Spark ai_query function against a Databricks Model Serving endpoint — no extra Python dependencies and no driver-side LLM calls — and anomalous rows are grouped by segment and top contributing features so the model is called once per group, keeping cost predictable on large datasets (bounded by max_groups). AI explanations are enabled by default and does not require additional settings. New parameters with good set of defaults include enable_ai_explanation, ai_explanation_llm_model_config, redact_columns (to keep sensitive columns out of the prompt and grouping), and max_groups. If the serving endpoint is unreachable, explanations are left null with a warning and scoring still succeeds. LLMModelConfig also gains max_tokens, temperature, timeout, and max_retries to bound LLM cost and latency and expose tuning parameters for the users if required.
• Added stratified sampling to the profiler (#1240). The profiler now accepts a sample_by option to perform stratified sampling based on column values. Users control the sampling fraction with either a single sample_fraction applied equally across all strata, or a dictionary mapping each stratum to its own fraction. When sample_by is omitted, the profiler continues to use uniform sampling across all rows.
• Added new row-level check function to validate email addresses called is_valid_email (#1158). A new is_valid_email check validates email addresses against a pragmatic, ReDoS-safe subset of RFC 5321/5322. Like the IP-address checks, it ignores null values (no violation reported).
• Added geofencing checks (#1164). Five new row-level geospatial checks validate topological relationships between a column geometry and a reference geometry: is_geo_contains, is_geo_covers, is_geo_intersects, is_geo_touches, and is_geo_within. By default they use exact, meter-level precision built on the ST_* family of functions; is_geo_covers and is_geo_intersects additionally support an approximate mode built on H3_* cell indexing with a configurable resolution for faster checks on large datasets. The reference geometry can be a literal WKT/WKB/EWKT/EWKB value or another column, with optional try_to_geometry conversion of either side. Running these checks requires Databricks serverless compute or runtime 17.1 or above.
• Added support for metrics-only writes (#1236). save_results_in_table and the corresponding workflow path can now persist summary metrics without requiring an output or quarantine table, supporting observability-focused pipelines that only need the metrics table. Batch observations are triggered before metrics are saved so the metrics table is populated correctly, and streaming and no-observer cases now raise explicit errors. Existing configurations with an output or quarantine table are unaffected.
• Allow custom check failure messages (#1092). DQRule now accepts an optional message_expr parameter that lets users define custom failure messages as a Spark Column or a SQL expression string. The same option is supported for checks defined declaratively in metadata (YAML/JSON), specified as a top-level message_expr key on the check definition alongside criticality and check. When omitted, the default message behavior is preserved; when provided, the custom message replaces the default message for failed rows.
• Added a Query Results Cookbook and aligned stored check names and fingerprints (#1193). A new reference page provides "copy-paste" SQL and PySpark recipes for querying DQX result tables (summary metrics, output, quarantine, and checks) to trace errors and warnings across runs, rows, and check definitions. To make the cookbook's fingerprint and name joins reliable, checks saved without an explicit name now store the same autogenerated name and name-inclusive rule_fingerprint that apply_checks writes to _errors/_warnings (named checks and for_each_column rules are byte-identical to before). Requesting summary metrics via metrics_config without a configured observer now fails fast with an InvalidParameterError instead of silently skipping the metrics table.
• Added in-app language switching to DQX Studio (#1172). DQX Studio now ships with four locales (English, Brazilian Portuguese, Italian, and Spanish), selectable from a new Preferences card on the user's Profile page. The choice is persisted per-browser via localStorage with no server-side or table changes, and the change is frontend-only. Non-English translations are AI-assisted and not yet reviewed by native speakers.
• DQX Studio: replaced the previous third-party build framework with first-party build and dev scripts (#1223). The app no longer depends on that external build package. scripts/build_app.py generates the FastAPI OpenAPI schema, runs orval, builds the frontend with Vite, and produces the application wheel (with a build-tagged local-version segment so successive deploys at the same commit always reinstall fresh code). scripts/dev.py runs uvicorn with reload alongside the Vite dev server, forwarding signals and tearing down both processes together. The bundle and warehouse-grant scripts were updated to support both bundle-managed and external (reuse) SQL warehouse modes. There is no runtime behavior change in the app itself.
• DQX Studio: added Lakebase storage backend to improve app latency with declarative storage and destroy protection (#1173). Schemas, the wheels volume, and the Lakebase instance and logical database are now declared in the bundle with prevent_destroy lifecycle protection, and make app-bind adopts pre-existing resources. OLTP tables (rules, settings, RBAC, comments, schedules) move to Postgres via a migration runner, while analytical tables (validation runs, profiling, quarantine, metrics) stay on Delta. Error, warning, and input row counts from the DQX observer are now persisted and surfaced in the UI, label badges and label filtering were added to rule selection and scheduling, and a Spark Connect Observation.get mutability bug that overwrote total row counts was fixed.
• Fixed quarantine-only writes when no output table is configured (#1183). apply_checks_and_save_in_table and apply_checks_by_metadata_and_save_in_table previously raised AttributeError when called with output_config=None and a quarantine_config. output_config is now optional and skipped when unset, so quarantine-only runs write just the invalid records; passing neither configuration raises a clear InvalidParameterError.
• Allow special characters in catalog and schema names (#1232). The validation regex for storage locations now accepts catalog and schema names that contain characters such as hyphens, which were previously rejected.
• Fixed installation when the anomaly-detection workflow is absent (#1194). Installation no longer fails when the Anomaly Trainer workflow is not present; its presence is now checked before it is appended to the workflow.
• Fixed data contract rule generation without the [llm] extra (#1191). DQLLMEngine was imported unconditionally in contract_rules_generator.py purely for a type annotation, causing an ImportError when the [llm] extra was not installed and producing a misleading "install datacontract-cli" error. The import is now guarded behind TYPE_CHECKING, so generate_rules_from_contract(..., process_text_rules=False) works without the [llm] extra.
• Added an installation wizard reference and promoted DQX Studio as the recommended no-code option (#1229).
• Added a data drift detection guide to the profiling documentation (#1205).
• Renamed user to client_id in the LakebaseChecksStorageConfig documentation to match the actual configuration field (#1201).

BREAKING CHANGES!

• Row-level anomaly detection rule now computes SHAP feature contributions by default — enable_contributions defaults to True (was False), adding scoring cost (requires the shap library, already included in the [anomaly] extra). Set enable_contributions=False to restore the previous behaviour. (#1129)
• Row-level anomaly detection now generates AI explanations by default — enable_ai_explanation defaults to True, so existing anomaly checks will make LLM calls against a Databricks Model Serving endpoint (default databricks-claude-sonnet-4-5) and incur cost. This requires Foundation Model APIs to be available in the workspace; if the endpoint is unreachable, explanations are skipped (null) with a warning rather than failing. Set enable_ai_explanation=False to opt out entirely. (#1129)
• The _dq_info[].anomaly output now contains an additional ai_explanation struct. Downstream consumers that assert on the exact anomaly struct schema should account for the new field. (#1129)
• Requesting summary metrics via metrics_config without a configured observer now raises InvalidParameterError instead of silently skipping the metrics table. (#1193)

0.14.0

• ML-based row-level anomaly detection (#990, #1055, #1062). DQX now offers ML-based row anomaly detection that automatically identifies unusual rows in data without requiring manually specified thresholds, enabling the detection of issues missed by rule-based checks. Users provide recent representative data, and DQX trains an Isolation Forest model that flags rows deviating from typical patterns at scoring time, with auto-discovery of relevant columns and segmentation where appropriate, plus per-row explanations of why a record was flagged. The feature integrates with MLflow for model registry, supports both training and scoring workflows, and complements existing rule-based and aggregate checks.
• DQX Studio app (Beta) — MVP release of DQX App (#1090) (#1040) (#1050) (#1034). DQX Studio is the no-code Databricks App for authoring and managing data quality rules through a browser-based UI. AI-assisted rule generation, in-app dry-run validation, scheduled rule execution with run history and per-check summary metrics, role-based access control (Admin, Approver, Author, Viewer, plus an orthogonal Runner role) backed by Databricks workspace groups, and a contextual AI assistant integrated into the UI. The bundle provisions all required resources automatically (app, SQL warehouse, task-runner job, schemas, volume) and exposes per-target variables for catalog, admin group, app name, warehouse name, and schema overrides. The app uses On-Behalf-Of (OBO) authentication so end users only see data they can access in Unity Catalog, and validates user-supplied checks with proper HTTP status codes (400 for malformed input). LLM configuration uses the calling user's OBO token on every request to ensure correct identity propagation in the deployed Apps environment.
• Added AI agent skills for DQX (#1125) (#1056). DQX now ships with Agent Skills under skills/ that teach AI assistants (Databricks Genie Code, Claude Code, Cursor, Copilot, and other tools following the open standard) how to use DQX correctly. The skills cover the public-API capabilities and are accompanied by an AGENTS.md canonical onboarding guide for AI coding agents, with a thin CLAUDE.md redirect for tools that look for it. A new docs guide documents installation and usage for each supported tool.
• Added has_no_aggr_outliers stateless rolling-window sigma outlier check (#1118). A new dataset-level quality check, has_no_aggr_outliers, has been introduced that detects outliers in time-series aggregates using a stateless rolling-window sigma method. The check is suitable for monitoring metrics such as daily transaction counts, hourly throughput, or any aggregate where deviations from a rolling baseline indicate quality issues, and complements the existing has_no_outliers MAD-based row-level check.
• Added are_polygons_mutually_disjoint geometry check function (#1061). A new geospatial check, are_polygons_mutually_disjoint, validates whether polygons in a column are mutually disjoint using ST_Intersects. The check supports row_filter, handles nulls and invalid geometries gracefully, and uses native Spark spatial intersections (rather than H3 indexing) for compatibility with Photon's spatial optimizations.
• Added null-safe support to foreign key check (#1106). The foreign_key check now accepts a null_safe parameter. By default, NULL values in the foreign key columns are ignored (SQL ANSI behavior). When null_safe=True, NULL foreign-key values are matched against NULL reference values. Note: enabling null_safe=True on a previously non-null-safe single-column FK changes the auto-generated rule name (a_not_exists_in_ref_b → struct_a_as_a_not_exists_in_ref_struct_b_as_a) and the violation message format.
• Added variable substitution support for check definitions (#1078). Check definitions now support {{ placeholder }} syntax for reusable templates, resolved at load time via a new variables parameter on load_checks() and load_checks_from_local_file(), or via default variables passed through ExtraParams at engine construction. The new resolve_variables() utility recursively replaces placeholders in all string fields of check definitions in a single pass and supports scalar types (str, int, float, bool, Decimal, datetime.date, datetime.datetime, datetime.time). Unresolved placeholders are logged as warnings.
• Added suppress_skipped option and skipped flag for skipped checks (#1063). A new suppress_skipped: bool = False option in ExtraParams allows checks skipped due to missing columns or invalid filters to produce no entry in _errors/_warnings and not cause rows to appear in the invalid DataFrame. Additionally, a new skipped boolean field has been added to dq_result_item_schema so skipped checks can be identified structurally without string-parsing the violation message.
• Added per-check-name breakdowns to summary metrics (#1097). The DQMetricsObserver now emits a new check_metrics row alongside the existing aggregates (input_row_count, error_row_count, warning_row_count, valid_row_count). The value is a JSON array of structs — one per check — with check_name, error_count, and warning_count, fitting the existing metric_name/metric_value schema without widening it. The change is backward compatible: existing metrics are unchanged and the new row is additive.
• Added versioning of checks with rule fingerprints (#1044). Checks now carry rule_fingerprint, rule_set_fingerprint, and created_at fields when saved to Delta or Lakebase storage, and rule_set_fingerprint is also stamped on summary metrics so every metric row can be traced back to the exact rule version that produced it. Each save creates a new versioned entry rather than overwriting prior history.
• Added partition and clustering support for output tables (#1012). The OutputConfig now accepts partition_by and cluster_by fields, allowing users to save DataFrames as partitioned or clustered tables. Liquid clustering is automatically applied the first time checks are saved to a liquid-clustered table, and the integration tests verify both partitioning and clustering behaviour end to end.
• Added configurable default criticality for profiler job (#1117). The profiler workflow now accepts a parameter to specify the default criticality (error or warn) for generated rules, allowing users to control rule severity at generation time rather than relying on a hardcoded default.
• Added schema validation rules generation from data contracts (#1043). The data contract rule generator now produces schema-validation rules from ODCS contracts (enabled via generate_schema_validation, defaulting to True), ensuring dataset schemas match contract definitions. A new InvalidPhysicalTypeError provides clearer error handling when physical types are missing or invalid in schema properties.
• Added end-to-end methods that load checks from storage (#1064). apply_checks_and_save_in_table and apply_checks_by_metadata_and_save_in_table now optionally load checks directly from a storage location (table or file), in addition to the existing option of using preloaded checks. Best-practice documentation has been updated with the recommended end-to-end patterns.
• Added solutions accelerators and industry demos (#1100). New industry-focused accelerators have been added under demos/dqx_demo_industry/: a Banking demo (dqx_banking_demo.py) focused on fraud detection and transaction monitoring, and a rebuilt Fashion demo (dqx_fashion_demo.py) with industry-specific custom check functions and 11 quality rules. The Manufacturing demo has been moved into the same subdirectory for consistency, and the demo documentation has been updated with a new "Industry Accelerators" section.
• Added intermediate demo for new users (#1041). A new intermediate demo has been added that can be presented in 5–10 minutes and showcases DQX's core functionality to someone seeing it for the first time.
• Added LLM-friendly documentation with llms.txt generation (#1029). The Docusaurus build now generates AI-accessible documentation in the standardized llms.txt format via the @signalwire/docusaurus-plugin-llms-txt plugin, with hierarchical organization so AI assistants and LLM-powered tools can consume DQX documentation more efficiently.
• Updated profiler implementation with rules-based profile builders (#1059). The DQX profiler has been refactored around a rules-based approach: profiles are now generated via registered profile builders, making it straightforward to add new profile types without modifying core profiler code.
• Improved summary stats report for string datatype columns (#1104). String columns no longer report min and max (lexicographic min/max is not meaningful for text data), and a count_distinct metric is now included for all column types in the profiler's summary stats output.
• Added py.typed marker for PEP 561 compliance (#1115). The package now ships with a py.typed marker file so downstream tools (mypy, pyright, etc.) recognise its existing type annotations instead of treating all databricks.labs.dqx imports as untyped.
• Support custom folder in CLI uninstall (#1066). The databricks labs uninstall dqx command now prompts for a custom workspace folder path (mirroring the install flow) and uses the new install_folder parameter on InstallationService.current() to locate installations outside the default /Users/<user>/.dqx location.
• Fixed AI-assisted sql_query generation and made has_valid_schema compatible with older Spark versions (#995). Resolved a subquery issue with sql_expression checks in Serverless v5 when the check name is auto-derived, made has_valid_schema compatible with Spark < 4, improved validation of required check function arguments, added agent guidelines, and added documentation on configuring DQX with Lakeflow Declarative Pipelines (LDP/DLT) for Materialized View incrementalization.
• Fixed has_valid_schema silently skipping validation for missing columns (#1042). A critical bug where has_valid_schema silently skipped validation for columns missing from the checked DataFrame has been fixed; missing columns are now reported as schema violations.
• Fixed save_results_in_table when DQObserver is present without metrics config (#1124). save_results_in_table now correctly handles the case where the calling DQEngine has an associated observer but no observation or metrics configuration is passed. The bundle has also been updated to use the direct deployment engine.
• Switched to WorkspaceClient for checking table existence in checks storage (#1035). TableChecksStorageHandler now uses WorkspaceClient to check for table existence when saving checks, replacing previous spark.catalog calls and improving compatibility across compute environments.
• Fixed Makefile coverage target to call the renamed test_coverage script (#1052).
• Migrated from hatch to uv package manager and hardened GitHub Actions workflows (#1098) (#1094) (#1096). The library has been migrated from hatch to uv for dependency and build management, GHA workflows have been refactored to increase infrastructure isolation and remove the Azure-login dependency, and performance benchmarks have been moved from per-PR runs to nightly. Dependency versions have been tightened, GitHub Actions are now pinned by SHA, and lock files have been cleaned up to remove registry-specific URLs and unused entries.
• Migrated test suite to native PySpark testing utilities (#1107). All tests now use pyspark.testing.utils.assertDataFrameEqual instead of chispa.assert_df_equality. The chispa test dependency has been removed, the centralized assert_df_equality_ignore_fingerprints wrapper has been updated to translate chispa-style kwargs (ignore_nullable, ignore_column_order, ignore_row_order) to PySpark equivalents, and chispa-specific transforms handling in the e2e PII notebook has been migrated to apply transforms before assertion.

BREAKING CHANGES!

• The default save mode for storing checks in Delta and Lakebase tables changed from overwrite to append. Rules are now versioned going forward — every save produces a new entry stamped with created_at, rule_set_fingerprint, and rule_fingerprint. To preserve the previous overwrite behaviour, explicitly pass mode="overwrite" when saving checks. (#1044)
• Parameter order changed in apply_checks_and_save_in_table and apply_checks_by_metadata_and_save_in_table. Update callers accordingly — see the methods' updated docstrings for the new signature. (#1064)

0.13.0

• New DQX Data Quality Dashboard (#1019). The data quality dashboard has been significantly enhanced to provide a centralized view of data quality metrics across all tables, allowing users to monitor and track data quality issues with greater ease. The dashboard now consists of three tabs - Data Quality Summary, Data Quality by Table (Time Series), and Data Quality by Table (Full Snapshot) - each catering to different monitoring scenarios, and offers customizable parameters for reporting column names and filtering tables with data quality issues. Additionally, the installation process for the dashboard has been simplified, with options to import it directly to a Workspace or deploy it automatically using the Databricks CLI.
• DQX App Skeleton (#982). The DQX application (frontend and backend) has been built with a core set of features, including configuration management and AI-assisted rule generation based on natural-language input from users. A comprehensive README documents the application architecture as well as development and deployment workflows. Future versions of DQX will introduce additional functionality (loading/saving rules, rules authoring in graphical form) and provide a streamlined, user-friendly way to deploy the application directly into a Databricks workspace.
• Added Decimal support to check functions and to min_max generator (#1013) (#1017). The data quality checks have been enhanced to support Python's Decimal type, in addition to int and float, for min/max validation checks, enabling proper data quality checks for decimal-precise financial and scientific data where floating-point precision issues would cause false positives.
• Added DQX production best practices and fix datetime limit handling (#997). Practical guidance and best practices for using DQX in production have been added, covering aspects such as storing checks in Delta tables, enforcing access controls, and optimizing rules for performance and scalability. Fixes have also been implemented to address issues related to handling date and datetime limits, particularly when provided as strings.
• Added new row-level check functions: is_null, is_empty, and is_null_or_empty (#1015). DQX now includes three new check functions, is_null, is_empty, and is_null_or_empty, which enable verification of column values as null, empty strings, or both, complementing existing checks like is_not_null, is_not_empty, and is_not_null_and_not_empty. The functions also support optional arguments, like trim_strings to trim spaces from strings.
• Added tolerance to equality and non-equality check functions (#1011). The library's quality check functionality has been enhanced to support absolute and relative tolerance parameters for numeric value comparisons in is_equal_to, is_not_equal_to, is_aggr_equal and is_aggr_not_equal checks, allowing for more flexible and precise control over data validation. The introduction of tolerance logic, which checks for absolute and relative differences within specified thresholds via abs_tolerance and rel_tolerance parameters, provides more nuanced comparisons for numeric data.
• Allow new lines in sql expression checks (#1009). SQL expression check function (sql_expression) has been updated to support new lines in its expression argument, allowing for more complex and formatted SQL expressions.
• Allow summary metrics with SparkConnect sessions (#1000). The library now supports writing summary metrics directly to a table with SparkConnect sessions, eliminating the need for a classic compute cluster in Dedicated access mode. This change lifts the previous restriction and enables generating summary metrics using Serverless and all standard clusters with Databricks Runtime 17.3LTS or higeher.
• Fixed loading checks from a delta table with special characters (#992). The loading checks functionality from a delta table has been fixed to handle special characters in the fully qualified table.
• Fixed resolution of pii detection check function (#1003). The PII detection check function resolution has been enhanced to support the application of checks defined as metadata (YAML).
• Fixed serialization/deserialization of row filter parameter for dataset-level rules (#1021). The filter field in checks definition now correctly pushes down the filter condition defined at the check-level as row_filter to the check function, allowing checks to operate on the relevant subset of rows before aggregation. The documentation has been updated to advice users to use top-level filter condition for consistency instead of row_filter parameter. Overall, these changes aim to enhance the overall user experience.
• Improved Lakeflow Declarative Pipeline tests (#1010). The Lakeflow Declarative Pipeline (LDP) tests have been enhanced to utilize full Unity Catalog mode, enabling support for writing to arbitrary catalogs and schemas, and performing additional checks to prevent certain operations.
• Updated Lakebase authentication method (#975). The Lakebase authentication method has been updated to utilize a client ID instead of a username, simplifying its use in the context of a Databricks App. The lakebase_user parameter has been replaced with lakebase_client_id, an optional service principal client ID used to connect to Lakebase, defaulting to the caller's identity if not provided. This change enhances the security and reliability of the authentication process, making it easier to work with Lakebase as a checks storage.
• Updated handling of metadata columns during schema validation (#1002). The has_valid_schema check has been enhanced to provide more flexibility in schema validation by introducing an optional exclude_columns parameter, allowing users to specify columns to ignore during validation. This parameter can be used to exclude metadata columns or other columns not relevant to schema validation, and it takes precedence over the columns list.
• Updated product info when missing in config while verifying workspace client (#987). The workspace client configuration has been enhanced to default product information to dqx with the current version when it is missing, ensuring that product information is always set for telemetry purposes.
• Updated profiler and generator documentation (#1026). The data profiling and quality checks generation feature has been enhanced with updated documentation, providing reference information for data quality profile types and associated rules.
• Added filter attribute in rules generated from ODCS (#978). The rules generation process has been enhanced with the introduction of a filter attribute in rules generated from Open Data Contract Standard (ODCS), allowing for more flexible and targeted rules creation.

0.12.0

• AI-Assisted rules generation from data profiles (#963). AI-assisted data quality rule generation was added, leveraging summary statistics from a profiler to create rules. The DQGenerator class includes a generate_dq_rules_ai_assisted method that can generate rules with or without user-provided input, using summary statistics to inform the rule creation process. This method offers flexibility in rule generation, allowing for both automated and user-guided creation of data quality rules.
• Added new checks for JSON validation (#616). DQX now includes three new quality checks for JSON data validation, especially useful for validating data coming from streaming systems such as Kafka: is_valid_json, has_json_keys, and has_valid_json_schema. The is_valid_json check verifies whether values in a specified column are valid JSON strings, while the has_json_keys check confirms the presence of specific keys in the outermost JSON object, allowing for optional parameters to require all keys to be present. The has_valid_json_schema check ensures that JSON strings conform to an expected schema, ignoring extra fields not defined in the schema.
• Added geometry row-level checks (#636). The library has been enhanced with new row-level checks for geometry columns, including checks for area and number of points, such as is_area_not_less_than, is_area_not_greater_than, is_area_equal_to, is_area_not_equal_to, is_num_points_not_less_than, is_num_points_not_greater_than, is_num_points_equal_to, and is_num_points_not_equal_to. These checks allow users to validate geometric data based on specific criteria, with options to specify the spatial reference system (SRID) and use geodesic area calculations. These changes enable more effective validation and quality control of geometric data, and are supported in Databricks serverless compute or runtime versions 17.1 and later.
• Added support to write using delta table path (#594). The quality check results saving functionality has been enhanced to support saving to Unity Catalog Volume paths, S3, ADLS, or GCS in addition to tables, providing more flexibility in storing and managing results. The save_results_in_table method now accepts output configurations with volume paths, and the OutputConfig object has been updated to support table names with 2 or 3-level namespace, storage paths including Volume paths, S3, ADLS, or GCS, and optional trigger settings for streaming output. Furthermore, the code now supports saving DataFrames to both Delta tables and storage paths, with the save_dataframe_as_table function taking an output_config object that determines whether to save the DataFrame to a table or a path. The functionality includes support for batch and streaming writes, input validation, and error handling, with the existing functionality of saving to Delta tables preserved and new functionality added for saving to storage paths.
• Extended aggregation check function to support more aggregation types (#951). The aggregation check function has been significantly enhanced to support a wide range of aggregate functions, including 20 curated statistical and percentile-based functions, as well as any Databricks built-in aggregate function, with runtime validation to ensure compatibility and trigger warnings for non-curated functions. The function now accepts an aggr_params parameter to pass parameters to aggregate functions, such as percentile calculations, and supports two-stage aggregation for window-incompatible aggregates like count_distinct. Additionally, the function includes improved error handling, human-readable violation messages, and performance benchmarks for various aggregation scenarios, enabling advanced data quality monitoring and validation capabilities for data engineers and analysts.
• Added new is_not_in_list check function (#969). A new check function, is_not_in_list, has been added to verify that values in a specified column are not present in a given list of forbidden values, allowing for null values and optional case-insensitive comparisons. This function is suitable for columns that are not of type MapType or StructType, and for optimal performance with large lists of forbidden values, it is recommended to use the foreign_key dataset-level check with the negate argument set to Trueumn to check, the list of forbidden values, and optionally the case sensitivity of the comparison, and its implementation includes input validation and custom error messages, with additional benchmark tests to measure its performance.
• Improve Generator to emit temporal checks for min/max date & datetime (#624). The data quality generator has been enhanced to support temporal checks for columns with datetime and date types, in addition to numeric types. The generator now creates rules with "is_in_range", "is_not_less_than", and is_not_greater_than functions based on the provided minimum and maximum limits, ensuring correct comparison by verifying that both limit values are of the same type. This update preserves the existing numeric behavior and introduces support for timestamp and date checks, while maintaining the ability to handle Python numeric types without stringification.
• Improved sql query check funciton to make merge columns parameter optional (#945). The sql_query check has been enhanced to support both row-level and dataset-level validation, allowing for more flexible data validation scenarios. In row-level validation, the check joins query results back to the input data to mark specific rows, whereas in dataset-level validation, the check result applies to all rows, making it suitable for aggregate validations with custom metrics. The merge_columns parameter is now optional, and when not provided, the check performs a dataset-level validation, providing a convenient way to validate entire datasets without requiring specific column mappings. Additionally, the check has been made more robust with input validation and error handling, ensuring that users can perform checks at both the row and dataset levels while preventing incorrect usage with informative error messages.
• Outlier detection numerical values (#944). The has_no_outliers function has been introduced to detect outliers in numeric columns using the Median Absolute Deviation (MAD) method, which calculates the lower and upper limits as median - 3.5 * MAD and median + 3.5 * MAD, respectively, and considers values outside these limits as outliers. The function is designed to work with numeric columns of type int, float, long, and decimal, and it raises an error if the specified column is not of numeric type. The addition of this function enables the detection of outlier numeric values, enhancing the overall data validation capabilities.
• Library improvements (#966). The library has undergone updates to improve its functionality, performance, and documentation. The has_json_keys function has been updated to treat NULL values as valid, ensuring consistent behavior across ANSI and non-ANSI modes. Additionally, the functionality of saving DataFrames as tables has been improved, with updated regular expression patterns for table names and enhanced handling of streaming and non-streaming DataFrames.
• Updated has_valid_schema check to accept a reference dataframe or table (#960). The has_valid_schema check has been enhanced to support validation against a reference dataframe or table, in addition to the existing expected schema. This allows users to verify the schema of their input dataframe against a reference dataframe or table by specifying either the ref_df_name or ref_table parameter, with exactly one of expected_schema, ref_df_name, or ref_table required. The check can be performed in strict mode for exact schema matching or in non-strict mode, which permits extra columns, and users can also specify particular columns to validate using the columns parameter. The function's update includes improved parameter validation, ensuring that only one valid schema source is specified, and new test cases have been added to cover various scenarios, including the use of reference tables and dataframes for schema validation, as well as parameter validation logic.
• Updated dashboards deployment to use standard lakeview dashboard definitions (#950). The dashboard installer has been updated to use standard Lakeview dashboard definitions.
• Added null island gemetry check function (#613). A new quality check called is_not_null_island has been introduced to verify whether values in a specified column are NULL island geometries, such as POINT(0 0), POINTZ(0 0 0), or POINTZM(0 0 0 0). The is_not_null_island function requires Databricks serverless compute or runtime version 17.1 or higher.
• Added float support for range and compare functions (#962). The comparison and validation functions have been enhanced to support float values, in addition to existing support for integers, dates, timestamps, and strings. This update allows for more flexible and nuanced comparisons and range checks, enabling precise and robust validation operations, particularly in scenarios involving decimal or fractional values. The functions now accept float values for limit parameters, and the range checks are inclusive of both boundaries. This enhancement enables users to specify minimum and maximum limits with decimal points, allowing for more precise data validation.

0.11.1

• Updated log level for spark connect to supress telemetry warnings in serverless.

0.11.0

• Generationg of DQX rules from ODCS Data Contracts (#932). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions.
• AI-Assisted Primary Key Detection and Uniqueness Rules Generation (#934). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The DQProfiler class now includes a detect_primary_keys_with_llm method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The DQGenerator class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an llm_primary_key_detection option, which allows users to control whether AI-assisted primary key detection is enabled or disabled.
• AI-Assisted Rules Generation Improvements (#925). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The generate_dq_rules_ai_assisted method now accepts an InputConfig object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the DQGenerator class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The InputConfig class provides a flexible way to configure the input data, including its location and format, and the get_column_metadata function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules.
• Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks (#673). The is_in_list and is_not_null_and_is_in_list check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional case_sensitive boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the is_not_null_and_is_in_list check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the foreign_key dataset-level check for large lists of allowed values or for columns of type MapType or StructType, as these checks are not suitable for such scenarios.
• Added documentation for using DQX in streaming scenarios with foreach batch (#948). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function.
• Added telemetry to track count of input tables (#954). Added additional telemetry for better trakcing of DQX usage to help improve the product.
• Added support for installing DQX from private PYPI repositories (#930). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked.
• Support Custom Folder Installation for CLI Commands (#942). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new --install-folder argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs.
• Enhancement to end to end tests (#921). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally.

BREAKING CHANGES!

• Renamed level parameter to criticality in generate_dq_rules method of DQGenerator for consistency.
• Replaced table: str parameter with input_config: InputConfig in profile_table method of DQProfiler for greater flexibility.
• Replaced table_name: str parameter with input_config: InputConfig in generate_dq_rules_ai_assisted method of DQGenerator for greater flexibility.

0.10.0

• Added Data Quality Summary Metrics (#553). The data quality engine has been enhanced with the ability to track and manage summary metrics for data quality validation, leveraging Spark's Observation feature. A new DQMetricsObserver class has been introduced to manage Spark observations and track summary metrics on datasets checked with the engine. The DQEngine class has been updated to optionally return the Spark observation associated with a given run, allowing users to access and save summary metrics. The engine now supports also writing summary metrics to a table using the metrics_config parameter, and a new save_summary_metrics method has been added to save data quality summary metrics to a table. Additionally, the engine has been updated to include a unique run_id field in the detailed per-row quality results, enabling cross-referencing with summary metrics. The changes also include updates to the configuration file to support the storage of summary metrics. Overall, these enhancements provide a more comprehensive and flexible data quality checking capability, allowing users to track and analyze data quality issues more effectively.
• LLM assisted rules generation (#577). This release introduces a significant enhancement to the data quality rules generation process with the integration of AI-assisted rules generation using large language models (LLMs). The DQGenerator class now includes a generate_dq_rules_ai_assisted method, which takes user input in natural language and optionally a schema from an input table to generate data quality rules. These rules are then validated for correctness. The AI-assisted rules generation feature supports both programmatic and no-code approaches. Additionally, the feature enables the use of different LLM models and gives the possibility to use custom check functions. The release also includes various updates to the documentation, configuration files, and testing framework to support the new AI-assisted rules generation feature, ensuring a more streamlined and efficient process for defining and applying data quality rules.
• Added Lakebase checks storage backend (#550). A Lakebase checks storage backend was added, allowing users to store and manage their data quality rules in a centralized lakabase table, in addition to the existing Delta table storage. The checks_location resolution has been updated to accommodate Lakebase, supporting both table and file storage, with flexible formatting options, including "catalog.schema.table" and "database.schema.table". The Lakebase checks storage backend is configurable through the LakebaseChecksStorageConfig class, which includes fields for instance name, user, location, port, run configuration name, and write mode. This update provides users with more flexibility in storing and loading quality checks, ensuring that checks are saved correctly regardless of the specified location format.
• Added runtime validation of sql expressions (#625). The data quality check functionality has been enhanced with runtime validation of SQL expressions, ensuring that specified fields can be resolved in the input DataFrame and that SQL expressions are valid before evaluation. If an SQL expression is invalid, the check evaluation is skipped and the results include a check failure with a descriptive message. Additionally, the configuration validation for Unity Catalog volume file paths has been improved to enforce a specific format, preventing invalid configurations and providing more informative error messages.
• Fixed docs (#598). The documentation build process has undergone significant improvements to enhance efficiency and maintainability.
• Improved Config Serialization (#676). Several updates have been made to improve the functionality, consistency, and maintainability of the codebase. The configuration loading functionality has been refactored to utilize the ConfigSerializer class, which handles the serialization and deserialization of workspace and run configurations.
• Restore use of hatch-fancy-pypi-readme to fix images in PyPi (#601). The image source path for the logo in the README has been modified to correctly display the logo image when rendered, particularly on PyPi.
• Skip check evaluation if columns or filter cannot be resolved in the input DataFrame (#609). DQX now skip check evaluation if columns or filters are incorrect allowing other checks to proceed even if one rule fails. The DQX engine validates specified column, columns and filter fields against the input DataFrame before applying checks, skipping evaluation and providing informative error messages if any fields are invalid.
• Updated user guide docs (#607). The documentation for quality checking and integration options has been updated to provide accurate and detailed information on supported types and approaches. Quality checking can be performed in-transit (pre-commit), validating data on the fly during processing, or at-rest, checking existing data stored in tables.
• Improved build process (#618). The hatch version has been updated to 1.15.0 to avoid compatibility issues with click version 8.3 and later, which introduced a bug affecting hatch. Additionally, the project's dependencies have been updated, including bumping the databricks-labs-pytester version from 0.7.2 to 0.7.4, and code refactoring has been done to use a single Lakebase instance for all integration tests, with retry logic added to handle cases where the workspace quota limit for the number of Lakebase instances is exceeded, enhancing the testing infrastructure and improving test reliability. Furthermore, documentation updates have been made to clarify the application of quality checks to data using DQX. These changes aim to improve the efficiency, reliability, and clarity of the project's testing and documentation infrastructure.

0.9.3

• Added support for running checks on multiple tables (#566). Added more flexibility and functionality in running data quality checks, allowing users to run checks on multiple tables in a single method call and as part of Workflows execution. Provided options to run checks for all configured run configs or for a specific run config, or for tables/views matching wildcard patterns. The CLI commands for running workflows have been updated to reflect and support these new functionalities. Additionally, new parameters have been added to configuration file to control the level of parallelism for these operations, such as profiler_max_parallelism and quality_checker_max_parallelism. A new demo has been added to showcases how to use the profiler and apply checks across multiple tables. The changes aim to improve scalability of DQX.
• Added New Row-level Checks: IPv6 Address Validation (#578). DQX now includes 2 new row-level checks: validation of IPv6 address (is_valid_ipv6_address check function), and validation if IPv6 address is within provided CIDR block (is_ipv6_address_in_cidr check function).
• Added New Dataset-level Check: Schema Validation check (#568). The has_valid_schema check function has been introduced to validate whether a DataFrame conforms to a specified schema, with results reported at the row level for consistency with other checks. This function can operate in non-strict mode, where it verifies the existence of expected columns with compatible types, or in strict mode, where it enforces an exact schema match, including column order and types. It accepts parameters such as the expected schema, which can be defined as a DDL string or a StructType object, and optional arguments to specify columns to validate and strict mode.
• Added New Row-level Checks: Spatial data validations (#581). Specialized data validation checks for geospatial data have been introduced, enabling verification of valid latitude and longitude values, various geometry and geography types, such as points, linestrings, polygons, multipoints, multilinestrings, and multipolygons, as well as checks for Open Geospatial Consortium (OGC) validity, non-empty geometries, and specific dimensions or coordinate ranges. These checks are implemented as check functions, including is_latitude, is_longitude, is_geometry, is_geography, is_point, is_linestring, is_polygon, is_multipoint, is_multilinestring, is_multipolygon, is_ogc_valid, is_non_empty_geometry, has_dimension, has_x_coordinate_between, and has_y_coordinate_between. The addition of these geospatial data validation checks enhances the overall data quality capabilities, allowing for more accurate and reliable geospatial data processing and analysis. Running these checks requires Databricks serverless or cluster with runtime 17.1 or above.
• Added absolute and relative tolerance to comparison of datasets (#574). The compare_datasets check has been enhanced with the introduction of absolute and relative tolerance parameters, enabling more flexible comparisons of decimal values. These tolerances can be applied to numeric columns.
• Added detailed telemetry (#561). Telemetry has been enhanced across multiple functionalities to provide better visibility into DQX usage, including which features and checks are used most frequently. This will help us focus development efforts on the areas that matter most to our users.
• Allow installation in a custom folder (#575). The installation process for the library has been enhanced to offer flexible installation options, allowing users to install the library in a custom workspace folder, in addition to the default user home directory or a global folder. When installing DQX as a workspace tool using the Databricks CLI, users are prompted to optionally specify a custom workspace path for the installation. Allowing custom installation folder makes it possible to use DQX on group assigned cluster.
• Profile subset dataframe (#589). The data profiling feature has been enhanced to allow users to profile and generate rules on a subset of the input data by introducing a filter option, which is a string SQL expression that can be used to filter the input data. This filter can be specified in the configuration file or when using the profiler, providing more flexibility in analyzing subsets of data. The profiler supports extensive configuration options to customize the profiling process, including sampling, limiting, and computing statistics on the sampled data. The new filter option enables users to generate more targeted and relevant rules, and it can be used to focus on particular segments of the data, such as rows that match certain conditions.
• Added custom exceptions (#582). The codebase now utilizes custom exceptions to handle various error scenarios, providing more specific and informative error messages compared to generic exceptions.

BREAKING CHANGES!

• Workflows run by default for all run configs from configuration file. Previously, the default behaviour was to run them for a specific run config only.
• The following depreciated methods are removed from the DQEngine: load_checks_from_local_file, load_checks_from_workspace_file, load_checks_from_table, load_checks_from_installation, save_checks_in_local_file, save_checks_in_workspace_file, save_checks_in_table,, save_checks_in_installation, load_run_config. For loading and saving checks, users are advised to use load_checks and save_checks of the DQEngine described here, which support various storage types.

0.9.2

• Added performance benchmarks (#548). Performance tests are run to ensure performance does not degrade by more than 25% by any change. Benchmark results are published in the documentation in the reference section. The benchmark covers all check functions, running all funcitons at once and applying the same funcitons at once for multiple columns using foreach column. A new performance GitHub workflow has been introduced to automate performance benchmarking, generating a new benchmark baseline, updating the existing baseline, and running performance tests to compare with the baseline.
• Declare readme in the project (#547). The project configuration has been updated to include README file in the released package so that it is visible in PyPi.
• Fixed deserializing to DataFrame to assign columns properly (#559). The deserialize_checks_to_dataframe function has been enhanced to correctly handle columns for sql_expression by removing the unnecessary check for DQDatasetRule instance and directly verifying if dq_rule_check.columns is not None.
• Fixed lsql dependency (#564). The lsql dependency has been updated to address a sqlglot dependency issue that arises when imported in artifacts repositories.

0.9.1

• Added quality checker and end to end workflows (#519). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations.
• Added built-in row-level check for PII detection (#486). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the does_not_contain_pii check function and can be customized to suit specific use cases. The check requires pii extras to be installed: pip install databricks-labs-dqx[pii]. Furthermore, a new enum class NLPEngineConfig has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data.
• Added equality row-level checks (#535). Two new row-level checks, is_equal_to and is_not_equal_to, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal.
• Added demo for Spark Structured Streaming (#518). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace.
• Added clarification to profiler summary statistics (#523). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields.
• Fixed rounding datetimes in the checks generator (#517). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day.
• Added API Docs (#520). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style.
• Improved test automation by adding end-to-end test for the asset bundles demo (#533).

BREAKING CHANGES!

• ExtraParams was moved from databricks.labs.dqx.rule module to databricks.labs.dqx.config

0.8.0

• Added new row-level freshness check (#495). A new data quality check function, is_data_fresh, has been introduced to identify stale data resulting from delayed pipelines, enabling early detection of upstream issues. This function assesses whether the values in a specified timestamp column are within a specified number of minutes from a base timestamp column. The function takes three parameters: the column to check, the maximum age in minutes before data is considered stale, and an optional base timestamp column, defaulting to the current timestamp if not provided.
• Added new dataset-level freshess check (#499). A new dataset-level check function, is_data_fresh_per_time_window, has been added to validate whether at least a specified minimum number of records arrive within every specified time window, ensuring data freshness. This function is customizable, allowing users to define the time window, minimum records per window, and lookback period.
• Improvements have been made to the performance of aggregation check functions, and the check message format has been updated for better readability.
• Created llm util function to get check functions details (#469). A new utility function has been introduced to provide definitions of all check functions, enabling the generation of prompts for Large Language Models (LLMs) to create check functions.
• Added equality safe row and column matching in compare datasets check (#473). The compare datasets check functionality has been enhanced to handle null values during row matching and column value comparisons, improving its robustness and flexibility. Two new optional parameters, null_safe_row_matching and null_safe_column_value_matching, have been introduced to control how null values are handled, both defaulting to True. These parameters allow for null-safe primary key matching and column value matching, ensuring accurate comparison results even when null values are present in the data. The check now excludes specific columns from value comparison using the exclude_columns parameter while still considering them for row matching.
• Fixed datetime rounding logic in profiler (#483). The datetime rounding logic has been improved in profiler to respect the round=False option, which was previously ignored. The code now handles the OverflowError that occurs when rounding up the maximum datetime value by capping the result and logging a warning.
• Added loading and saving checks from file in Unity Catalog Volume (#512). This change introduces support for storing quality checks in a Unity Catalog Volume, in addition to existing storage types such as tables, files, and workspace files. The storage location of quality checks has been unified into a single configuration field called checks_location, replacing the previous checks_file and checks_table fields, to simplify the configuration and remove ambiguity by ensuring only one storage location can be defined per run configuration. The checks_location field can point to a file in the local path, workspace, installation folder, or Unity Catalog Volume, providing users with more flexibility and clarity when managing their quality checks.
• Refactored methods for loading and saving checks (#487). The DQEngine class has undergone significant changes to improve modularity and maintainability, including the unification of methods for loading and saving checks under the load_checks and save_checks methods, which take a config parameter to determine the storage type, such as FileChecksStorageConfig, WorkspaceFileChecksStorageConfig, TableChecksStorageConfig, or InstallationChecksStorageConfig.
• Storing checks using dqx classes (#474). The data quality engine has been enhanced with methods to convert quality checks between DQRule objects and Python dictionaries, allowing for flexibility in check definition and usage. The serialize_checks method converts a list of DQRule instances into a dictionary representation, while the deserialize_checks method performs the reverse operation, converting a dictionary representation back into a list of DQRule instances. Additionally, the DQRule class now includes a to_dict method to convert a DQRule instance into a structured dictionary, providing a standardized representation of the rule's metadata. These changes enable users to work with checks in both formats, store and retrieve checks easily, and improve the overall management and storage of data quality checks. The conversion process supports local execution and handles non-complex column expressions, although complex PySpark expressions or Python functions may not be fully reconstructable when converting from class to metadata format.
• Added llm utility funciton to extract checks examples in yaml from docs (#506). This is achieved through a new Python script that extracts YAML examples from MDX documentation files and creates a combined YAML file with all the extracted examples. The script utilizes regular expressions to extract YAML code blocks from MDX content, validates each YAML block, and combines all valid blocks into a single list. The combined YAML file is then created in the LLM resources directory for use in language model processing.

BREAKING CHANGES!

• The checks_file and checks_table fields have been removed from the installation run configuration. They are now consolidated into the single checks_location field. This change simplifies the configuration and clearly defines where checks are stored.
• The load_run_config method has been moved to config_loader.RunConfigLoader, as it is not intended for direct use and falls outside the DQEngine core responsibilities.

DEPRECIATION CHANGES!

If you are loading or saving checks from a storage (file, workspace file, table, installation), you are affected. We are deprecating the below methods. We are keeping the methods in the DQEngine but you should update your code as these methods will be removed in future versions.

• Loading checks to storage has been unified under load_checks method. The following methods have been removed from the DQEngine: load_checks_from_local_file, load_checks_from_workspace_file, load_checks_from_installation, load_checks_from_table.
• Saving checks in storage has been unified under load_checks method. The following methods have been removed from the DQEngine: save_checks_in_local_file, save_checks_in_workspace_file, save_checks_in_installation, save_checks_in_table.

The save_checks and load_checks take config as a parameter, which determines the storage types used. The following storage configs are currently supported:

• FileChecksStorageConfig: file in the local filesystem (YAML or JSON)
• WorkspaceFileChecksStorageConfig: file in the workspace (YAML or JSON)
• TableChecksStorageConfig: a table
• InstallationChecksStorageConfig: storage defined in the installation context, using either the checks_table or checks_file field from the run configuration.

0.7.1

• Added type validation for apply checks method (#465). The library now enforces stricter type validation for data quality rules, ensuring all elements in the checks list are instances of DQRule. If invalid types are encountered, a TypeError is raised with a descriptive error message, suggesting alternative methods for passing checks as dictionaries. Additionally, input attribute validation has been enhanced to verify the criticality value, which must be either warn or "error", and raises a ValueError for invalid values.
• Databricks Asset Bundle (DAB) demo (#443). A new demo showcasing the usage of DQX with DAB has been added.
• Check to compare datasets (#463). A new dataset-level check, compare_datasets, has been introduced to compare two DataFrames at both row and column levels, providing detailed information about differences, including new or missing rows and column-level changes. This check compares only columns present in both DataFrames, excludes map type columns, and can be customized to exclude specific columns or perform a FULL OUTER JOIN to identify missing records. The compare_datasets check can be used with a reference DataFrame or table name, and its results include information about missing and extra rows, as well as a map of changed columns and their differences.
• Demo on how to use DQX with dbt projects (#460). New demo has been added to showcase on how to use DQX with dbt transformation framework.
• IP V4 address validation (#464). The library has been enhanced with new checks to validate IPv4 address. Two new row checks, is_valid_ipv4_address and is_ipv4_address_in_cidr, have been introduced to verify whether values in a specified column are valid IPv4 addresses and whether they fall within a given CIDR block, respectively.
• Improved loading checks from Delta table (#462). Loading checks from Delta tables have been improved to eliminate the need to escape string arguments, providing a more robust and user-friendly experience for working with quality checks defined in Delta tables.

0.7.0

• Added end-to-end quality checking methods (#364). The library now includes end-to-end quality checking methods, allowing users to read data from a table or view, apply checks, and write the results to a table. The DQEngine class has been updated to utilize InputConfig and OutputConfig objects to handle input and output configurations, providing more flexibility in the quality checking flow. The apply_checks_and_write_to_table and apply_checks_by_metadata_and_write_to_table methods have been introduced to support this functionality, applying checks using DQX classes and configuration, respectively. Additionally, the profiler configuration options have been reorganized into input_config and profiler_config sections, making it easier to understand and customize the profiling process. The changes aim to provide a more streamlined and efficient way to perform end-to-end quality checking and data validation, with improved configuration flexibility and readability.
• Added equality checks for aggregate values and negate option for Foreign Key (#387). The library now includes two new checks, is_aggr_equal and is_aggr_not_equal, which enable users to perform equality checks on aggregate values, such as count, sum, average, minimum, and maximum, allowing verification that an aggregation on a column or group of columns is equal to or not equal to a specified limit. These checks can be configured with a criticality level of either error or warn and can be applied to specific columns or groups of columns. Additionally, the foreign_key check has been updated with a negate option, allowing the condition to be negated so that the check fails when the foreign key values exist in the reference dataframe or table, rather than when they do not exist. This expanded functionality enhances the library's data quality checking capabilities, providing more flexibility and power in validating data integrity.
• Extend options for profiling multiple tables (#420). The profiler now supports wildcard patterns for profiling multiple tables, replacing the previous regex pattern support, and options can be passed as a list of dictionaries to apply different options to each table based on pattern matching. The profiler job is setup now with IO cache enabled cluste.
• Improved quick demo to showcase defining checks using DQX classes and renamed DLT into Lakeflow Pipeline in docs (#399). DLT has been renamed to Lakeflow Pipeline in documentation and docstrings, to maintain consistency in terminology. The quick demo has been enhanced to showcase defining checks using DQX classes, providing a more comprehensive approach to data quality validation. Additionally, performance information related to dataset-level checks has been added to the documentation, and instructions on how to use the Environment to install DQX in Lakeflow Pipelines have been provided.
• Populate columns in the results from kwargs of the check if provided (#416). Additionally, the sql_expression now supports optional columns argument that is propagated to the results.

0.6.0

• Added Demo from Data and AI Summit 2025 - DQX Demo for Manufacturing Industry (#391).
• Added Github Action to check if all commits are signed (#392). The library now includes a GitHub Action that automates the verification of signed commits in pull requests, enhancing the security and integrity of the codebase. This action checks each commit in a pull request to ensure it is signed using the git commit -S command, and if any unsigned commits are found, it posts a comment with instructions on how to properly sign commits.
• Added methods to profile multiple tables (#374). The data profiling feature has been significantly enhanced with the introduction of two new methods, profile_table and profile_tables, which enable direct profiling of Delta tables, allowing users to generate summary statistics and candidate data quality rules. These methods provide a convenient way to profile data stored in Delta tables, with profile_table generating a profile from a single Delta table and profile_tables generating profiles from multiple Delta tables using explicit table lists or regex patterns for inclusion and exclusion. The profiling process is highly customizable, supporting extensive configuration options such as sampling, outlier detection, null value handling, and string handling. The generated profiles can be used to create Delta Live Tables expectations for enforcing data quality rules, and the profiling results can be stored in a table or file as YAML or JSON for easy management and reuse.
• Created a quick start demo (#367). A new demo notebook has been introduced to provide a quickstart guide for utilizing the library, enabling users to easily test features using the Databricks Power Tools and Databricks Extension in VS Code. This demo notebook showcases both configuration styles side by side, applying the same rules for direct comparison, and includes a small, hardcoded sample dataset for quick experimentation, designed to be executed cell-by-cell in VS Code. The addition of this demo aims to help new users understand and compare both configuration approaches in a practical context, facilitating a smoother onboarding experience.
• Added Dataset-level checks, Foreign Key and SQL Script checks (#375). The data quality library has been enhanced with the introduction of dataset-level checks, which allow users to apply quality checks at the dataset level, in addition to existing row-level checks. Similar to row-level checks, the results of the dataset-level quality checks are reported for each individual row in the result columns. A new DQDatasetRule class has been added to define dataset-level checks, and several new check functions have been added including the foreign_key and sql_query dataset-level checks. The library now also supports custom dataset-level checks using arbitrary SQL queries and provides the ability to define checks on multiple DataFrames or Tables. The DQEngine class has been modified to optionally accept Spark session as a parameter in its constructor, allowing users to pass their own Spark session. Major internal refactorization has been carried out to improve code maintenance and structure.
• Pin GitHub URLs in docs to the latest released version (#390). The formatting process now includes an additional step to update GitHub URLs, ensuring they point to the latest released version instead of the main branch, which helps prevent access to unreleased changes. This update is automated during the release process and allows users to review changes before committing.

BREAKING CHANGES!

• Moved existing is_unique, is_aggr_not_greater_than and is_aggr_not_less_than checks under dataset-level checks umbrella. These checks must be defined using DQDatasetRule class and not DQRowRule anymore. Input parameters remain the same as before. This is a breaking change for checks defined using DQX classes. Yaml/Json definitions are not affected.
• DQRowRuleForEachCol has been renamed to DQForEachColRule to make it generic and handle both row and dataset level rules.
• Renamed column_names to result_column_names in the ExtraParams for clarity as they may be confused with column(s) specified for the rules itself. This is a breaking change!

0.5.0

• Fix spark remote version detection in CI (#342) #342
• Fix spark remote installation #346
• Load and save checks from a Delta table #339
• Handle nulls in uniqueness check for composite keys (#345)
• Allow user metadata for individual checks #352
• Add functionality to save results in delta table #319
• Fix checks older than #354
• Add PII-detection example #358
• Add aggregation type of checks #357

0.4.0

• Added input spark options and schema for reading from the storage (#312). This commit enhances the data quality framework used for profiling and validating data in a Databricks workspace with new options and functionality for reading data from storage. It allows for the usage of input spark options and schema, and supports fully qualified Unity Catalog or Hive Metastore table names in the format of catalog.schema.table or schema.table. Additionally, the code now includes a new dataclass field, input_schema, and a new dictionary field, input_read_options, to the RunConfig class. The documentation is updated with examples of how to use the new functionality.
• Added an example of uniqueness check for composite key (#312). Additionally, the code now includes a new dataclass field, input_schema, and a new dictionary field, input_read_options, to the RunConfig class. The documentation is updated with examples of how to use the new functionality.
• Renamed row checks module for more clarity (#314). This change renames the col_check_functions module to row_checks for clarity and to distinguish it from other types of checks. The import * syntax is removed and unused imports are removed from the demo. This change requires updating import statements that reference col_check_functions to use the new name row_checks. Checks defined using DQX classes require a simple update.

0.3.1

• Removed usage of lambda in quality checking (#310). We have replaced the usage of lambda functions n the quality checking with a more efficient implementation, and updated the method to handle optional arguments in validation. These changes improve the performance of the quality checking.

0.3.0

• Added sampling to the profiler (#303). The profiler's performance has been significantly improved in this release through the addition of sampling and limiting the input data. The profiler now samples input data with a 30% sampling factor and limits the number of records to 1000 by default, reducing the amount of data processed and enhancing performance. These changes are configurable and can be customized. This resolves issue #215.
• Added support for complex column types like struct, map and array. The support is added by extending the col_name to accept expressions (#214). Comprehensive examples have been included in the demo and documentation on how to apply checks on complex types.
• Fixed profiler bug when trying to cast a decimal string to int (#211). This modification resolves issue #172 and ensures proper handling of decimal strings during the casting process. This enhancement improves the profiler's robustness and flexibility when processing different data types, specifically integers and decimals.
• Renamed DQRule to DQColRule, and DQRuleColSet to DQColSetRule (#300). In this release, the class names DQRule and DQRuleColSet have been renamed to DQRuleCol and DQColSetRule, respectively, to support the addition of more rule types in the future, such as DQDatasetRule. The renaming includes corresponding changes in imports and method calls throughout the codebase. A deprecation warning has been added to the old classes. In addition, the col_functions module has been renamed to col_check_functions. This introduces a breaking change!. It is recommended to to update any references to the old class names in your code to ensure a smooth transition.
• Trim autogenerated check name to 255 chars (#301). This change ensures that potential issues arising from long check names are avoided by truncating the auto-generated check name to a reasonable lenght.
• Updated sql expression logic (#212). In this release, the SQL expression logic in our data quality library has been updated to cause the sql_expression check to fail if the condition is not met, introducing a potential breaking change.
• Added context info to output (#206). Additional context information is now added to the results of quality checks, including name, message, column name, filter, function, runtime, and user-provided metadata for every failed check. This allows users to provide custom metadata that is stored in the reporting columns for failed checks. This change is a breaking change for checks defined using classes! It is advised to consult the latest documentation for the updated syntax of defining checks using DQX classes.

0.2.0

• Added uniqueness check(#200). A uniqueness check has been added, which reports an issue for each row containing a duplicate value in a specified column. This resolves issue 154.
• Added column expression support for limits in not less and not greater than checks, and updated docs (#200). This commit introduces several changes to simplify and enhance data quality checking in PySpark workloads for both streaming and batch data. The naming conventions of rule functions have been unified, and the is_not_less_than and is_not_greater_than functions now accept column names or expressions as limits. The input parameters for range checks have been unified, and the logic of is_not_in_range has been updated to be inclusive of the boundaries. The project's documentation has been improved, with the addition of comprehensive examples, and the contribution guidelines have been clarified. This change includes a breaking change for some of the checks. Users are advised to review and test the changes before implementation to ensure compatibility and avoid any disruptions. Reslves issues: 131, 197, 175, 205
• Include predefined check functions by default when applying custom checks by metadata (#203). The data quality engine has been updated to include predefined check functions by default when applying custom checks using metadata in the form of YAML or JSON. This change simplifies the process of defining custom checks, as users no longer need to manually import predefined functions, which were previously required and could be cumbersome. The default behavior now is to import all predefined checks. The validate_checks method has been updated to accept a dictionary of custom check functions instead of global variables. This improvement resolves issue #48.

0.1.13

• Fixed cli installation and demo (#177). In this release, changes have been made to adjust the dashboard name, ensuring compliance with new API naming rules. The dashboard name now only contains alphanumeric characters, hyphens, or underscores, and the reference section has been split for clarity. In addition, demo for the tool has been updated to work regardless if a path or UC table is provided in the config. Furthermore, documentation has been refactored and udpated to improve clarity. The following issue have been closed: #171 and #198.
• [Feature] Update is_(not)_in_range (#87) to support max/min limits from col (#153). In this release, the is_in_range and is_not_in_range quality rule functions have been updated to support a column as the minimum or maximum limit, in addition to a literal value. This change is accomplished through the introduction of optional min_limit_col_expr and max_limit_col_expr arguments, allowing users to specify a column expression as the minimum or maximum limit. Extensive testing, including unit tests and integration tests, has been conducted to ensure the correct behavior of the new functionality. These enhancements offer increased flexibility when defining quality rules, catering to a broader range of use cases and scenarios.

0.1.12

• Fixed installation process for Serverless (#150). This commit removes the pyspark dependency from the library to avoid spark version conflicts in Serverless and future DBR versions. CLI has been updated to install pyspark for local command execution.
• Updated demos and documentation (#169). In this release, the quality checks in the demos have been updated to better showcase the capabilities of DQX. Documentation has been updated in various places for increased clarity. Additional contributing guides have been added.

0.1.11

• Provided option to customize reporting column names (#127). In this release, the DQEngine library has been enhanced to allow for customizable reporting column names. A new constructor has been added to DQEngine, which accepts an optional ExtraParams object for extra configurations. A new Enum class, DefaultColumnNames, has been added to represent the columns used for error and warning reporting. New tests have been added to verify the application of checks with custom column naming. These changes aim to improve the customizability, flexibility, and user experience of DQEngine by providing more control over the reporting columns and resolving issue #46.
• Fixed parsing error when loading checks from a file (#165). In this release, we have addressed a parsing error that occurred when loading checks (data quality rules) from a file, fixing issue #162. The specific issue being resolved is a SQL expression parsing error. The changes include refactoring tests to eliminate code duplication and improve maintainability, as well as updating method and variable names to use filepath instead of "path". Additionally, new unit and integration tests have been added and manually tested to ensure the correct functionality of the updated code.
• Removed usage of try_cast spark function from the checks to make sure DQX can be run on more runtimes (#163). In this release, we have refactored the code to remove the usage of the try_cast Spark function and replace it with cast and isNull checks to improve code compatibility, particularly for runtimes where try_cast is not available. The affected functionality includes null and empty column checks, checking if a column value is in a list, and checking if a column value is a valid date or timestamp. We have added unit and integration tests to ensure functionality is working as intended.
• Added filter to rules so that you can make conditional checks (#141). The filter serves as a condition that data must meet to be evaluated by the check function. The filters restrict the evaluation of checks to only apply to rows that meet the specified conditions. This feature enhances the flexibility and customizability of data quality checks in the DQEngine.

0.1.10

• Support datetime arguments for column range functions (#142) View
• DQX engine refactor and docs update (#138) View
• Add column functions to check for valid date strings (#144) View
• Generate rules for DLT as Python dictionary (#148) View
• Make DQX compatible with Serverless (#147) View

0.1.9

• New dashboard query, Update to demos and docs #133
• Patch user agent to enable tracking #121
• Added docs build on push #129

0.1.8

• Add Dashboard as Code, DQX Data Quality Summmary Dashboard #86
• Updated profiling documentation with cost consideration #126
• Improve docs styling #118
• Added search for docs #119
• Updated docs #117

0.1.7

• Bug fixed profiler
• Released docs at https://databrickslabs.github.io/dqx/
• Updated README

0.1.6

• Added new check: is_not_null_and_not_empty_array
• Fixed links to image for pypi
• Minor documentation updates

0.1.5

• Updated release process
• Updated README

0.1.4

• Updated release process

0.1.1

• Bug fixed cli installation
• Fixed Github release process
• Updated demos

0.1.0

Initial release of the project

0.0.0

Initial dqx commit