Airflow Data Quality Provider part 1

[gopidesupavan]

Adds a new apache-airflow-providers-common-dataquality provider for DbApiHook-based data quality checks.

Airflow already has SQL check operators, and many users rely on them for data quality today. This provider adds a DQRule / RuleSet layer for checks that need stable rule identity, persisted history, and a connection to Airflow assets. That makes quality results easier to analyze over time, lets downstream asset consumers gate on recent quality, and gives LLM-assisted workflows one schema to generate when proposing checks from table context. Execution still goes through existing common.sql / DbApiHook connections.

This PR is the backend/provider slice only. The UI plugin and read-only API are intentionally left for a follow-up PR.

Ships:

• DQRule and RuleSet models for named data quality rules.
• Built-in SQL checks for common table and column checks, executed through common.sql / DbApiHook.
• custom_sql support for database-specific or more complex checks.
• DQCheckOperator and the @task.dq_check TaskFlow decorator.
• A configurable results backend under [dq] results_path for task, run, and rule-level history.
• Experimental asset helpers, asset_quality() and require_quality(), that attach provider-owned quality metadata to assets without changing Airflow core.
• Documentation and example Dags covering end-to-end usage with and without LLM-generated rules.
• A DQ rule-authoring skill that LLM-assisted workflows can use to generate rules from table/schema context:
  https://github.com/gopidesupavan/airflow/blob/f32940bd261b94238256eaced9150dd51329ce3e/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md

This first version is intentionally focused on the backend contract: deterministic rule definitions, SQL execution through existing Airflow SQL providers, persisted results, and asset-linked quality summaries.

Design decisions:

• Results are stored through an object-storage/local-file backend instead of adding new metadata DB tables in the first provider drop. This keeps the provider self-contained, avoids Airflow core migrations, and lets deployments choose a durable store such as S3, GCS, or local files via [dq] results_path.
• The backend stores keyed JSON records for task runs, task instances, and per-rule history so later readers, including a future UI/API layer, can access common views without scanning unrelated runs.
• Asset support is implemented with provider-owned metadata, not Airflow core changes. Static quality configuration is attached to Asset.extra["airflow.dq"]; runtime summaries are attached to asset events under extra["airflow.dq.result"].
• The first release starts with DbApiHook / SQL execution because Airflow already has broad database coverage through common.sql. File and object-store data checks are left for a later iteration.

Later iterations:

• Read-only API and minimal Airflow UI plugin for viewing task/run results and rule history.
• File/object-store based checks, where Airflow reads data from S3/GCS/local files or other object stores and runs quality rules directly against that data.
• OpenLineage integration for data quality facets.
• More built-in checks.
• Trigger support for DQ checks.
• DQProfileOperator.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  codex

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[gopidesupavan]

This is part 1 , the UI plugin is part of this big PR #69413, i will ship that separate. but to show how the UI looks like here is some screenshots #69413 (comment) (this is minimal what had done with help of UI not an UI expert anyone can modify or help with this 😄 )

[Copilot]

Pull request overview

This PR introduces a new provider package, apache-airflow-providers-common-dataquality, adding a declarative data-quality layer (RuleSet/DQRule) executed via common.sql/DbApiHook, with persisted JSON results and optional asset-based quality gating. It also wires the new provider into the monorepo (extras, uv workspace, CI/dev tooling) and includes initial documentation and tests.

Changes:

• Add core data-quality primitives (rules, built-in check catalog, SQL engine) plus result records and an object-storage/local-files results backend.
• Add runtime entrypoints: DQCheckOperator, @task.dq_check, and asset helpers (asset_quality(), require_quality()), with provider docs/examples.
• Integrate the provider into packaging/extras, CI docker-compose sources, Breeze constants, and spelling wordlist.

Reviewed changes

Copilot reviewed 90 out of 101 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
uv.lock	Adds the new provider workspace member and extras wiring to the lockfile.
scripts/ci/prek/generate_common_dataquality_ruleset_schema.py	Prek script to regenerate the RuleSet JSON schema snapshot via uv run.
scripts/ci/docker-compose/tests-sources.yml	Mounts the new provider’s tests into CI docker-compose.
scripts/ci/docker-compose/remove-sources.yml	Adds the new provider source directory to the “remove sources” mapping.
pyproject.toml	Adds common.dataquality extra, mypy paths, and workspace member for the provider.
providers/common/dataquality/tests/conftest.py	Enables shared pytest plugin for provider tests.
providers/common/dataquality/tests/system/__init__.py	System tests package init.
providers/common/dataquality/tests/system/common/__init__.py	System tests namespace init.
providers/common/dataquality/tests/system/common/dataquality/__init__.py	System tests package init for dataquality.
providers/common/dataquality/tests/system/common/dataquality/example_dq_check.py	System test/example Dag for the operator and persistence path configuration.
providers/common/dataquality/tests/unit/__init__.py	Unit tests namespace init.
providers/common/dataquality/tests/unit/common/__init__.py	Unit tests namespace init.
providers/common/dataquality/tests/unit/common/dataquality/__init__.py	Unit tests package init for dataquality.
providers/common/dataquality/tests/unit/common/dataquality/test_results.py	Unit tests for score computation, summaries, and result serialization.
providers/common/dataquality/tests/unit/common/dataquality/test_assets.py	Unit tests for asset config and quality gating decision logic.
providers/common/dataquality/tests/unit/common/dataquality/backends/__init__.py	Unit tests package init for backends.
providers/common/dataquality/tests/unit/common/dataquality/decorators/__init__.py	Unit tests package init for decorators.
providers/common/dataquality/tests/unit/common/dataquality/decorators/test_dq_check.py	Unit tests for @task.dq_check runtime ruleset behavior.
providers/common/dataquality/tests/unit/common/dataquality/engines/__init__.py	Unit tests package init for engines.
providers/common/dataquality/tests/unit/common/dataquality/engines/test_sql.py	Unit tests for SQL engine query building and measurement behavior.
providers/common/dataquality/tests/unit/common/dataquality/operators/__init__.py	Unit tests package init for operators.
providers/common/dataquality/tests/unit/common/dataquality/operators/test_dq_check.py	Unit tests for DQCheckOperator execution paths and persistence interactions.
providers/common/dataquality/tests/unit/common/dataquality/rules/__init__.py	Unit tests package init for rules.
providers/common/dataquality/tests/unit/common/dataquality/rules/test_checks.py	Unit tests validating the built-in check catalog invariants.
providers/common/dataquality/src/airflow/__init__.py	Provider package namespace init.
providers/common/dataquality/src/airflow/providers/__init__.py	Provider namespace init.
providers/common/dataquality/src/airflow/providers/common/__init__.py	Common provider namespace init.
providers/common/dataquality/src/airflow/providers/common/dataquality/__init__.py	Provider version gate and package metadata init.
providers/common/dataquality/src/airflow/providers/common/dataquality/get_provider_info.py	Generated provider info metadata (operators/decorators/config).
providers/common/dataquality/src/airflow/providers/common/dataquality/version_compat.py	Airflow-version compatibility helper constants.
providers/common/dataquality/src/airflow/providers/common/dataquality/exceptions.py	Provider-specific exceptions.
providers/common/dataquality/src/airflow/providers/common/dataquality/results.py	Immutable result record models and summary/score utilities.
providers/common/dataquality/src/airflow/providers/common/dataquality/rules/__init__.py	Public rules API exports.
providers/common/dataquality/src/airflow/providers/common/dataquality/rules/checks.py	Built-in check catalog and dimensions.
providers/common/dataquality/src/airflow/providers/common/dataquality/rules/rule.py	Pydantic models for Condition, DQRule, and RuleSet plus identity/serialization helpers.
providers/common/dataquality/src/airflow/providers/common/dataquality/engines/__init__.py	Public engine API exports.
providers/common/dataquality/src/airflow/providers/common/dataquality/engines/sql.py	SQL measurement engine (batched built-ins + per-rule custom SQL).
providers/common/dataquality/src/airflow/providers/common/dataquality/backends/__init__.py	Backend factory driven by [common.dataquality] config.
providers/common/dataquality/src/airflow/providers/common/dataquality/backends/object_storage.py	Object-storage/local-file JSON results backend with read indexes.
providers/common/dataquality/src/airflow/providers/common/dataquality/operators/__init__.py	Operator export.
providers/common/dataquality/src/airflow/providers/common/dataquality/operators/dq_check.py	DQCheckOperator implementation (evaluation, persistence, asset event attachment).
providers/common/dataquality/src/airflow/providers/common/dataquality/decorators/__init__.py	Decorators package init.
providers/common/dataquality/src/airflow/providers/common/dataquality/decorators/dq_check.py	@task.dq_check TaskFlow decorator implementation.
providers/common/dataquality/src/airflow/providers/common/dataquality/assets.py	Asset config attachment and consumer gating (require_quality).
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/__init__.py	Example Dag package init.
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/orders_ruleset.yaml	Example YAML ruleset file.
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/example_dq_ruleset_from_yaml.py	Example Dag: ruleset loaded from YAML.
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/example_dq_require_quality.py	Example Dags: producer check + consumer gated on quality score.
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/example_dq_llm_generated_ruleset.py	Example Dag: LLM-generated ruleset + runtime ruleset check.
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/example_dq_check_decorator_dynamic.py	Example Dag: decorator with runtime ruleset selection.
providers/common/dataquality/src/airflow/providers/common/dataquality/example_dags/example_dq_check_custom_sql.py	Example Dag: custom_sql rules usage.
providers/common/dataquality/src/airflow/providers/common/dataquality/skills/dataquality-rule-authoring/SKILL.md	Agent skill doc describing the RuleSet/DQRule JSON shape and check catalog.
providers/common/dataquality/src/airflow/providers/common/dataquality/skills/dataquality-rule-authoring/references/ruleset.schema.json	Pydantic-generated JSON schema snapshot for RuleSet.
providers/common/dataquality/README.rst	Provider package readme (generated style).
providers/common/dataquality/pyproject.toml	Provider package metadata, deps, and uv workspace sources.
providers/common/dataquality/provider.yaml	Provider metadata (operators, decorators, config section/options).
providers/common/dataquality/LICENSE	Provider distribution license.
providers/common/dataquality/NOTICE	Provider distribution notice.
providers/common/dataquality/docs/conf.py	Sphinx config for provider docs build.
providers/common/dataquality/docs/index.rst	Provider docs home page and toctrees.
providers/common/dataquality/docs/rules.rst	Rules/checks/conditions documentation.
providers/common/dataquality/docs/operators.rst	DQCheckOperator documentation.
providers/common/dataquality/docs/decorators.rst	@task.dq_check documentation.
providers/common/dataquality/docs/assets.rst	Asset attachment and gating documentation.
providers/common/dataquality/docs/agents.rst	LLM/agent skill integration documentation.
providers/common/dataquality/docs/security.rst	Provider security include.
providers/common/dataquality/docs/configurations-ref.rst	Provider config reference includes.
providers/common/dataquality/docs/installing-providers-from-sources.rst	Standard “installing from sources” include.
providers/common/dataquality/docs/changelog.rst	Initial provider changelog stub.
providers/common/dataquality/docs/commits.rst	Generated commits list template hook.
providers/common/dataquality/.pre-commit-config.yaml	Provider-local prek hook to keep schema snapshot up to date.
providers/common/dataquality/.gitignore	Provider-local gitignore.
docs/spelling_wordlist.txt	Adds “ConfigDict” and “dataquality” to spelling allowlist.
dev/breeze/tests/test_selective_checks.py	Updates selective-checks expected provider doc list to include common.dataquality.
dev/breeze/src/airflow_breeze/global_constants.py	Adds common.dataquality to a Breeze “remove-providers” list for a matrix entry.
dev/breeze/doc/images/output_workflow-run_publish-docs.txt	Updates a recorded output hash for docs publish workflow.
dev/breeze/doc/images/output_sbom_generate-providers-requirements.txt	Updates recorded output hash for SBOM provider requirements.
dev/breeze/doc/images/output_release-management_publish-docs.txt	Updates recorded output hash for release-management docs publish.
dev/breeze/doc/images/output_release-management_prepare-provider-documentation.txt	Updates recorded output hash for provider docs preparation.
dev/breeze/doc/images/output_release-management_prepare-provider-distributions.txt	Updates recorded output hash for provider distribution preparation.
dev/breeze/doc/images/output_release-management_generate-providers-metadata.txt	Updates recorded output hash for providers metadata generation.
dev/breeze/doc/images/output_release-management_generate-issue-content-providers.txt	Updates recorded output hash for issue content generation.
dev/breeze/doc/images/output_release-management_classify-provider-changes.txt	Updates recorded output hash for provider change classification.
dev/breeze/doc/images/output_release-management_add-back-references.txt	Updates recorded output hash for adding back-references.
dev/breeze/doc/images/output_build-docs.txt	Updates recorded output hash for docs build output.
airflow-core/tests/unit/always/test_project_structure.py	Adds an “overlooked tests” entry for a missing provider test file.
airflow-core/docs/extra-packages-ref.rst	Documents the new common-dataquality extra.
.github/ISSUE_TEMPLATE/1-airflow_bug_report.yml	Adds common-dataquality as a selectable provider in bug reports.
.github/boring-cyborg.yml	Adds a path-based label rule for providers/common/dataquality/**.

[kaxil]

Since all built-ins run as one UNION ALL, a single bad rule (say an LLM-generated MIN(no_such_column)) fails the whole batch here, and every other rule gets recorded as ERROR with that unrelated database message. _raise_for_failures then fails the task on ERROR regardless of fail_on, so one bad generated rule takes down rules that would have passed, and their persisted history shows an error that isn't theirs -- which cuts against "Every rule is evaluated and recorded regardless of the task outcome" in the operator docstring. A per-rule fallback when the batch errors would keep the batching win on the happy path and isolate the blast radius. Related: duration_ms on each observation is the whole batch's elapsed time, which will mislead once the UI charts per-rule durations.

[kaxil]

Design question, non-blocking: core dedupes assets by (name, uri) and sync_assets overwrites model.extra on every parse, last writer wins -- so a second Dag referencing Asset("orders") without quality config wipes the registry copy of the ruleset, and two Dags with different rulesets flip-flop. This slice is unaffected (the operator reads the in-memory object, the gate reads event extra), but the #69413 UI presumably reads the registry copy. How is the attached config expected to survive there?

[gopidesupavan]

ah good point , The intended contract is one owner per asset: whichever Dag calls asset_quality() on a given (name, uri) should be the only one attaching config to it. this mirrors the existing last-writer-wins behavior on extra in general, just newly visible through quality config.

I will make that explicit in the docstring, I think a real merge strategy across multiple owners would be a bigger design change ? may be that can be future..