Fix MariaDB compatibility#66711
Open
rusher wants to merge 3 commits into
Open
Conversation
* with_row_locks: Removed incorrect supports_for_update_of guard: MariaDB 10.6+ supports FOR UPDATE, NOWAIT, SKIP LOCKED, and SQLAlchemy support it natively well.
* Migration 0036: Fix MariaDB error 1093 ("can't specify target
table in FROM clause of a subquery that modifies it") by wrapping
the subquery in a derived table.
* Migration 0049: Branch on dialect.is_mariadb to use the 3-argument
PCRE2 REGEXP_REPLACE signature MariaDB supports, instead of MySQL 8's
6-argument form.
* scheduler_job_runner.py: Fix _orphan_unreferenced_assets for MariaDB
by switching the orphan query from .cte() to .subquery(). MariaDB's
parser rejects `WITH cte AS (...) DELETE FROM t ...` (corrected in 12.3)
* test_exceptions.py: Update assertions to tolerate driver/server
format variants
…lavored/mariadb-version
rusher
requested review from
XD-DENG,
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl,
kaxil,
potiuk and
vatsrahul1001
as code owners
boring-cyborg
Bot
added
area:API
jscheffl
reviewed
May 11, 2026
Contributor
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
I would be okay by adding a "non committed" support for mariadb but have no strong opinion. Assuming it is compatible and for the moment not a promise that Airflow itself is fully compatible and tested against. A kind of pre-experimental stage. Changes look minimal but if any problems and missing maintainer support then being at risk being removed again.
Comment on lines
+330
to
+349
| option_mysql_flavor = click.option( | ||
| "--mysql-flavor", | ||
| help=( | ||
| "Flavor of the mysql backend to run. 'mysql' (default) uses upstream MySQL with " | ||
| "--mysql-version. 'mariadb' substitutes a MariaDB image (selected with --mariadb-version) " | ||
| "but airflow still connects via mysql+pymysql:// — SQLAlchemy detects MariaDB at runtime." | ||
| ), | ||
| type=CacheableChoice(ALLOWED_MYSQL_FLAVORS), | ||
| default=CacheableDefault(ALLOWED_MYSQL_FLAVORS[0]), | ||
| envvar="MYSQL_FLAVOR", | ||
| show_default=True, | ||
| ) | ||
| option_mariadb_version = click.option( | ||
| "--mariadb-version", | ||
| help="Version of MariaDB to run when --mysql-flavor=mariadb. Ignored otherwise.", | ||
| type=BackendVersionChoice(ALLOWED_MARIADB_VERSIONS), | ||
| default=CacheableDefault(ALLOWED_MARIADB_VERSIONS[-1]), | ||
| envvar="MARIADB_VERSION", | ||
| show_default=True, | ||
| ) |
Contributor
There was a problem hiding this comment.
Instead of adding two CLI options would it be maybe rather favorable to add one switch --flavor [mariadb|mysql]? Then we only need one additional CLI argument. Defaulting to mysql
Contributor
Not sure though if we should backport - only it if loweres compelxity of tooling mainteanance but for sure not to add complexity on 3.2 maintenance. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Airflow’s ability to run and be tested against MariaDB while still using the existing MySQL backend code paths (via SQLAlchemy’s dialect.is_mariadb), and adds a scheduled “canary” CI workflow to exercise DB tests on MariaDB.
Changes:
- Add Breeze support for choosing a MySQL “flavor” (
mysqlvsmariadb) and selecting a MariaDB version, including a MariaDB docker-compose backend file. - Add a scheduled + manually-triggered GitHub Actions workflow that runs the DB-test subset against MariaDB.
- Apply targeted MariaDB SQL compatibility adjustments (migrations, row-lock helper, scheduler asset orphan cleanup) and relax test assertions where DB/driver formatting differs.
Reviewed changes
Copilot reviewed 129 out of 140 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/docker/entrypoint_ci.sh | Print MariaDB backend/version when Breeze selects MariaDB flavor. |
| scripts/ci/docker-compose/backend-mysql-mariadb.yml | New MariaDB-backed compose file while keeping BACKEND=mysql and mysql dialect. |
| Dockerfile.ci | Mirror CI entrypoint backend info output for MariaDB flavor. |
| dev/breeze/src/airflow_breeze/params/shell_params.py | Add mysql_flavor/mariadb_version params and select MariaDB compose file. |
| dev/breeze/src/airflow_breeze/global_constants.py | Define allowed MySQL flavors and MariaDB versions for Breeze. |
| dev/breeze/src/airflow_breeze/commands/testing_commands.py | Thread new options through testing commands. |
| dev/breeze/src/airflow_breeze/commands/testing_commands_config.py | Expose new CLI flags in testing command config. |
| dev/breeze/src/airflow_breeze/commands/setup_commands.py | Expose new CLI flags in setup commands and print config. |
| dev/breeze/src/airflow_breeze/commands/setup_commands_config.py | Expose new CLI flags in setup command config. |
| dev/breeze/src/airflow_breeze/commands/main_command.py | Add new common CLI options at the top-level command. |
| dev/breeze/src/airflow_breeze/commands/developer_commands.py | Thread new options through developer commands (shell/start/doctor/run). |
| dev/breeze/src/airflow_breeze/commands/developer_commands_config.py | Expose new CLI flags in developer command config. |
| dev/breeze/src/airflow_breeze/commands/common_options.py | Add --mysql-flavor and --mariadb-version click options. |
| dev/breeze/doc/images/output_workflow-run_publish-docs.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ui_compile-assets.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ui_check-translation-completeness.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_ui-e2e-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_task-sdk-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_task-sdk-integration-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_system-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_python-api-client-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_python-api-client-tests.svg | Regenerated Breeze CLI help terminal SVG (new options visible). |
| dev/breeze/doc/images/output_testing_providers-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_providers-integration-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_providers-integration-tests.svg | Regenerated Breeze CLI help terminal SVG (new options visible). |
| dev/breeze/doc/images/output_testing_helm-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_docker-compose-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_core-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_core-integration-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_core-integration-tests.svg | Regenerated Breeze CLI help terminal SVG (new options visible). |
| dev/breeze/doc/images/output_testing_airflow-e2e-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_airflow-ctl-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_testing_airflow-ctl-integration-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_start-airflow.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_shell.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_setup_version.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_setup_synchronize-local-mounts.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_setup_regenerate-command-images.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_setup_config.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_setup_config.svg | Regenerated Breeze CLI help terminal SVG (new options visible). |
| dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_setup_autocomplete.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_sbom_update-sbom-information.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_sbom_generate-providers-requirements.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_sbom_export-dependency-information.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_sbom_build-all-airflow-images.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_run.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_verify-rc-by-pmc.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_verify-provider-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_update-providers-next-version.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_update-constraints.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_tag-providers.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_start-release.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_start-rc-process.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_release-prod-images.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_publish-docs.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_publish-docs-to-s3.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-task-sdk-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-tarball.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-python-client.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-provider-documentation.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-provider-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-mypy-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-helm-chart-tarball.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-helm-chart-package.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-airflow-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_merge-prod-images.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_install-provider-distributions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-providers-metadata.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-issue-content-providers.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-issue-content-helm-chart.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-issue-content-core.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-issue-content-airflow-ctl.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-constraints.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_generate-airflowctl-changelog.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_create-minor-branch.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_constraints-version-check.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_clean-old-provider-artifacts.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_check-release-files.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_release-management_add-back-references.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_registry_publish-versions.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_registry_extract-data.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_registry_backfill.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_prod-image_verify.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_prod-image_save.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_prod-image_pull.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_prod-image_load.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_prod-image_build.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_upload-k8s-image.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_status.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_shell.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_setup-env.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_run-complete-tests.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_logs.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_k9s.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_dev.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_deploy-cluster.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_deploy-airflow.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_delete-cluster.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_create-cluster.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_configure-cluster.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_k8s_build-k8s-image.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_issues_unassign.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_generate-migration-file.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_exec.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_down.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_doctor.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_cleanup.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_verify.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_save.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_pull.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_load.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_import-mount-cache.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_export-mount-cache.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci-image_build.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_upgrade.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_set-milestone.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_selective-check.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_resource-check.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_get-workflow-info.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_free-space.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_ci_fix-ownership.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/images/output_build-docs.txt | Regenerated Breeze command output artifact checksum. |
| dev/breeze/doc/03_developer_tasks.rst | Document how to run Breeze against MariaDB and reference the canary workflow. |
| airflow-core/tests/unit/utils/test_sqlalchemy.py | Update row-lock helper expectations for mysql dialect. |
| airflow-core/tests/unit/api_fastapi/common/test_exceptions.py | Relax expected error formatting to allow MariaDB/driver variants. |
| airflow-core/src/airflow/utils/sqlalchemy.py | Adjust row-locking logic and update docstring to reflect MariaDB behavior. |
| airflow-core/src/airflow/migrations/versions/0049_3_0_0_remove_pickled_data_from_xcom_table.py | Handle MariaDB’s different REGEXP_REPLACE signature in migration. |
| airflow-core/src/airflow/migrations/versions/0036_3_0_0_add_name_field_to_dataset_model.py | Work around MySQL/MariaDB 1093 limitation during downgrade cleanup. |
| airflow-core/src/airflow/jobs/scheduler_job_runner.py | Use subquery instead of CTE for MariaDB parser compatibility in delete path. |
| .github/workflows/mariadb-compat-canary.yml | Add scheduled/manual MariaDB compatibility workflow running DB tests via Breeze. |
| else: | ||
| op.execute( | ||
| """ | ||
| with unique_dataset as (select min(id) as min_id, uri as uri from dataset group by id), |
Comment on lines
+291
to
+296
| response_detail = exeinfo_response_error.value.detail | ||
| expected_detail = expected_exception.detail | ||
| for key in ("statement", "orig_error"): | ||
| actual_val = response_detail.pop(key, None) # type: ignore[attr-defined] | ||
| expected_val = expected_detail.pop(key, None) # type: ignore[attr-defined] | ||
| if isinstance(expected_val, list): |
Comment on lines
408
to
+413
| response_detail = exeinfo_response_error.value.detail | ||
| expected_detail = expected_exception.detail | ||
| actual_statement = response_detail.pop("statement", None) # type: ignore[attr-defined] | ||
| expected_detail.pop("statement", None) | ||
| expected_detail.pop("statement", None) # type: ignore[attr-defined] | ||
| actual_orig = response_detail.pop("orig_error", None) # type: ignore[attr-defined] | ||
| expected_orig = expected_detail.pop("orig_error", None) # type: ignore[attr-defined] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds MariaDB compatibility to Airflow, using the existing MySQL backend.
this follow #60133, correcting Copilot review
What's included
0036: wrap subquery in derived table to avoid MariaDB error 1093.0049: branch ondialect.is_mariadbfor the 3-argument PCRE2REGEXP_REPLACEsignature.with_row_locks: drop the incorrectsupports_for_update_ofguard(MariaDB 10.6+ supports
FOR UPDATE … NOWAIT / SKIP LOCKEDnatively)._orphan_unreferenced_assets: switch.cte()→.subquery()for theWITH … DELETEpattern MariaDB's parser rejects pre-12.3.test_exceptions: tolerate driver/server format variants in assertions.in order to permit testing mariadb without testing each commit (1118fc6):
Breeze support: new
--mysql-flavorand--mariadb-versionoptions sothe same backend code path can target either MySQL or MariaDB locally.
Weekly canary CI: scheduled workflow that builds the CI image and runs
breeze testing core-tests --backend mysql --mysql-flavor mariadb --run-db-tests-only --run-in-parallel. Failures are informational, not aPR gate.
canary test works on on fork https://github.com/rusher/airflow/actions/runs/25660511634/job/75319570502