Skip to content

fix(db_cleanup): handle NULL start_date in dag_run cleanup#69879

Open
bileyroy wants to merge 1 commit into
apache:mainfrom
bileyroy:fix-db-clean-null-start-date
Open

fix(db_cleanup): handle NULL start_date in dag_run cleanup#69879
bileyroy wants to merge 1 commit into
apache:mainfrom
bileyroy:fix-db-clean-null-start-date

Conversation

@bileyroy

@bileyroy bileyroy commented Jul 14, 2026

Copy link
Copy Markdown

closes: #69872

Summary

When start_date is NULL (e.g. dag_runs that were marked failed before execution began), the cleanup query's WHERE start_date < cutoff evaluates to NULL and silently skips these rows forever.

Changes

  • Add fallback_recency_column_name field to _TableConfig (set to created_at for dag_run)
  • Add --fallback-cleanup-on-null CLI flag (opt-in, off by default)
  • When enabled, apply COALESCE(start_date, created_at) to:
    • Main WHERE condition: so NULL start_date rows are found
    • keep_last subquery (MAX): so the latest run is correctly identified via fallback
    • keep_last JOIN condition: so the latest run is properly protected
  • Add 5 unit tests covering flag behavior and keep_last protection

Test plan

  • pytest airflow-core/tests/unit/utils/test_db_cleanup.py -xvs — all passed
  • ruff check / ruff format — clean
  • mypy — clean
  • Manual testing via airflow standalone

Testing with manual runs (keep_last not applicable):

image image image

Testing with backfill runs (keep_last applicable)

All 3 backfill runs have start_date = NULL. With --fallback-cleanup-on-null, only 2 rows are returned for deletion, the newest run is correctly protected by keep_last via COALESCE(start_date, created_at):

image image

@boring-cyborg

boring-cyborg Bot commented Jul 14, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

  When start_date is NULL (e.g. runs that failed or were killed before
  execution began), the WHERE condition start_date < cutoff evaluates
  to NULL and silently skips these rows forever.

  Add --fallback-cleanup-on-null CLI flag that uses
  COALESCE(start_date, created_at) in the WHERE condition, keep_last
  subquery, and JOIN so that NULL start_date rows are cleaned based on
  their created_at timestamp while the most recent run per dag is still
  protected by keep_last.

  closes: apache#69872
@bileyroy bileyroy force-pushed the fix-db-clean-null-start-date branch from f173ee5 to 9ad7c67 Compare July 14, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow db clean cannot delete dag_run records with NULL start_date

1 participant