Restoring `include_downstream_dags` logic to clear downstream Dags by jroachgolf84 · Pull Request #65314 · apache/airflow

jroachgolf84 · 2026-04-15T12:50:50Z

Description

Restoring behavior to clear downstream DAGs when a Task state is cleared (if applicable).

closes: #61451

Testing

Unit-tests were added/updated to test these changes. The changes were also tested E2E.

Unit Tests

The unit-tests added/updated as part of this effort can be found in these files:

airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_task_instances.py
airflow-core/tests/unit/serialization/definitions/test_dag.py

These tests can be executed using the commands below:

breeze testing core-tests airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_task_instances.py

breeze testing core-tests airflow-core/tests/unit/serialization/definitions/test_dag.py

E2E Testing

The DAG below was used to test the changes int his PR. This DAG is made up of a parent and child DAG. The ExternalTaskMarker Operator is cleared in the parent DAG, the downstream Tasks in the child DAG are also cleared. That's showed in the video below.

from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.sensors.external_task import ExternalTaskMarker, ExternalTaskSensor
from airflow.sdk import DAG

from datetime import datetime


with DAG(
    dag_id="parent_dag_61451",
    start_date=datetime(2026, 1, 18),
    schedule="@daily",
    catchup=False,
    tags=["issue-61451"],
) as parent_dag:
    parent_echo = BashOperator(
        task_id="parent_echo",
        bash_command="echo parent_dag_61451",
    )

    trigger_child = ExternalTaskMarker(
        task_id="trigger_child",
        external_dag_id="child_dag_61451",
        external_task_id="receive_call_from_parent",
    )

    parent_echo >> trigger_child

with DAG(
    dag_id="child_dag_61451",
    start_date=datetime(2026, 1, 18),
    schedule="@daily",
    catchup=False,
    tags=["issue-61451"],
) as child_dag:
    receive_call_from_parent = ExternalTaskSensor(
        task_id="receive_call_from_parent",
        external_dag_id="parent_dag_61451",
        external_task_id="trigger_child",
        poke_interval=5,
    )

    child_echo = BashOperator(
        task_id="child_echo",
        bash_command="echo DONE",
    )

    receive_call_from_parent >> child_echo

Video

issue-61451-include-downstream-tasks.mov

LinasData · 2026-05-05T10:05:08Z

Hey, any updates on this?

jroachgolf84 · 2026-06-18T20:39:15Z

@jason810496 - can you take a look at this one for me?

jroachgolf84 · 2026-06-26T12:07:24Z

@bugraoz93 - do you mind taking a look at this one?

jroachgolf84 · 2026-07-10T12:50:48Z

@kaxil - do you mind taking another look?

eladkal

Overall LGTM
but needs 2nd eye from @kaxil

github-actions · 2026-07-20T15:46:49Z

uv.lock on main just moved via #70217 ("[main] Upgrade important CI environment"), commit 97515b5 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

jroachgolf84 · 2026-07-22T03:10:11Z

@kaxil - feedback implemented. Do you mind taking another look? Thanks!

Lee-W

Mostly nits, but overall this looks good. The CI failure is unrelated—rebasing on the main branch should fix it.

Lee-W · 2026-07-22T08:35:02Z

+        )
+
+        assert response.status_code == 404
+        assert "child_dag" in response.json()["detail"]


let's check the full message

Lee-W · 2026-07-22T08:35:15Z

+    @mock.patch("airflow.serialization.definitions.dag.SerializedDAG.clear")
+    def test_invalid_external_task_marker_logical_date_returns_400(self, mock_clear, test_client, session):
+        """A non-ISO logical_date rendered from an ExternalTaskMarker template must return 400, not 500."""
+        from pendulum.parsing.exceptions import ParserError


let's move it to the top

@Lee-W - do you mean the import?

Co-authored-by: Wei Lee <hello@wei-lee.me>

Lee-W · 2026-07-23T06:22:40Z

+        raise HTTPException(status.HTTP_404_NOT_FOUND, str(e)) from e
+
+    if include_dependent_dags:
+        # Ensure proper access to downstream dags/tasks with dag.clear and include_dependent_dags


Suggested change

# Ensure proper access to downstream dags/tasks with dag.clear and include_dependent_dags

# Ensure proper access to downstream Dags/tasks with dag.clear and include_dependent_dags

Lee-W · 2026-07-23T06:30:54Z

+
+
+def test_clear_raises_when_recursion_depth_exceeded(dag_maker, session):
+    """AirflowException is raised when the dependency chain depth exceeds recursion_depth."""


Should it be MaxRecursionDepthError instead?

boring-cyborg Bot added area:API Airflow's REST/HTTP API area:DAG-processing labels Apr 15, 2026

jroachgolf84 mentioned this pull request Apr 15, 2026

Fix ExternalTaskMarker cross-DAG recursive clearing in Airflow 3 #65240

Closed

2 tasks

jroachgolf84 marked this pull request as ready for review May 22, 2026 20:45

jroachgolf84 requested review from ashb, bbovenzi, bolkedebruin, bugraoz93, choo121600, ephraimbuddy, guan404ming, jason810496, pierrejeambrun, rawwar, ryanahamilton, shubhamraj-git and vatsrahul1001 as code owners May 22, 2026 20:45

jroachgolf84 requested review from dheerajturaga and potiuk as code owners May 23, 2026 13:29

jroachgolf84 added the ready for maintainer review Set after triaging when all criteria pass. label Jun 13, 2026

kaxil reviewed Jun 26, 2026

View reviewed changes

kaxil changed the title ~~issue-61451: Restoring include_downstream_dags logic~~ Restoring include_downstream_dags logic to clear downstream Dags Jun 26, 2026

jroachgolf84 requested a review from henry3260 as a code owner July 6, 2026 13:29

jroachgolf84 requested a review from kaxil July 6, 2026 13:29

eladkal added this to the Airflow 3.4.0 milestone Jul 17, 2026

eladkal added the type:improvement Changelog: Improvements label Jul 17, 2026

eladkal approved these changes Jul 17, 2026

View reviewed changes

kaxil reviewed Jul 17, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/serialization/definitions/dag.py

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/task_instances.py Outdated

Comment thread airflow-core/src/airflow/serialization/definitions/dag.py

kaxil reviewed Jul 21, 2026

View reviewed changes

Comment thread airflow-core/newsfragments/65314.improvement.rst Outdated

jroachgolf84 added 7 commits July 21, 2026 22:03

issue-61451: Restoring include_downstream_dags logic

550eb73

issue-61451: Error handling, adding tests

9805cce

issue-61451: Generating datamodels

86a75fe

issue-61451: Fixing CI failures

70c4154

issue-61451: Fixing CI failures

1950d94

issue-63451: Implementing feedback

4eaeb1b

issue-61451: Implementing feedback for clearing TI's

bd41ac1

jroachgolf84 force-pushed the issue-61451 branch from 8ada24e to bd41ac1 Compare July 22, 2026 02:05

issue-61451: Passing through DagBag on clear

5ea452a

jroachgolf84 requested a review from XD-DENG as a code owner July 22, 2026 03:08

Merge branch 'main' into issue-61451

29ae371

jroachgolf84 requested a review from kaxil July 22, 2026 03:09

Lee-W reviewed Jul 22, 2026

View reviewed changes

jroachgolf84 and others added 2 commits July 22, 2026 23:22

Apply suggestions from code review

1456d06

Co-authored-by: Wei Lee <hello@wei-lee.me>

Apply suggestions from code review

c2df99c

Co-authored-by: Wei Lee <hello@wei-lee.me>

Lee-W reviewed Jul 23, 2026

View reviewed changes

	# Ensure proper access to downstream dags/tasks with dag.clear and include_dependent_dags
	# Ensure proper access to downstream Dags/tasks with dag.clear and include_dependent_dags



		def test_clear_raises_when_recursion_depth_exceeded(dag_maker, session):
		"""AirflowException is raised when the dependency chain depth exceeds recursion_depth."""

Uh oh!

Conversation

jroachgolf84 commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Unit Tests

E2E Testing

Video

Uh oh!

LinasData commented May 5, 2026

Uh oh!

jroachgolf84 commented Jun 18, 2026

Uh oh!

jroachgolf84 commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jroachgolf84 commented Jul 10, 2026

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jroachgolf84 commented Jul 22, 2026

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lee-W Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Lee-W Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

jroachgolf84 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Lee-W Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Lee-W Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Lee-W Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jroachgolf84 commented Apr 15, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading