Fix N+1 XCom fetches when iterating a mapped task's lazy sequence - #40
Open
nancy-wang-dd wants to merge 1 commit into
Open
Fix N+1 XCom fetches when iterating a mapped task's lazy sequence#40nancy-wang-dd wants to merge 1 commit into
nancy-wang-dd wants to merge 1 commit into
Conversation
Forward iteration over LazyXComSequence previously issued one HTTP request per item via the Execution API. A task iterating a mapped upstream task's full output (e.g. via zip()/concat() on XComArgs) would fire a sequential HTTP call per item instead of fetching in bulk. Forward iteration now fetches items in chunks using the existing slice endpoint, matching the batching already used by BaseXCom.get_all(). Backward iteration and single-item/slice access are unchanged.
nancy-wang-dd
marked this pull request as ready for review
July 16, 2026 20:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward iteration over LazyXComSequence (task-sdk/src/airflow/sdk/execution_time/lazy_sequence.py) issued one Execution API HTTP request per item. A task iterating a mapped upstream task's full XCom output — directly, or via zip()/concat() on XComArgs — fired a sequential HTTP call per item, which Datadog APM's "Sequential API Calls" detector flagged under airflow-worker's execute_workload span.
Jira Ticket
What
LazyXComIterator.next now fetches items in chunks (50 at a time) using the existing GetXComSequenceSlice / get_sequence_slice endpoint, instead of one GetXComSequenceItem request per item. This mirrors the batching already used by BaseXCom.get_all().
How it behaves
Test plan
concurrency/shared-state, error-handling divessues found.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 5) following the guidelines (https://github.com/apache/airflow/blob/main/requests.rst#gen-ai-assisted-contributions)