Skip to content

[Opt](lambda) Aviod redundant copies during lambda execute - #66590

Open
linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:opt-array-map
Open

[Opt](lambda) Aviod redundant copies during lambda execute#66590
linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:opt-array-map

Conversation

@linrrzqqq

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

array_map flattens array elements into an internal block before executing its lambda expression. The previous implementation introduced several redundant copies and materializations on this path:

  • Nested array elements were copied into new columns even when all elements could be processed in a single lambda batch.
  • Captured outer columns were expanded repeatedly for each outer row.
  • Constant captured columns were materialized as full columns.
  • Sparse, unused input positions could create unnecessary placeholder column work.
  • The first lambda result batch was copied into a newly created result column.
  • Internal lambda execution could receive the outer selector even though its input block had already been selected and flattened.

These operations increased CPU usage and peak memory consumption, especially for large arrays, wide captured inputs, and complex lambda expressions.

This change optimizes the array_map execution path by:

  • Reusing the nested array data columns directly when all nested rows fit within one lambda batch.
  • Preserving captured constant columns as ColumnConst.
  • Expanding row-dependent captured columns in one operation with selector-aware source row indices.
  • Representing unused sparse input positions with ColumnNothing.
  • Reusing lambda block column allocations across batches.
  • Taking ownership of the first lambda result column instead of cloning and copying it.
  • Passing no outer selector when executing the already-materialized internal lambda block.
  • Centralizing nullable array result construction to keep the execution path simpler.
  • Sizing internal lambda batches according to the runtime row and byte budgets.

The optimization preserves array element order, nullable semantics, captured-column mapping, nested lambda behavior, and SQL result compatibility.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again.
Workflow run: https://github.com/apache/doris/actions/runs/31246269731

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

@linrrzqqq review

@linrrzqqq linrrzqqq left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rarion independent review completed and converged.

No findings.

This result is based on an independent static code review. No builds or tests were run; refer to CI for test status.

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant