GH-50634: [C++][Engine] Fix Substrait consumer silently dropping modern FetchRel/AggregateRel fields by fangchenli · Pull Request #50635 · apache/arrow

fangchenli · 2026-07-25T04:57:04Z

Rationale for this change

Arrow's Substrait consumer read several fields through deprecated accessors from a vendored substrait proto pinned to v0.44.0, so values sent by current producers (which use the newer forms introduced by ~v0.63.0) were silently dropped and defaulted, producing wrong results with no error.

FetchRel (LIMIT/OFFSET): count/offset became oneofs (count_mode/offset_mode), and current producers (e.g. DataFusion, substrait-rs 0.63.0) emit the count_expr/offset_expr expression arms, leaving the scalar fields unset. The consumer reads the default 0, turning every LIMIT n into LIMIT 0 (empty result) and dropping OFFSET (skipping no rows).

AggregateRel (GROUP BY): grouping expressions moved from the deprecated inline Grouping.grouping_expressions to Grouping.expression_references, indexing the AggregateRel-level grouping_expressions. Reading only the deprecated field dropped the group keys, collapsing every row into a single group.

I'm not sure about the backward compatibility policy for Substrait, so I kept the old path and only bumped the version to 0.63.0 to match DataFusion.

What changes are included in this PR?

Bump the vendored substrait proto from v0.44.0 to v0.63.0 (matching DataFusion's pinned substrait-rs), where these newer forms were introduced while the deprecated arms remain for backward compatibility.
Rewrite the FetchRel consumer to read the count_mode/offset_mode oneofs: evaluate count_expr/offset_expr to a constant int64 literal, fall back to the deprecated scalar arm, treat an unset/null count as ALL and an unset/null offset as 0. A no-op fetch (ALL, offset 0) is skipped so it does not impose the fetch node's input-ordering requirement on plans that never limit output.
Read AggregateRel grouping keys via expression_references into the AggregateRel-level grouping_expressions, falling back to the deprecated inline field for older producers.
Update the JoinRel consumer for the renamed JoinType enum values (JOIN_TYPE_SEMI/ANTI -> JOIN_TYPE_LEFT_SEMI/LEFT_ANTI) surfaced by the bump.
Add regression tests for the count_expr/offset_expr arms, unset-count = ALL, and grouping via expression_references.

Are these changes tested?

Yes, tests added.

Are there any user-facing changes?

Yes, bug fix.

Closes #50634

This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)

GitHub Issue: [C++][Engine] Substrait consumer silently drops modern FetchRel and AggregateRel fields #50634

github-actions · 2026-07-25T04:57:27Z

⚠️ GitHub issue #50634 has been automatically assigned in GitHub to PR creator.

Copilot

Pull request overview

Updates Arrow C++ Substrait support to correctly consume modern Substrait v0.63-style fields (notably FetchRel count/offset oneofs and AggregateRel grouping expression references) after bumping the vendored Substrait protos, preventing silently wrong LIMIT/OFFSET and GROUP BY behavior.

Changes:

Bump vendored Substrait proto version to v0.63.0 and adjust protobuf tooling requirements/flags accordingly.
Fix Substrait relation deserialization to correctly interpret FetchRel count_expr/offset_expr and AggregateRel grouping expression_references (with backward-compatible fallbacks).
Add regression tests covering modern FetchRel expression arms, unset-count semantics, and AggregateRel grouping references; update JoinRel enum handling for renamed values.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cpp/thirdparty/versions.txt	Bumps the vendored Substrait tarball version/hash to v0.63.0.
cpp/src/arrow/engine/substrait/test_plan_builder.cc	Suppresses deprecation warnings when intentionally emitting deprecated grouping fields in test helpers.
cpp/src/arrow/engine/substrait/serde_test.cc	Adds regression tests for FetchRel expression arms, unset-count semantics, and AggregateRel grouping via expression references.
cpp/src/arrow/engine/substrait/relation_internal.cc	Updates Substrait-to-Acero conversion for modern FetchRel/AggregateRel fields and JoinRel enum rename handling.
cpp/cmake_modules/ThirdpartyToolchain.cmake	Raises required protobuf version for Substrait and adds protoc flags for proto3 optional support where needed.

+ARROW_SUBSTRAIT_BUILD_VERSION=v0.63.0
+ARROW_SUBSTRAIT_BUILD_SHA256_CHECKSUM=14aabf828d54fbae38aa90f2c10006952c0040feff8f813cabad5bc13d6fbcfe


+  auto sp_ext_id_reg = MakeExtensionIdRegistry();
+  ASSERT_OK_AND_ASSIGN(auto sink_decls,
+                       DeserializePlans(*buf, [] { return kNullConsumer; }));


…g modern FetchRel/AggregateRel fields Arrow's Substrait consumer read several fields through deprecated accessors from a vendored substrait proto pinned to v0.44.0, so values sent by current producers (which use the newer forms introduced by ~v0.63.0) were silently dropped and defaulted, producing wrong results with no error. FetchRel (LIMIT/OFFSET): count/offset became oneofs (count_mode/offset_mode) and current producers (e.g. DataFusion, substrait-rs 0.63.0) emit the count_expr/offset_expr expression arms, leaving the scalar fields unset. The consumer read the default 0, turning every `LIMIT n` into `LIMIT 0` (empty result) and dropping OFFSET (skipping no rows). AggregateRel (GROUP BY): grouping expressions moved from the deprecated inline `Grouping.grouping_expressions` to `Grouping.expression_references` indexing the AggregateRel-level `grouping_expressions`. Reading only the deprecated field dropped the group keys, collapsing every row into a single group. - Bump the vendored substrait proto from v0.44.0 to v0.63.0 (matching DataFusion's pinned substrait-rs), where these newer forms were introduced while the deprecated arms remain for backward compatibility. - Rewrite the FetchRel consumer to read the count_mode/offset_mode oneofs: evaluate count_expr/offset_expr to a constant int64 literal, fall back to the deprecated scalar arm, treat an unset/null count as ALL and an unset/null offset as 0. A no-op fetch (ALL, offset 0) is skipped so it does not impose the fetch node's input-ordering requirement on plans that never limit output. - Read AggregateRel grouping keys via expression_references into the AggregateRel-level grouping_expressions, falling back to the deprecated inline field for older producers. - Update the JoinRel consumer for the renamed JoinType enum values (JOIN_TYPE_SEMI/ANTI -> JOIN_TYPE_LEFT_SEMI/LEFT_ANTI) surfaced by the bump. - Add regression tests for the count_expr/offset_expr arms, unset-count = ALL, and grouping via expression_references. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

kou

Is this really a "Critical Fix"?

Could you read https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code ?

kou · 2026-07-25T08:57:42Z

@@ -2257,8 +2257,10 @@ if(ARROW_WITH_PROTOBUF)
    # 3.15.0-: don't need --experimental_allow_proto3_optional
    set(ARROW_PROTOBUF_REQUIRED_VERSION "3.12.0")
  elseif(ARROW_SUBSTRAIT)


Can we unify the ARROW_FLIGHT_SQL clause and the ARROW_SUBSTRAIT clause?

github-actions Bot added Component: C++ awaiting review Awaiting review labels Jul 25, 2026

fangchenli force-pushed the gh-substrait-fetchrel-limit branch 2 times, most recently from 7c14788 to 54ff52a Compare July 25, 2026 05:34

fangchenli marked this pull request as ready for review July 25, 2026 06:14

fangchenli requested a review from pitrou as a code owner July 25, 2026 06:14

Copilot AI review requested due to automatic review settings July 25, 2026 06:14

Copilot started reviewing on behalf of fangchenli July 25, 2026 06:14 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 25, 2026 06:22

fangchenli force-pushed the gh-substrait-fetchrel-limit branch from 54ff52a to 6ce2c2e Compare July 25, 2026 06:22

Copilot started reviewing on behalf of fangchenli July 25, 2026 06:23 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

kou reviewed Jul 25, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Jul 25, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-50634: [C++][Engine] Fix Substrait consumer silently dropping modern FetchRel/AggregateRel fields#50635

GH-50634: [C++][Engine] Fix Substrait consumer silently dropping modern FetchRel/AggregateRel fields#50635
fangchenli wants to merge 1 commit into
apache:mainfrom
fangchenli:gh-substrait-fetchrel-limit

fangchenli commented Jul 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

kou left a comment

Uh oh!

kou Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		ARROW_SUBSTRAIT_BUILD_VERSION=v0.63.0
		ARROW_SUBSTRAIT_BUILD_SHA256_CHECKSUM=14aabf828d54fbae38aa90f2c10006952c0040feff8f813cabad5bc13d6fbcfe

Uh oh!

Conversation

fangchenli commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

kou Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fangchenli commented Jul 25, 2026 •

edited

Loading