Skip to content

feat: expose residual_expression and post_join_filter on join builders#225

Open
nielspardon wants to merge 1 commit into
substrait-io:mainfrom
nielspardon:worktree-issue-190-residual-expression
Open

feat: expose residual_expression and post_join_filter on join builders#225
nielspardon wants to merge 1 commit into
substrait-io:mainfrom
nielspardon:worktree-issue-190-residual-expression

Conversation

@nielspardon

Copy link
Copy Markdown
Member

Completes the remaining work for #190: surface the two Expression fields spec #1044 added to HashJoinRel and MergeJoinRelresidual_expression and post_join_filter — on the ergonomic physical-join builders (hash_join / merge_join) and the corresponding DataFrame methods. They were previously only settable on the raw proto.

Binding semantics

Each expression is resolved against the schema the spec prescribes:

  • residual_expression is "evaluated on each candidate key-match" (both rows present), so it binds against the combined left+right schema.
  • post_join_filter is "applied to each output record after join-type-specific output formation... semantically equivalent to placing a FilterRel directly above this join", so it binds against the join output schema, reusing the existing _join_output_struct / join_output_names helpers.

Also fixes the logical join builder

The logical join builder previously bound post_join_filter against the combined schema too. For inner/outer/left/right/single/mark joins the output equals (or prefixes) the combined schema, so behavior is unchanged. For right_semi / right_anti (output is the right side only) a filter on a right column now uses the correct output-relative field index instead of a dangling combined-schema index, and a filter referencing a dropped side fails fast instead of emitting an invalid reference.

Notes

  • The optional predicate/extension arguments are keyword-only, matching the logical join builder's convention.
  • New tests cover inner + right_semi output binding, residual_expression combined binding, fail-fast on a dropped-side reference, keyword-only enforcement, and byte-identical DataFrame↔builder parity across inner/right_semi.

Closes #190

🤖 Generated with AI

Surface the two Expression fields spec #1044 added to HashJoinRel and
MergeJoinRel — residual_expression and post_join_filter — on the ergonomic
physical-join builders (hash_join / merge_join) and the DataFrame
hash_join / merge_join methods, completing the remaining work for substrait-io#190.

Bind each expression against the schema the spec prescribes:

- residual_expression is evaluated on each candidate key-match (both rows
  present), so it resolves against the combined left+right schema.
- post_join_filter is applied to each output record after join-type-specific
  output formation ("semantically a FilterRel directly above this join"), so
  it resolves against the join output schema, reusing the existing
  _join_output_struct / join_output_names helpers.

The same fix is applied to the logical join builder, which previously bound
post_join_filter against the combined schema. For inner/outer/left/right/
single/mark joins the output equals (or prefixes) the combined schema so
behavior is unchanged; for right_semi / right_anti (output is the right side
only) a filter on a right column now uses the correct output-relative field
index instead of a dangling combined-schema index, and a filter on a dropped
side fails fast instead of emitting an invalid reference.

The optional predicate/extension arguments are keyword-only, matching the
logical join builder's convention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update to Substrait v0.91.0

1 participant