feat: expose residual_expression and post_join_filter on join builders#225
Open
nielspardon wants to merge 1 commit into
Open
feat: expose residual_expression and post_join_filter on join builders#225nielspardon wants to merge 1 commit into
nielspardon wants to merge 1 commit into
Conversation
Surface the two Expression fields spec #1044 added to HashJoinRel and MergeJoinRel — residual_expression and post_join_filter — on the ergonomic physical-join builders (hash_join / merge_join) and the DataFrame hash_join / merge_join methods, completing the remaining work for substrait-io#190. Bind each expression against the schema the spec prescribes: - residual_expression is evaluated on each candidate key-match (both rows present), so it resolves against the combined left+right schema. - post_join_filter is applied to each output record after join-type-specific output formation ("semantically a FilterRel directly above this join"), so it resolves against the join output schema, reusing the existing _join_output_struct / join_output_names helpers. The same fix is applied to the logical join builder, which previously bound post_join_filter against the combined schema. For inner/outer/left/right/ single/mark joins the output equals (or prefixes) the combined schema so behavior is unchanged; for right_semi / right_anti (output is the right side only) a filter on a right column now uses the correct output-relative field index instead of a dangling combined-schema index, and a filter on a dropped side fails fast instead of emitting an invalid reference. The optional predicate/extension arguments are keyword-only, matching the logical join builder's convention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the remaining work for #190: surface the two
Expressionfields spec #1044 added toHashJoinRelandMergeJoinRel—residual_expressionandpost_join_filter— on the ergonomic physical-join builders (hash_join/merge_join) and the correspondingDataFramemethods. They were previously only settable on the raw proto.Binding semantics
Each expression is resolved against the schema the spec prescribes:
residual_expressionis "evaluated on each candidate key-match" (both rows present), so it binds against the combined left+right schema.post_join_filteris "applied to each output record after join-type-specific output formation... semantically equivalent to placing a FilterRel directly above this join", so it binds against the join output schema, reusing the existing_join_output_struct/join_output_nameshelpers.Also fixes the logical
joinbuilderThe logical
joinbuilder previously boundpost_join_filteragainst the combined schema too. Forinner/outer/left/right/single/markjoins the output equals (or prefixes) the combined schema, so behavior is unchanged. Forright_semi/right_anti(output is the right side only) a filter on a right column now uses the correct output-relative field index instead of a dangling combined-schema index, and a filter referencing a dropped side fails fast instead of emitting an invalid reference.Notes
joinbuilder's convention.right_semioutput binding,residual_expressioncombined binding, fail-fast on a dropped-side reference, keyword-only enforcement, and byte-identical DataFrame↔builder parity acrossinner/right_semi.Closes #190
🤖 Generated with AI