Spark: Apply extra snapshot properties to DELETE operations#17016
Open
yadavay-amzn wants to merge 1 commit into
Open
Spark: Apply extra snapshot properties to DELETE operations#17016yadavay-amzn wants to merge 1 commit into
yadavay-amzn wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Propagates the configured extra snapshot properties (
SparkWriteConf.extraSnapshotMetadata()) to theDeleteFilesoperation inSparkTable.deleteWhere, mirroring the write path inSparkWrite, so metadata-only DELETEs carry the user'ssnapshot-property.*session-config metadata in the resulting snapshot summary (previously they were dropped).Applied to Spark v3.5, v4.0, and v4.1. For v3.5,
SparkWriteConf.extraSnapshotMetadata()is also brought to parity with v4.0/v4.1 (reads session config in addition to write options), sincedeleteWherehas no write-options injection point and session config is the only mechanism available to inject snapshot properties on a metadata-only delete.Tests: per-version cases set a
snapshot-property.*session config, run a partition-aligned metadata-only DELETE, and assert the property appears in the snapshot summary.Closes #15060.
AI assistance: authored with assistance by Claude Opus 4.8. I understand the change end-to-end and have verified it locally (per-version tests green; spotless and checkstyle clean).