[SPARK-58414][SQL][TESTS] Add e2e coverage for nanosecond timestamps nested in complex types in the Arrow cache - #57619
Open
viirya wants to merge 1 commit into
Open
Conversation
…nested in complex types in the Arrow cache The Arrow cache suite covers top-level nanosecond timestamps and nested CalendarInterval, but had no round trip for nanosecond timestamps nested in complex types. The nested read path relies on every container accessor in ArrowColumnVector wrapping its element vector through the constructor that runs the tagged-struct recognizers, and the write path on ArrowWriter's recursive field-writer dispatch -- pin that machinery with array, struct, and map-value round trips, using a value outside the int64 epoch-nanos window that the standard interchange encoding cannot represent. Co-authored-by: Claude Code
uros-b
approved these changes
Jul 29, 2026
Member
|
Thank you @viirya! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a test to
ArrowCachedBatchSerializerSuitethat round-trips nanosecond timestamps nested inside an array, a struct field, and a map value through the Arrow cache, using a value outside the int64 epoch-nanos window (year 3000 at nanosecond precision) that the standard interchange encoding cannot represent, alongside an in-window value.Why are the changes needed?
The suite covers top-level nanosecond timestamps and nested
CalendarInterval, but had no round trip for nanosecond timestamps nested in complex types. The nested path relies on recursion in two places:ArrowWriter's field writers dispatch on(type, vector)recursively on the write side, and on the read side every container accessor inArrowColumnVector(struct children,ArrayAccessor's data vector,MapAccessor's keys/values) wraps its element vector through the constructor that runs the lossless tagged-struct recognizers. A regression at any nesting level would silently decode wrong values, so the machinery deserves an end-to-end pin for the one lossless type family that has out-of-window domain values.Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
The new test; full
ArrowCachedBatchSerializerSuiteandArrowCachedBatchKryoRegistrationSuitepass (76 tests).Was this patch authored or co-authored using generative AI tooling?
Yes, this pull request and its description were written by Claude Code.