[SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer - #57607
Closed
viirya wants to merge 1 commit into
Closed
[SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer#57607viirya wants to merge 1 commit into
viirya wants to merge 1 commit into
Conversation
Since SPARK-57735, cached-batch statistics rows carry TimestampNanosVal min/max bounds for nanosecond-timestamp columns (TimestampNanosColumnStats in the default cache serializer, and the Arrow cache serializer's vector-side statistics). Persisting such a cache with a serialized storage level under spark.serializer=KryoSerializer and spark.kryo.registrationRequired=true failed at cache materialization with "Class is not registered: org.apache.spark.unsafe.types.TimestampNanosVal", because the class was never added to the registration list. Register it next to UTF8String, which reached the list the same way (SPARK-51790). Co-authored-by: Claude Code
viirya
added a commit
that referenced
this pull request
Jul 28, 2026
### What changes were proposed in this pull request? Register `org.apache.spark.unsafe.types.TimestampNanosVal` in `KryoSerializer`, next to `UTF8String` which reached the registration list the same way (SPARK-51790). ### Why are the changes needed? Since SPARK-57735, cached-batch statistics rows carry `TimestampNanosVal` min/max bounds for nanosecond-timestamp columns -- `TimestampNanosColumnStats` in the default cache serializer, and the Arrow cache serializer's vector-side statistics (SPARK-57268). The statistics `InternalRow` is a field of `DefaultCachedBatch` / `ArrowCachedBatch`, so it is serialized with the batch whenever the cache uses a serialized storage level. With `spark.serializer=KryoSerializer` and `spark.kryo.registrationRequired=true`, materializing such a cache fails at the first block write: ``` com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.unsafe.types.TimestampNanosVal ``` This is the same defect class as SPARK-51777 (`CachedBatch` classes) and SPARK-51790 (`UTF8String`): a new class reachable from the cached-batch object graph was introduced without a matching Kryo registration. Every other bound type in statistics rows (`UTF8String`, `Decimal`, primitives) is already registered; `TimestampNanosVal` was the only gap -- the remaining stats collectors for non-orderable types (Variant, CalendarInterval, Geometry) record only sizes, no value objects. Users with the default `registrationRequired=false` are unaffected (Kryo falls back to writing the class name). Nanosecond timestamp types are an unreleased 4.3.0 preview feature. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New test in `CacheTableInKryoSuite` (the suite from SPARK-51777/51790, which runs with `registrationRequired=true`): persists a `TIMESTAMP_NTZ(9)` column with `DISK_ONLY`. It fails without the registration with the exception above and passes with it. `CacheTableInKryoSuite` (4 tests) and `core`'s `*KryoSerializer*` suites (81 tests) pass. ### Was this patch authored or co-authored using generative AI tooling? Yes, this pull request and its description were written by Claude Code. Closes #57607 from viirya/kryo-register-timestamp-nanos. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 177ce56) Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
Member
Author
Member
Author
|
Thanks @dongjoon-hyun |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Register
org.apache.spark.unsafe.types.TimestampNanosValinKryoSerializer, next toUTF8Stringwhich reached the registration list the same way (SPARK-51790).Why are the changes needed?
Since SPARK-57735, cached-batch statistics rows carry
TimestampNanosValmin/max bounds for nanosecond-timestamp columns --TimestampNanosColumnStatsin the default cache serializer, and the Arrow cache serializer's vector-side statistics (SPARK-57268). The statisticsInternalRowis a field ofDefaultCachedBatch/ArrowCachedBatch, so it is serialized with the batch whenever the cache uses a serialized storage level.With
spark.serializer=KryoSerializerandspark.kryo.registrationRequired=true, materializing such a cache fails at the first block write:This is the same defect class as SPARK-51777 (
CachedBatchclasses) and SPARK-51790 (UTF8String): a new class reachable from the cached-batch object graph was introduced without a matching Kryo registration. Every other bound type in statistics rows (UTF8String,Decimal, primitives) is already registered;TimestampNanosValwas the only gap -- the remaining stats collectors for non-orderable types (Variant, CalendarInterval, Geometry) record only sizes, no value objects.Users with the default
registrationRequired=falseare unaffected (Kryo falls back to writing the class name). Nanosecond timestamp types are an unreleased 4.3.0 preview feature.Does this PR introduce any user-facing change?
No.
How was this patch tested?
New test in
CacheTableInKryoSuite(the suite from SPARK-51777/51790, which runs withregistrationRequired=true): persists aTIMESTAMP_NTZ(9)column withDISK_ONLY. It fails without the registration with the exception above and passes with it.CacheTableInKryoSuite(4 tests) andcore's*KryoSerializer*suites (81 tests) pass.Was this patch authored or co-authored using generative AI tooling?
Yes, this pull request and its description were written by Claude Code.