[feature](timestamp_ns) Add basic TIMESTAMP_NS type support - #66591
Open
jacktengg wants to merge 2 commits into
Open
[feature](timestamp_ns) Add basic TIMESTAMP_NS type support#66591jacktengg wants to merge 2 commits into
jacktengg wants to merge 2 commits into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#66333 Problem Summary: Doris DATETIME and DATETIMEV2 support at most six fractional-second digits. Extending their existing packed civil-time representation to nanoseconds would mix incompatible storage representations and semantics. Introduce TIMESTAMP_NS as a separate, fixed-precision timestamp type: - TIMESTAMP_NS always has nine fractional-second digits and does not accept a precision parameter. - Values are stored as signed Int64 nanoseconds relative to the Unix epoch. - The supported range is [1677-09-21 00:12:43.145224192, 2262-04-11 23:47:16.854775807]. - TIMESTAMP_NS has no time-zone attribute. UTC is used as the stable mapping between its civil-time fields and physical epoch-nanosecond representation. - DATETIME(p) and DATETIMEV2(p) retain their existing behavior and continue to support only precision 0 through 6. Add the new primitive type to Thrift, protobuf, FE type metadata, Nereids, BE data types, and cloud schema metadata. Add dedicated TIMESTAMP_NS literal, value, column, and SerDe implementations. Support the basic end-to-end SQL and storage paths, including: - DDL, literals, default values, partitions, buckets, and schema metadata - Insert, stream load, partial update, and decoded timestamp input - Segment storage, key coding, zone map, bloom filter, and inverted index - Comparison, predicates, hashing, grouping, ordering, joins, runtime filters, and basic aggregate functions - FE literal validation, constant folding, type coercion, and predicate simplification - MySQL text and prepared-statement binary results, using string-compatible output to preserve all nine fractional digits Arrow Flight SQL output, ORC OUTFILE, and Java UDF signatures explicitly reject TIMESTAMP_NS because those integrations are not supported yet. The broader explicit cast matrix and scalar date/time function signatures will be implemented in follow-up changes. ### Release note Add the fixed-precision TIMESTAMP_NS SQL type for nanosecond timestamps. TIMESTAMP_NS stores signed Int64 Unix-epoch nanoseconds, supports the range from 1677-09-21 00:12:43.145224192 through 2262-04-11 23:47:16.854775807, and always displays nine fractional digits. Existing DATETIME and DATETIMEV2 types remain limited to precision 0 through 6. ### Check List (For Author) - Test: Unit Test and Regression Test - Added BE unit tests for TIMESTAMP_NS values, SerDe, storage indexes, hashing, predicates, aggregates, and partitioning - Added FE unit tests for types, literals, boundaries, partitions, coercion, folding, MySQL output, and unsupported integrations - Added regression tests for DDL, predicates, storage, stream load, partial/default values, and MySQL prepared-statement output - Behavior changed: Yes; adds the TIMESTAMP_NS SQL and storage type without changing DATETIME or DATETIMEV2 precision semantics - Does this need documentation: Yes; TIMESTAMP_NS syntax, range, storage semantics, and current integration limitations require documentation
### What problem does this PR solve? Issue Number: None Related PR: apache#66333 Problem Summary: The initial TIMESTAMP_NS implementation mixed the signed epoch-nanosecond type into DATETIMEV2 precision families and date/time function and cast paths. This made legacy datetime literals retain nanosecond-only state, advertised unsupported output and Java-function surfaces, and left inconsistent coercion and primitive classification. Keep TIMESTAMP_NS as an independent fixed-precision type, limit basic support to storage, serialization, literals, predicates, string conversion, and MySQL output, explicitly reject deferred Arrow/ORC/function/cross-temporal cast paths, and remove unused TimeStampNsValue APIs. ### Release note Add the basic fixed-precision TIMESTAMP_NS type backed by signed Int64 epoch nanoseconds. Cross-date/time casts and date/time functions remain unsupported in this PR. ### Check List (For Author) - Test: Unit Test and Regression test - Standard ASAN BE and FE builds passed - BE ASAN UT: 201 broad tests passed; focused reruns of 21 and 19 tests passed - FE UT: 495 related tests passed, then the 19-test connector rerun passed after the fix - Regression tests: test_timestamp_ns, test_timestamp_ns_binary_output, test_timestamp_ns_ddl_predicate, and test_timestamp_ns_stream_load passed - Behavior changed: Yes. TIMESTAMP_NS remains independent from DATETIMEV2 and unsupported follow-up surfaces fail explicitly. - Does this need documentation: No
jacktengg
requested review from
924060929,
airborne12,
csun5285,
eldenmoon,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde,
starocean999 and
yiguolei
as code owners
August 9, 2026 18:59
Contributor
Author
|
run buildall |
2 similar comments
Contributor
Author
|
run buildall |
Contributor
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29208 ms |
Contributor
TPC-DS: Total hot run time: 159258 ms |
Contributor
ClickBench: Total hot run time: 23.77 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #66333
Problem Summary: Add an independent fixed-precision TIMESTAMP_NS type backed by signed Int64 nanoseconds from the Unix epoch. The basic implementation covers FE/BE type metadata, literals, storage, serialization, predicates, string input, constant folding, MySQL text and prepared-statement output, and cloud schema persistence. It deliberately keeps TIMESTAMP_NS outside the DATETIMEV2 precision family and rejects deferred Arrow/ORC output, Java UDF support, cross-temporal casts, and date/time function signatures.
Release note
Add basic TIMESTAMP_NS type support with nanosecond precision and the range 1677-09-21 00:12:43.145224192 through 2262-04-11 23:47:16.854775807.
Check List (For Author)