[SPARK-58309][SS] Anchor now() and current_time() to the batch timestamp in streaming queries#57480
Open
PorridgeSwim wants to merge 1 commit into
Open
Conversation
…amp in streaming queries ### What changes were proposed in this pull request? In `MicroBatchExecution`, the per-batch rewrite that replaces `CURRENT_LIKE` temporal expressions with `CurrentBatchTimestamp` only matched `CurrentTimestamp`, `CurrentDate` and `LocalTimestamp`. Two expressions carrying the `CURRENT_LIKE` tree pattern were missed: - `Now` (`now()`) -- a sibling of `CurrentTimestamp` under the `CurrentTimestampLike` base class. The match on the leaf type `CurrentTimestamp` did not catch it. - `CurrentTime` (`current_time()` / `localtime()`, a `TIME` value added in 4.x). This PR: - Widens the `CurrentTimestamp` case to `CurrentTimestampLike`, covering both `current_timestamp()` and `now()`. - Adds a `CurrentTime` case, and a `TimeType` branch to `CurrentBatchTimestamp.toLiteral` that derives the time-of-day (nanoseconds since midnight, in the session time zone) from the batch timestamp, truncated to the type's precision. ### Why are the changes needed? Expressions left unrewritten survive to the per-batch optimizer, where `ComputeCurrentTime` folds them using the wall clock at plan-optimization time. That value is not reproducible when a batch is replayed after failure or on restart from a checkpoint, breaking the determinism guarantee that `CurrentBatchTimestamp` (anchored to the batch timestamp persisted in the offset log) exists to provide. `now()` and `current_time()` were therefore non-deterministic across batch replays. ### Does this PR introduce _any_ user-facing change? Yes. In a streaming query, `now()` and `current_time()` now return the batch timestamp (stable across replay/restart of the same batch), consistent with `current_timestamp()`, `current_date()` and `localtimestamp()`, instead of the wall clock at plan-optimization time. ### How was this patch tested? Added a parameterized test in `StreamSuite` covering `current_timestamp()`, `now()`, `current_date()` and `current_time()`. Using a manual clock it asserts the exact per-batch value, then purges the last batch's commit and restarts with the wall clock advanced far ahead, asserting the replayed batch reproduces its persisted timestamp rather than the advanced clock. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Co-authored-by: Isaac
zikangh
reviewed
Jul 23, 2026
| case _: TimestampNTZType => | ||
| Literal(convertTz(timestampUs, ZoneOffset.UTC, zoneId), TimestampNTZType) | ||
| case _: DateType => Literal(microsToDays(timestampUs, zoneId), DateType) | ||
| case t: TimeType => |
Contributor
There was a problem hiding this comment.
Please guard this behind a flag. also we should think through if this is a breaking change or behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In
MicroBatchExecution, the per-batch rewrite that replacesCURRENT_LIKEtemporal expressions withCurrentBatchTimestamponly matchedCurrentTimestamp,CurrentDateandLocalTimestamp. Two expressions carrying theCURRENT_LIKEtree pattern were missed:Now(now()) -- a sibling ofCurrentTimestampunder theCurrentTimestampLikebase class. The match on the leaf typeCurrentTimestampdid not catch it.CurrentTime(current_time()/localtime(), theTIMEvalue type).This PR:
CurrentTimestampcase toCurrentTimestampLike, covering bothcurrent_timestamp()andnow().CurrentTimecase, and aTimeTypebranch toCurrentBatchTimestamp.toLiteralthat derives the time-of-day (nanoseconds since midnight, in the session time zone) from the batch timestamp, truncated to the type's precision. (Because the batch timestamp is millisecond-resolution, sub-millisecond digits ofcurrent_time()are zero in streaming.)Why are the changes needed?
Expressions left unrewritten survive to the per-batch optimizer, where
ComputeCurrentTimefolds them using the wall clock at plan-optimization time. That value is not reproducible when a batch is replayed after failure or on restart from a checkpoint, breaking the determinism guarantee thatCurrentBatchTimestamp(anchored to the batch timestamp persisted in the offset log) exists to provide.now()andcurrent_time()were therefore non-deterministic across batch replays.Does this PR introduce any user-facing change?
Yes. In a streaming query,
now()andcurrent_time()now return the batch timestamp (stable across replay/restart of the same batch), consistent withcurrent_timestamp(),current_date()andlocaltimestamp(), instead of the wall clock at plan-optimization time.How was this patch tested?
Added a parameterized test in
StreamSuitecoveringcurrent_timestamp(),now(),current_date()andcurrent_time(). Using a manual clock it asserts the exact per-batch value, then purges the last batch's commit and restarts with the wall clock advanced far ahead, asserting the replayed batch reproduces its persisted timestamp rather than the advanced clock.Was this patch authored or co-authored using generative AI tooling?
No