Skip to content

to_timestamp_* discards the timezone of already-timezone-aware inputs #23841

Description

@nevi-me

Describe the bug

Since #19078, the to_timestamp / to_timestamp_seconds / to_timestamp_millis / to_timestamp_micros / to_timestamp_nanos functions override the timezone of an input that is already timezone-aware with the session's datafusion.execution.time_zone.

When that config is unset (the default None), an input of type Timestamp(_, Some(tz)) is returned as Timestamp(_, None). The zone annotation is dropped while the underlying epoch is kept, which silently changes the wall-clock meaning of the value.

Previously, these functions had a dedicated match arm that preserved a zoned input's timezone:

// before #19078
Timestamp(_, Some(tz)) => args[0].cast_to(&Timestamp(Millisecond, Some(tz)), None),

#19078 collapsed that arm into the generic one, which casts to the execution timezone:

// after #19078 (current main)
Null | Int32 | Int64 | Timestamp(_, _) => args[0].cast_to(&Timestamp(Millisecond, self.timezone.clone()), None),

The PR's rationale when I read it, only concerns naive timestamp strings being interpreted in the execution timezone, which is reasonable. Dropping the timezone of inputs that are already zoned appears to be an unintended side effectas it makes a normalisation function silently discard a caller's explicit timezone.

To Reproduce

Run in datafusion-cli with default settings (no SET datafusion.execution.time_zone):

-- input epoch = 2024-01-01T00:00:00Z, annotated +05:00 => local wall clock 05:00
SELECT
  arrow_typeof(to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS result_type,
  date_part('hour', arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))')) AS hour_direct,
  date_part('hour', to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS hour_via_to_timestamp;

Actual

result_type hour_direct hour_via_to_timestamp
Timestamp(Millisecond, None) 5 0

Befor the change

result_type hour_direct hour_via_to_timestamp
Timestamp(Millisecond, Some("+05:00")) 5 5

Expected behavior

to_timestamp_millis (and the rest of the family) should not silently drop the timezone of an already-zoned input. Passing a Timestamp(_, Some("+05:00")) through the function should either:

  1. preserve the input zone, or
  2. if honoring datafusion.execution.time_zone, convert the timezone but never relabel to a naive type, which changes the wall-clock value.

In particular, when datafusion.execution.time_zone is unset (None), a zoned input should keep its zone rather than becoming naive. As it stands, date_part('hour', to_timestamp_millis(<zoned column>)) returns the UTC hour instead of the local hour, which corrupts any hour-of-day / day-of-month bucketing over zoned columns.

Additional context

Relevant commit: ada0923a3927d16dc5d810637cfbea4146c54f54 (PR #19078 that closed #17998)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions