Skip to content

feat: add Spark date_from_unix_date function to datafusion-spark#23852

Closed
andygrove wants to merge 0 commit into
apache:mainfrom
andygrove:date-from-unix-date
Closed

feat: add Spark date_from_unix_date function to datafusion-spark#23852
andygrove wants to merge 0 commit into
apache:mainfrom
andygrove:date-from-unix-date

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of the datafusion-spark epic: #15914

Rationale for this change

date_from_unix_date is a Spark built-in that creates a date from the number of days since the Unix epoch (1970-01-01). It is the inverse of the existing unix_date function and is not yet available in the datafusion-spark crate.

What changes are included in this PR?

  • Adds SparkDateFromUnixDate under datafusion/spark/src/function/datetime/, registered in the datetime module. The implementation lowers to a single Int32 -> Date32 cast in simplify(), mirroring the existing unix_date idiom, since both types represent days since epoch.
  • Adds sqllogictest coverage under test_files/spark/datetime/date_from_unix_date.slt for scalar, array, negative, and NULL inputs.

Are these changes tested?

Yes, via sqllogictest with scalar and array inputs (including negative and NULL cases).

@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.00000% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.71%. Comparing base (ced2492) to head (f46403c).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
...spark/src/function/datetime/date_from_unix_date.rs 82.82% 10 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23852      +/-   ##
==========================================
- Coverage   80.74%   80.71%   -0.03%     
==========================================
  Files        1089     1091       +2     
  Lines      369207   369498     +291     
  Branches   369207   369498     +291     
==========================================
+ Hits       298102   298231     +129     
- Misses      53347    53485     +138     
- Partials    17758    17782      +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andygrove
andygrove marked this pull request as draft July 23, 2026 20:21
@andygrove

Copy link
Copy Markdown
Member Author

Leaving this as draft for now. I am assuming that the SLT tests will be using the simplify path and therefore is not testing the invoke_with_args path, although there are unit tests for that path. I wonder if we can have the SLT tests test both paths? Maybe we need a config to disable simplify?

@andygrove

Copy link
Copy Markdown
Member Author

Leaving this as draft for now. I am assuming that the SLT tests will be using the simplify path and therefore is not testing the invoke_with_args path, although there are unit tests for that path. I wonder if we can have the SLT tests test both paths? Maybe we need a config to disable simplify?

I updated the slt test to test both paths

@andygrove
andygrove marked this pull request as ready for review July 23, 2026 21:24
@andygrove
andygrove requested a review from comphead July 23, 2026 21:24
##########

statement ok
set datafusion.optimizer.max_passes = 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we prob need to document that we don't want simplify to be called.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I should just remove the simplify path from this PR, but my concern is that someone will later implement it and remove invoke_with_args and then Comet will no longer be able to leverage this work

}

#[cfg(test)]
mod tests {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the tests might be not needed as it should have already been covere by slt, however nice test would be to test simplify output and invoke_with_args_output are the same

@andygrove andygrove closed this Jul 25, 2026
@andygrove
andygrove force-pushed the date-from-unix-date branch from f46403c to f1ab86d Compare July 25, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants