Skip to content

PERF: speed up strftime for common format directives#64609

Merged
mroeschke merged 2 commits intopandas-dev:mainfrom
jbrockmendel:perf-strftime
Apr 16, 2026
Merged

PERF: speed up strftime for common format directives#64609
mroeschke merged 2 commits intopandas-dev:mainfrom
jbrockmendel:perf-strftime

Conversation

@jbrockmendel
Copy link
Copy Markdown
Member

@jbrockmendel jbrockmendel commented Mar 15, 2026

Summary

  • For formats composed of common directives (%Y, %m, %d, %H, %M, %S, %f), parse the format string once into a str.format() template and format directly from npy_datetimestruct fields, avoiding per-element Timestamp creation.
  • For tz-aware data, converts UTC to local time via Localizer.utc_val_to_local_val before extracting struct fields (works because _convert_strftime_format returns None for formats containing %z/%Z).
  • Unsupported directives (%A, %B, %z, %Z, etc.) fall back to the existing per-element Timestamp.strftime() path.

Benchmarks (10.5M elements, %Y/%m/%dT%H:%M:%S)

Scenario Before After Speedup
tz-naive ~28s 6.7s 4.2x
UTC ~28s 6.7s 4.2x
US/Eastern ~28s 9.9s 2.8x

Test plan

  • All 144 existing strftime tests pass
  • IO format tests pass (2295 passed)
  • Correctness verified against per-element Timestamp.strftime() for multiple formats
  • Verified NaT handling (tz-naive and tz-aware)
  • Verified fallback for unsupported directives (%A, %Z)
  • Verified tz-aware correctness for UTC, US/Eastern (DST transitions), fixed-offset

🤖 Generated with Claude Code

@jbrockmendel jbrockmendel added the Performance Memory or execution speed performance label Mar 15, 2026
@jbrockmendel jbrockmendel force-pushed the perf-strftime branch 2 times, most recently from 2ddad72 to 762c01c Compare March 16, 2026 18:43
@jbrockmendel jbrockmendel marked this pull request as ready for review March 16, 2026 19:09
@jbrockmendel
Copy link
Copy Markdown
Member Author

Possible follow-up: C snprintf path

If we need more speed, there's a straightforward follow-up that gets the custom-format path close to the hardcoded f-string performance. Instead of str.format(), use C snprintf with POSIX positional specifiers (%1$ld, %2$02ld, etc.) to format the npy_datetimestruct fields directly into a stack buffer, then decode to a Python string. This avoids all Python method dispatch overhead in the inner loop.

On my machine (10.5M elements, %Y/%m/%dT%H:%M:%S):

  • Current .format() path: 6.7s
  • With snprintf: 2.8s
  • Hardcoded f-string path: 1.8s

The implementation is simple — always pass all 7 struct fields in fixed order, and the template references them by position — but POSIX positional specifiers aren't available on Windows, so it would need a runtime check (_check_posix_snprintf()) and fall back to .format() on Windows. I had this working in an earlier iteration of this PR but stripped it out to keep the diff reviewable.

@jbrockmendel jbrockmendel force-pushed the perf-strftime branch 2 times, most recently from 6457039 to 2dc41a2 Compare March 19, 2026 14:43
@jbrockmendel jbrockmendel force-pushed the perf-strftime branch 2 times, most recently from d7dd9db to 6836ab8 Compare April 13, 2026 22:28
Comment thread pandas/_libs/tslib.pyx
return None
result.append(replacement)
i += 2
elif fmt[i] in ("{", "}"):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We support replacement brackets in a format string like `format="%Y-{1:02d}"? Do we have tests for this case?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will add a test

jbrockmendel and others added 2 commits April 16, 2026 08:09
For formats composed of common directives (%Y, %m, %d, %H, %M, %S, %f),
parse the format string once into a str.format() template and format
directly from npy_datetimestruct fields, avoiding per-element Timestamp
creation. For tz-aware data, converts UTC to local time via Localizer
before extracting struct fields.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mroeschke mroeschke added this to the 3.1 milestone Apr 16, 2026
@mroeschke mroeschke merged commit a94ce90 into pandas-dev:main Apr 16, 2026
45 checks passed
@mroeschke
Copy link
Copy Markdown
Member

Thanks @jbrockmendel

@jbrockmendel jbrockmendel deleted the perf-strftime branch April 16, 2026 16:32
@smarie smarie mentioned this pull request Apr 29, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Performance Memory or execution speed performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PERF: strftime is slow

2 participants