perf: avoid per-row String allocation in Spark bin and char#23881
Merged
Conversation
bin formatted every row with format!("{value:b}"), and char built a String
for a single character with ch.to_string(), before collecting into a
StringArray.
Both render into a stack buffer instead: bin writes its digits right-aligned
in a [u8; 64] and char uses char::encode_utf8 into a [u8; 4]. Values are
appended to a pre-sized StringBuilder rather than collected.
bin -74% on small values and -50% on full-width ones, char -76%. The bin
benchmark is added in apache#23882.
andygrove
force-pushed
the
opt/spark-bin-char
branch
from
July 25, 2026 14:08
033cd58 to
8c59488
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23881 +/- ##
==========================================
- Coverage 80.65% 80.65% -0.01%
==========================================
Files 1091 1091
Lines 371031 371043 +12
Branches 371031 371043 +12
==========================================
- Hits 299256 299254 -2
- Misses 53935 53941 +6
- Partials 17840 17848 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Two Spark functions allocated a
Stringper row purely to render a small,bounded amount of text:
bincalledformat!("{value:b}")for every row.charcalledch.to_string()for every row — a heap allocation for a singlecharacter.
In both cases the output has a known upper bound (64 binary digits for an
i64,4 bytes for a UTF-8 character), so the rendering fits in a stack buffer and the
result can be appended straight to a pre-sized builder.
What changes are included in this PR?
math/bin.rs:spark_binnow writes digits right-aligned into a caller-supplied[u8; 64]and returns a
&strborrowed from it, instead of returning an ownedString.collect::<StringArray>()becomes an explicit loop over aStringBuilder::with_capacity, sized at 8 digits per row.{:b}. The digit loop is aloop, not awhile, so zero renders as"0"rather than the empty string.
string/char.rs:ch.to_string()becomesch.encode_utf8(&mut encoded)against a[u8; 4]hoisted out of the loop.
Output is unchanged in both cases.
Are these changes tested?
Existing coverage pins the behaviour.
spark/math/bin.sltasserts concreteoutput for the cases the rewrite had to get right: zero, negative values,
i64::MIN(-9223372036854775808),i64::MAX, and-2147483648/-32768widened from narrower integer types.spark/string/char.sltcoversthe negative-input empty string, the null path, and characters on both sides of
the ASCII boundary (
char(256)and above wrap via% 256).All 119
spark/mathandspark/stringsqllogictest files pass, along with the258
datafusion-sparkunit tests.The
binbenchmark used below,datafusion/spark/benches/bin.rs, is addedseparately in #23882 so the baseline can be measured on
mainbefore thischange lands. It covers 1024 and 8192 rows with 20% nulls over two value
distributions: small values that render to a handful of digits, and full-range
values that render to the maximum 64.
charalready haddatafusion/spark/benches/char.rsonmain, so no benchmark change is neededfor it.
Benchmarks
Criterion,
apache/main@f1ab86dadas baseline. Median of the reportedchange interval.
bin/smallbin/widechar(1024 rows): −76.1%.Are there any user-facing changes?
No. Both functions produce byte-identical output; this is purely an allocation
change.