Skip to content

perf: remove per-row String allocations from the Spark url functions#23884

Open
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:opt/spark-url-encode
Open

perf: remove per-row String allocations from the Spark url functions#23884
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:opt/spark-url-encode

Conversation

@andygrove

@andygrove andygrove commented Jul 25, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

Three allocation problems in the url module, all on the per-row path.

url_encode built a String for every row:

fn encode(value: &str) -> Result<String> {
    Ok(byte_serialize(value.as_bytes()).collect::<String>())
}

url_decode ended with .map(|parsed| parsed.into_owned()). Both
replace_plus and decode_utf8 return a Cow that borrows when there is
nothing to rewrite, so into_owned allocated a String for every row even when
the value contained no percent-escapes and no + at all.

parse_url built the all-null key array for the two-argument form by
calling append_null() once per row against a builder with no capacity.

What changes are included in this PR?

url_encode.rs:

  • The three type branches build a pre-sized builder and share an encode_all!
    loop that clears and refills one scratch String per batch.
  • UrlEncode::encode is removed; it returned a Result whose error arm was
    never constructed and now has no callers.

url_decode.rs:

  • decode returns Cow<'_, str> instead of String. When replace_plus
    borrows, the decode borrows straight from the input; when it has already
    allocated (the input contained +), owning the decoded form costs nothing
    beyond what was already spent.
  • The array paths append the Cow to a pre-sized builder rather than
    collecting Result<StringArray> from a per-row String.
  • API change: spark_handled_url_decode's second parameter changes from
    impl Fn(Result<Option<String>>) -> Result<Option<String>> to a new
    OnDecodeError enum. The String in that signature is what forced the
    allocation. Its only caller is try_url_decode, whose closure was
    Err(_) => Ok(None) — exactly OnDecodeError::Null.

parse_url.rs:

  • The append_null()-per-row loop becomes new_null_array(&DataType::Utf8, len).

No behaviour change in any of the three.

Are these changes tested?

Existing coverage pins the behaviour: spark/url/url_encode.slt,
url_decode.slt, try_url_decode.slt, and parse_url.slt assert concrete
results including reserved characters, + handling, malformed percent-encoding
(which must error for url_decode and yield NULL for try_url_decode), and the
two- versus three-argument parse_url forms. All 5 spark/url sqllogictest
files pass, along with the 258 datafusion-spark unit tests.

The try_url_decode unit test is what pins the OnDecodeError::Null path,
since it drives a malformed input through the refactored signature.

Benchmarks are added separately in #23882 so the baselines can be measured on
main before this lands.

Benchmarks

Criterion, apache/main @ f1ab86dad as baseline. Median of the reported
change interval.

url_encode:

Benchmark 1024 8192
url_encode/utf8 −57.1% −48.2%
url_encode/largeutf8 −56.1% −49.7%
url_encode/utf8view −56.0% −46.5%

url_decode, split by whether the input actually needs unescaping:

Benchmark 1024 8192
url_decode/plain_utf8 −26.4% −28.8%
url_decode/plain_utf8view −29.0% −27.2%
url_decode/escaped_utf8 −5.4% −3.6%
url_decode/escaped_largeutf8 −4.7% −3.2%

The plain rows are the ones that benefit: with nothing to unescape the decoded
value borrows its input. The escaped rows still have to allocate, so they gain
only the pre-sized builder and the removed intermediate — a few percent, as
expected.

parse_url has no benchmark; its change removes an append_null() loop that
runs once per batch rather than affecting a measured per-row path.

Are there any user-facing changes?

No change to SQL behaviour — all three functions return identical results.

spark_handled_url_decode is public Rust API and its signature changes as
described above; OnDecodeError is new public API on datafusion-spark.

url_encode built a String for every row via byte_serialize(..).collect::<String>()
and collected the results into a StringArray, so each row paid for its own
allocation.

The percent-encoded form is now assembled in a single scratch buffer reused
across rows and appended to a pre-sized builder. The now-unused UrlEncode::encode
helper, which returned a Result that was never an Err, is removed.

-56% at 1024 rows and -48% at 8192 against the benchmark added in apache#23882.
@codecov-commenter

codecov-commenter commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.92308% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.65%. Comparing base (f1ab86d) to head (3b3a748).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/spark/src/function/url/url_decode.rs 77.27% 0 Missing and 5 partials ⚠️
datafusion/spark/src/function/url/url_encode.rs 72.72% 0 Missing and 3 partials ⚠️
datafusion/spark/src/function/url/parse_url.rs 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23884      +/-   ##
==========================================
- Coverage   80.65%   80.65%   -0.01%     
==========================================
  Files        1091     1092       +1     
  Lines      371031   371089      +58     
  Branches   371031   371089      +58     
==========================================
+ Hits       299256   299293      +37     
- Misses      53935    53946      +11     
- Partials    17840    17850      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…key array directly

url_decode ended with .map(|parsed| parsed.into_owned()). Both replace_plus
and decode_utf8 borrow their input when there is nothing to rewrite, so that
allocated a String for every row even when the value needed no decoding at
all. decode now returns a Cow and the array paths append it to a pre-sized
builder.

To let the Cow survive, spark_handled_url_decode's error handler changes from
a closure over Result<Option<String>> to an OnDecodeError enum. Its only
caller, try_url_decode, passed a closure that mapped Err to Ok(None), which is
exactly OnDecodeError::Null.

parse_url built its all-null key array for the 2-argument form by calling
append_null() once per row on an uncapacitated builder; new_null_array does it
in one allocation.

url_decode -29% where nothing needs unescaping, -4% where it does, against the
benchmark added in apache#23882.
@andygrove andygrove changed the title perf: remove per-row String allocation from Spark url_encode perf: remove per-row String allocations from the Spark url functions Jul 25, 2026
@andygrove andygrove added the api change Changes the API exposed to users of the crate label Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate spark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants