[SPARK-58413][SQL][TESTS] Rename sequential fetch label to pipelined fetch (1 client) in NettyTransportBenchmark - #57616
Open
dongjoon-hyun wants to merge 1 commit into
Open
Conversation
…ed fetch (1 client)` in `NettyTransportBenchmark`
Member
Author
|
cc @LuciferYang |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR renames the benchmark case label
sequential fetchtopipelined fetch (1 client)in theFile-Backed Shuffle Block Fetchsuite ofNettyTransportBenchmark.Why are the changes needed?
The label is misleading. The case fires all 100
fetchChunkrequests at once on a single connection and then waits for all of them to complete via a semaphore (fetchChunksSync), so the client never waits for one chunk before requesting the next. This is a pipelined fetch over one connection, not a sequential request-response loop.The mislabel skews the interpretation of the results. For example:
spark/core/benchmarks/NettyTransportBenchmark-results.txt
Lines 132 to 133 in 12785d5
Read as "sequential vs parallel", the 1.8X looks like poor parallelization efficiency. The actual comparison is "pipelined over 1 socket vs pipelined over 4 sockets", where the single pipelined connection already runs at ~4.2 GB/s. The new label makes this clear and contrasts naturally with
parallel fetch (4 clients).Does this PR introduce any user-facing change?
No.
How was this patch tested?
Label-only change; manually reviewed.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5