storage: Fast approximate snapshot partitioning - #37994
Draft
peterdukelarsen wants to merge 1 commit into
Draft
Conversation
Replace the OFFSET-walking boundary discovery with a prefix-based partitioner in mz-mysql-util. Ranges the optimizer estimates too large are recursively subdivided at each distinct key prefix one character longer, then accumulated into per-worker buckets, so discovery costs EXPLAIN index dives instead of an O(rows) index pass. Only string primary keys are supported. Integer keys, which the OFFSET walk used to sample, now fall back to a single-worker whole-table read. Prefixes of a numeric key do not order consistently with its values, so they would need a separate numeric range splitter. All key ordering happens server-side under the column collation. The walk guards against non-advancing prefixes and caps children per split so a misbehaving server cannot hang it, and boundaries still pass the existing strict-monotonicity verification in each read transaction. The new mysql_source_snapshot_partition_min_rows dyncfg (default 50000) stops splitting below a minimum range size. Test configs set it low so the tiny tables in mysql-cdc testdrive and parallel-workload still exercise range reads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
peterdukelarsen
force-pushed
the
pl/mysql-snapshot-prefix-partition
branch
from
July 31, 2026 23:50
57d29d8 to
777e7e8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.