Replace the O(m·n) DP list diff with Myers' O((m+n)·D) algorithm#53
Merged
Conversation
diff_lists built a full Levenshtein-style DP table over the changed
region, so scattered edits in large lists paid quadratic time and
memory regardless of how much actually changed. The changed-region
diff is now Myers' greedy shortest-edit-script search (1986): cost
scales with the number of actual differences D, and the backtrack
trace keeps memory at O(D^2).
Patch semantics are unchanged:
- Myers scripts contain only insertions and deletions; consecutive
edits with no kept element in between are grouped into hunks, and
the k-th deletion pairs with the k-th insertion as a replace, so
paired containers still recurse into deep paths (all 25 pinned
exact-op expectations in the diff tests pass unmodified).
- The index-padding pass is factored into _pad_ops and shared by the
forward and reverse op streams.
- Prefix/suffix trimming is untouched.
Interleaved benchmark medians vs master (2 runs each, PYTHONHASHSEED=0,
GC off):
list_diff_medium[0.05] 185.1ms -> 492.1us (-99.7%)
list_diff_medium[0.1] 250.2ms -> 1.4ms (-99.5%)
list_diff_medium[0.5] 256.8ms -> 24.4ms (-90.6%)
list_diff_small_10pct 70.3us -> 13.0us (-81.5%)
completely_different 260.0ms -> 251.6ms (-3.4%)
identical (10k) 9.1us -> 9.1us (+0.0%)
localized/nested cases within noise (+0.2% .. +17% at the
smallest size, dominated by the deep-
equality prefix/suffix trim, well under
the 25% tripwire)
The internals docs (list diff section) are updated to describe the new
algorithm.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2
The benchmark guard caught the Myers worst case on CI (Python 3.14):
lists with almost nothing in common made the shortest-script search
quadratic, regressing completely_different by ~29% and the mixed
dict-of-lists workload by ~58% against the DP baseline.
Give up on minimality once D exceeds half the combined length - the
lists then share less than a quarter of their elements, and a shortest
script over the remainder benefits nobody. The whole region is emitted
as a single hunk, which the replace pairing turns into element-wise
replaces: exactly what the DP produced for such inputs. Regions
smaller than 128 combined elements are always solved exactly, so
pinned small-list outputs are unaffected.
The former regressions are now the biggest wins of the branch
(medians, this machine):
DP (master) Myers uncapped capped
completely_different py313 268.5ms 394.5ms 119.1ms
mixed_dict_with_lists py313 95.8ms 184.6ms 63.4ms
completely_different py311 260.0ms 251.6ms 76.0ms
Exact-search cases (medium change ratios, localized changes) are
untouched. Round-trip tests added for disjoint and mostly-different
large pairs; 407 tests at 100% coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2
Korijn
force-pushed
the
claude/myers-diff
branch
from
ddbf849 to
4aba07c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Track E, PR 9 (foundational) — the flagship list-diff optimization from the roadmap.
What
diff_listsbuilt a full Levenshtein-style DP table over the changed region, so scattered edits in large lists paid O(m·n) time and memory no matter how little actually changed. The changed-region diff is now Myers' greedy shortest-edit-script search (An O(ND) Difference Algorithm and Its Variations, 1986): cost scales with the number of actual differences D, with an O(D²) backtrack trace instead of the full table. Prefix/suffix trimming is untouched and still runs first.Worst-case cutoff (added after the benchmark guard caught the first version regressing on CI — see below): once D exceeds half the combined length, the lists share less than a quarter of their elements, and the search gives up on minimality — the whole region is emitted as one hunk of element-wise replaces, exactly what the DP produced for such inputs, at O(m+n) cost. Regions under 128 combined elements are always solved exactly, so all pinned small-list outputs are unaffected.
Patch semantics
_pad_opsused by both op streams.Benchmarks
The guard did its job: the first version of this PR tripped it on CI (py3.14:
completely_different+29%,mixed_dict_with_list_values+58% — the barely-common worst case, much worse on 3.13/3.14 than on the 3.11 I'd measured locally). The cutoff turns those into the branch's biggest wins. Medians:Docs
The Internals list-diff section is rewritten for the new algorithm, including the cutoff.
🤖 Generated with Claude Code
https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2