perf(paths): resolve the library root once, not on every path check (55x fewer stats) - #966
Merged
Merged
Conversation
`Path.resolve()` is a filesystem call — it lstats every component of the path.
`_resolve_dlc_path` and `safe_join` both re-resolved their ROOT on every single
call, and those run once per song, per art fetch, per scanned row.
Found while profiling a 2-fps report: on a real 50,944-song library the server
was issuing ~23,500 stat/lstat calls per second, re-walking the same three
parent directories over and over, and burning ~50% of a core doing it. It is
worst exactly where big libraries live — the library was on an NTFS-3G (FUSE)
mount, where every stat is a userspace round trip through mount.ntfs-3g (itself
visible in top). The cost was the constant re-resolution, not the work.
A root is fixed for the life of the process, so resolve it once
(safepath.resolved_root, lru_cache). Measured, 5,000 lookups against a real
library path:
before: 15,264 stat syscalls (54.2 ms)
after: 277 stat syscalls ( 0.8 ms) 55x fewer
Containment is unchanged, which is the part that matters:
- safe_join still resolves the CANDIDATE on every call — following its
symlinks IS the zip-slip / traversal defence, so it is never cached. Only
the server-owned root is.
- _resolve_dlc_path keeps its lexical containment check (deliberately does not
follow symlinks, so junction-mounted libraries keep working).
Tradeoff, documented on resolved_root: if a root's symlink is re-pointed at a
NEW target while the server runs, the old target stays in effect until restart.
Fine for a library path fixed at startup; the cache is keyed on the Path, so
switching library dir is a different key.
Tests: root resolved once across 500 lookups (the regression), a different root
is a different entry, and the containment contract re-pinned — traversal,
Windows drive-absolute, backslash, NUL, empty, and a symlink escaping the root
must still be refused. Full suite green.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds bounded caching for canonical server-owned roots in ChangesCached path resolution
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while profiling Byron's 2-fps report. Separate from the renderer bug (song-preview#7) — this one is the server side.
The bug
Path.resolve()is a filesystem call: it lstats every component of the path. Both_resolve_dlc_pathandsafe_joinre-resolved their root on every call — and those run once per song, per art fetch, per scanned row.On a real 50,944-song library,
straceon the running server caught it issuing ~23,500 stat/lstat calls per second, re-walking the same three parent directories forever:That 2:1:1 ratio is the parent-walk signature of
resolve(). It burned ~50% of a core.It is worst exactly where big libraries live: this one is on an NTFS-3G (FUSE) mount, so every stat is a userspace round trip through
mount.ntfs-3g(which was itself visible burning CPU intop). The cost was the constant **re-**resolution, not the work.The fix
A root is fixed for the life of the process. Resolve it once —
safepath.resolved_root, anlru_cached helper shared by both call sites.Measured, 5,000 lookups against a real library path:
55× fewer.
Containment is unchanged — the part that matters
safe_joinstill resolves the CANDIDATE on every call. Following the candidate's symlinks is the zip-slip / traversal defence, so it is never cached. Only the server-owned root is._resolve_dlc_pathkeeps its lexical containment check (it deliberately does not follow symlinks, so junction-mounted libraries keep working).Re-pinned in tests:
..traversal, backslash traversal, POSIX-absolute, Windows drive-absolute, embedded NUL, empty name — and a symlink escaping the root is still refused, which is the one that would catch a bad cache.Documented tradeoff
If a root's symlink/junction is re-pointed at a new target while the server is running, the old target stays in effect until restart. That is fine for a library path fixed at startup, and the cache is keyed on the
Path, so switching library dir is a different key. Called out in the docstring rather than left implicit.Tests
New
tests/test_resolved_root_cache.py: root resolved once across 500 lookups (the regression), a different root is a different cache entry, plus the full containment contract. Full suite green: 2,597 passed, 4 skipped.Summary by CodeRabbit
Bug Fixes
Tests