Release netCDF memory-map pages after each frame - #5463
Open
erikfransson wants to merge 1 commit into
Open
Conversation
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read. The map is read only, so dropping is safe: pages are faulted back in if the data are read again. MADV_DONTNEED does not exist on Windows, where the call is skipped and the reader is unchanged.
Documentation build overview
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #5463 +/- ##
========================================
Coverage 93.87% 93.88%
========================================
Files 182 182
Lines 22522 22526 +4
Branches 3206 3207 +1
========================================
+ Hits 21143 21148 +5
Misses 917 917
+ Partials 462 461 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read, as documented in discussion here. (ping @orbeckst )
MADV_DONTNEED does not exist on Windows, so the call is guarded by hasattr(mmap, "MADV_DONTNEED") and the reader is unchanged there.
Changes made in this Pull Request:
NCDFReader._read_framenow drops the pages of scipy's memory map once theframe has been read into the
Timestep.LLM / AI generated code disclosure
LLMs or other AI-powered tools (beyond simple IDE use cases) were used in this contribution: yes test written by AI
PR Checklist
package/CHANGELOGfile updated?package/AUTHORS?Developers Certificate of Origin
I certify that I can submit this code contribution as described in the Developer Certificate of Origin, under the MDAnalysis LICENSE.