Add host memory introspection utilities - #756
Open
dsaini2-sc wants to merge 1 commit into
Open
Conversation
psutil.virtual_memory().available reads /proc/meminfo, which inside a container reports the HOST's memory rather than the container's limit, so a memory budget check that trusts it alone can pass immediately before the container is killed. This module reads the cgroup limit alongside meminfo and believes the smaller. It resolves the process's own cgroup path rather than assuming the hierarchy root, translates that path through each mount's root, tries v2 then v1, and selects the cgroup with the least headroom, since every ancestor constrains the process and allocations compete with every other descendant of it. cgroup_memory_breakdown separates anonymous memory from reclaimable page cache, which is what decides whether sitting near the limit is survivable; for v1 it reads the hierarchical total_* fields so the breakdown describes the same population as the usage figure beside it. log_stage_memory logs that split at pipeline phase boundaries. Adds a module and its tests only; no existing code path changes.
dsaini2-sc
requested review from
kmontemayor2-sc,
mkolodner-sc,
nshah-sc,
svij-sc,
xgao4-sc,
yliu2-sc and
zfan3-sc
as code owners
August 18, 2026 00:37
dsaini2-sc
enabled auto-merge
August 18, 2026 00:40
zfan3-sc
approved these changes
Aug 18, 2026
| self.assertLessEqual(available, host_memory.psutil.virtual_memory().total) | ||
|
|
||
|
|
||
| class MemoryBreakdownTest(CgroupResolutionTest): |
Collaborator
There was a problem hiding this comment.
nit: inheriting from a concrete test class will inherits its test methods. consider refactor it to move test methods out of the base class
| limit, current = limits | ||
| parts.append( | ||
| f"cgroup {current / 2**30:.1f}/{limit / 2**30:.1f} GiB " | ||
| f"({100.0 * current / limit:.0f}%) headroom {(limit - current) / 2**30:.1f} GiB" |
Collaborator
There was a problem hiding this comment.
nit: guard against division by 0?
Comment on lines
+1
to
+6
| """How much host memory this process may still allocate. | ||
|
|
||
| ``psutil.virtual_memory().available`` reads ``/proc/meminfo``, which inside a container reports the | ||
| HOST's memory rather than the container's limit, so a budget check trusting it alone can pass | ||
| immediately before the container is killed. The cgroup knows the real limit; read both and believe | ||
| the smaller. |
Collaborator
There was a problem hiding this comment.
nit. Can we update this docstring for recommended usage of the module? The public functions are all at the bottom which can make it hard for human readers to discover
Comment on lines
+51
to
+53
| fields = line.strip().split(":", 2) | ||
| if len(fields) != 3: | ||
| continue |
Collaborator
There was a problem hiding this comment.
nit, can we leave the expected output here?
| parts = [part for part in path.split("/") if part] | ||
| while True: | ||
| candidate = "/" + "/".join(parts) | ||
| if candidate not in expanded: |
Collaborator
There was a problem hiding this comment.
BTW do we call this often? Do we suspect that using a set here and t hen converting to list later (if needed) would be faster?
Collaborator
There was a problem hiding this comment.
On the perf note (and if if this isn't expensive it's ok to ignore), can we do the following:
- have `parts: tuple[str, ...] = tuple(part for part in path.split("/") if part)
for i in range(len(parts)- if
parts[:len(parts-i) in ... - Add the sub-tuple to expanded (as a set)
- Covert all to str at the end.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
gigl/utils/host_memory.pyand its tests. Two new files; no existing code path changes.Why
psutil.virtual_memory().availablereads/proc/meminfo, which inside a container reports the host's memory rather than the container's limit. A memory budget check that trusts it alone can pass moments before the container is OOM-killed. This module reads the cgroup limit alongside meminfo and believes the smaller.What it handles
cgroup_memory_breakdownseparates anonymous memory from reclaimable page cache, which is what decides whether sitting near the limit is survivable.log_stage_memorylogs that split at pipeline phase boundaries.Testing
45 tests pass;
tyandruffclean. The tests build real cgroup hierarchies in temporary directories and read them through the real code path, mocking only cgroup discovery andpsutil.