Fix download cache bloat and stale disk capacity reporting causing Diego cell disk full#1163
Merged
Conversation
726f194 to
7e4fe43
Compare
ameowlia
requested changes
Jun 30, 2026
ameowlia
left a comment
Member
There was a problem hiding this comment.
This is a bug fix, but it is also a breaking change. Please make that clear in the PR description.
38d2f9a to
1cfcaa3
Compare
ameowlia
requested changes
Jul 1, 2026
ai-assisted=yes TNZ-111552
… config - TotalResources reverts to static startup value to keep total-remaining arithmetic consistent for BBS metrics - RemainingResources caps reported DiskMB at live partition free space (syscall.Statfs), so schedulers see reduced capacity when disk fills - MinCachePartitionFreeBytes changed to *uint64: nil means use the 5 GB default, 0 explicitly disables the check (previously impossible) ai-assisted=yes TNZ-111552
220d68d to
b3c9305
Compare
ai-assisted=yes TNZ-1234
ameowlia
approved these changes
Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Problem
Diego cells were hitting 90%+ disk usage while the rep kept reporting tens of GB available. Two independent bugs combined to cause this.
Bug 1 — Download cache grows past its 10 GB ceiling
Where: cacheddownloader/file_cache.go makeRoom()
How it worked before:
The cache enforced its size ceiling using only in-memory accounting — it tracked how many bytes it thought it had stored. Three things caused that number to drift below reality:
Real-world result: /var/vcap/data/rep/shared/garden/download_cache grew to 46 GB while in-memory tracking reported < 10 GB.
Fix:
Added a minimumPartitionFreeBytes field (default 5 GB) and a FreeSpaceFunc that calls syscall.Statfs to read actual partition free space. makeRoom() now loops until both conditions are satisfied:
in-memory usage < maxSizeInBytes
AND
real partition free space >= minimumPartitionFreeBytes
This means even if the in-memory accounting has drifted, the real disk pressure triggers eviction as a safety net.
Bug 2 — Rep advertises stale disk capacity forever
Where: executor/depot/depot.go TotalResources()
How it worked before:
At rep startup, fetchCapacity() computed available disk once:
DiskMB = (80 GiB total − 10 GiB cache reservation − ~188 MiB overhead) = 71,492 MB
This number was frozen for the entire lifetime of the rep. Every scheduling decision the auctioneer made used this stale value. As the
download cache grew past its 10 GB reservation, TotalRethe auctioneer kept scheduling new containers onto anearly-full cell.
Real-world result: cfdot cell-states showed AvailableResources.DiskMB = 47,940 MB while the physical disk had only 7.6 GB free.
Fix:
TotalResources() now calls syscall.Statfs on the disk peturns a live DiskMB value. MemoryMB and Containersremain frozen (they don't change at runtime — only disk does).
Backward Compatibility
Breaking Change? No