Skip to content

Fix download cache bloat and stale disk capacity reporting causing Diego cell disk full#1163

Merged
ameowlia merged 3 commits into
developfrom
TNZ-111552
Jul 2, 2026
Merged

Fix download cache bloat and stale disk capacity reporting causing Diego cell disk full#1163
ameowlia merged 3 commits into
developfrom
TNZ-111552

Conversation

@kart2bc

@kart2bc kart2bc commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

The Problem

Diego cells were hitting 90%+ disk usage while the rep kept reporting tens of GB available. Two independent bugs combined to cause this.


Bug 1 — Download cache grows past its 10 GB ceiling

Where: cacheddownloader/file_cache.go makeRoom()

How it worked before:
The cache enforced its size ceiling using only in-memory accounting — it tracked how many bytes it thought it had stored. Three things caused that number to drift below reality:

  • If all entries were in use when a new download arrived, makeRoom() skipped eviction entirely but still added the new file — silently exceeding the ceiling
  • GetDirectory() doubles entry.Size when both a .tar and its extracted .tar.d exist simultaneously. If the ref-count logic misfired, the doubling was never reversed — the in-memory tracker over-counted that entry, causing it to evict earlier than needed and allowing more real disk to be used than the ceiling allows
  • These drifted sizes were saved to saved_cache.json on disk and reloaded on restart, carrying the error forward

Real-world result: /var/vcap/data/rep/shared/garden/download_cache grew to 46 GB while in-memory tracking reported < 10 GB.

Fix:
Added a minimumPartitionFreeBytes field (default 5 GB) and a FreeSpaceFunc that calls syscall.Statfs to read actual partition free space. makeRoom() now loops until both conditions are satisfied:

in-memory usage < maxSizeInBytes
AND
real partition free space >= minimumPartitionFreeBytes

This means even if the in-memory accounting has drifted, the real disk pressure triggers eviction as a safety net.


Bug 2 — Rep advertises stale disk capacity forever

Where: executor/depot/depot.go TotalResources()

How it worked before:
At rep startup, fetchCapacity() computed available disk once:

DiskMB = (80 GiB total − 10 GiB cache reservation − ~188 MiB overhead) = 71,492 MB

This number was frozen for the entire lifetime of the rep. Every scheduling decision the auctioneer made used this stale value. As the
download cache grew past its 10 GB reservation, TotalRethe auctioneer kept scheduling new containers onto anearly-full cell.

Real-world result: cfdot cell-states showed AvailableResources.DiskMB = 47,940 MB while the physical disk had only 7.6 GB free.

Fix:
TotalResources() now calls syscall.Statfs on the disk peturns a live DiskMB value. MemoryMB and Containersremain frozen (they don't change at runtime — only disk does).


Backward Compatibility

Breaking Change? No

@kart2bc kart2bc changed the title Tnz 111552 Fix download cache bloat and stale disk capacity reporting causing Diego cell disk full Jun 30, 2026
@kart2bc kart2bc force-pushed the TNZ-111552 branch 4 times, most recently from 726f194 to 7e4fe43 Compare June 30, 2026 18:47
@kart2bc kart2bc marked this pull request as ready for review June 30, 2026 20:06
@kart2bc kart2bc requested a review from a team as a code owner June 30, 2026 20:06

@ameowlia ameowlia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bug fix, but it is also a breaking change. Please make that clear in the PR description.

Comment thread jobs/rep_windows/spec
Comment thread jobs/rep/templates/rep.json.erb Outdated
Comment thread jobs/rep_windows/templates/rep.json.erb Outdated
Comment thread jobs/rep/spec
Comment thread src/code.cloudfoundry.org/cacheddownloader/file_cache.go Outdated
Comment thread src/code.cloudfoundry.org/executor/depot/depot_test.go
@kart2bc kart2bc force-pushed the TNZ-111552 branch 2 times, most recently from 38d2f9a to 1cfcaa3 Compare July 1, 2026 14:17
Comment thread jobs/rep_windows/spec
Comment thread src/code.cloudfoundry.org/cacheddownloader/file_cache.go
kart2bc added 2 commits July 2, 2026 11:23
… config

- TotalResources reverts to static startup value to keep total-remaining
  arithmetic consistent for BBS metrics
- RemainingResources caps reported DiskMB at live partition free space
  (syscall.Statfs), so schedulers see reduced capacity when disk fills
- MinCachePartitionFreeBytes changed to *uint64: nil means use the 5 GB
  default, 0 explicitly disables the check (previously impossible)

ai-assisted=yes
TNZ-111552
@kart2bc kart2bc force-pushed the TNZ-111552 branch 2 times, most recently from 220d68d to b3c9305 Compare July 2, 2026 15:44
@github-project-automation github-project-automation Bot moved this from Inbox to Pending Merge | Prioritized in Application Runtime Platform Working Group Jul 2, 2026
@ameowlia ameowlia merged commit 8ab220a into develop Jul 2, 2026
25 of 27 checks passed
@github-project-automation github-project-automation Bot moved this from Pending Merge | Prioritized to Done in Application Runtime Platform Working Group Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

2 participants