gc: priority scheduling with dual watermarks and cross-scan quota#436
Open
xiaoxichen wants to merge 3 commits into
Open
gc: priority scheduling with dual watermarks and cross-scan quota#436xiaoxichen wants to merge 3 commits into
xiaoxichen wants to merge 3 commits into
Conversation
xiaoxichen
commented
Jun 17, 2026
xiaoxichen
commented
Collaborator
- Sort eligible chunks by garbage ratio (desc) before submission so the most garbage-heavy chunks are always GC'd first
- Add gc_garbage_rate_threshold_low (default 30%) as a low watermark; chunks between the two watermarks consume at most half the quota
- Track m_pending_normal_gc_task_count in pdev_gc_actor to reflect tasks queued or running in m_gc_executor across scan cycles; previous code only capped submissions per scan, allowing unbounded queue growth
- scan_chunks_for_gc now skips a pdev entirely when already at quota, and derives low_tier_cap proportionally from remaining_capacity
- Add ADR docs/adr/gc-priority-scheduling.md
- Sort eligible chunks by garbage ratio (desc) before submission so the most garbage-heavy chunks are always GC'd first - Add gc_garbage_rate_threshold_low (default 30%) as a low watermark; chunks between the two watermarks consume at most half the quota - Track m_pending_normal_gc_task_count in pdev_gc_actor to reflect tasks queued or running in m_gc_executor across scan cycles; previous code only capped submissions per scan, allowing unbounded queue growth - scan_chunks_for_gc now skips a pdev entirely when already at quota, and derives low_tier_cap proportionally from remaining_capacity - Add ADR docs/adr/gc-priority-scheduling.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## stable/v4.x #436 +/- ##
==============================================
Coverage ? 54.23%
==============================================
Files ? 36
Lines ? 5423
Branches ? 686
==============================================
Hits ? 2941
Misses ? 2175
Partials ? 307 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| eligible.push_back({chunk_id, ratio_pct}); | ||
| } | ||
|
|
||
| // Sort eligible chunks by garbage ratio descending so the most garbage-heavy chunks are GC'd first. |
Collaborator
There was a problem hiding this comment.
this is a topK problem.
suggest to use a heap with a capacity of remaining_capacity (priority_queue), so that we can hold at most remaining_capacity ChunkGCInfo in memory. and for any new ChunkGCInfo, we only need to compare it with the top of the heap.
Collaborator
Author
There was a problem hiding this comment.
IMO It doesnt worth those lines of code , considering the maximum chunk is 32K.
Collaborator
There was a problem hiding this comment.
better to have it since it is not complicated and will also make code more simple, for example, the loop on the entire chunk collection will not be involved.
Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
- scan_chunks_for_gc: replace std::vector + std::sort with a bounded std::priority_queue (min-heap) of capacity max_task_num. Memory is now O(K) per pdev regardless of chunk count. K is fixed at max_task_num so we always retain enough candidates if capacity opens up later in the scan; submission is still gated by the dynamic remaining_capacity / low_tier_cap. - add_gc_task: move m_pending_normal_gc_task_count.fetch_add(1) BEFORE m_gc_executor->add() so an immediately-completing task (which decrements via ~gc_task_guard) cannot underflow the counter. - process_gc_task: when we early-return because the chunk is not in GC state, gc_task_guard is never constructed, so its decrement never fires. Decrement m_pending_normal_gc_task_count for normal-priority tasks here to mirror the guard's bookkeeping and prevent a permanent quota leak. - docs/adr: update ADR to describe top-K heap selection instead of full sort.
Collaborator
Author
|
@JacksonYao287 addressed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.