search: attribute permit-wait latency to the blocking resource#6589
Draft
loutPhilipps wants to merge 1 commit into
Draft
search: attribute permit-wait latency to the blocking resource#6589loutPhilipps wants to merge 1 commit into
loutPhilipps wants to merge 1 commit into
Conversation
waiting_for_leaf_search_split_semaphore was a bare timer, so a long wait couldn't be attributed to the two things a split-search permit can block on: warmup/download slots or the memory budget. The actor now tracks the resource blocking the head of the queue in a shared BlockReasonHandle (an atomic). Because permits are served in order, whatever blocks the head blocks every waiter, so this queue-level reason is correct for a waiter at any position. Each SearchPermitFuture reads the handle, and a drop guard (WaitBlockReasonRecorder) on the wait span records blocked_on when the wait ends -- granted or cancelled -- gated on the wait being long (>=1ms) so instant grants stay unlabeled. Attributing on the wait side (not from the granted permit) is what lets the long, deadline-cancelled waits -- the ones we care about -- carry a reason even though they never get a permit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
waiting_for_leaf_search_split_semaphorewas a bare timer, so a long wait couldn't be attributed to the two things a split-search permit can block on: warmup/download slots or the memory budget.The actor now tracks the resource blocking the head of the queue in a shared
BlockReasonHandle(an atomic). Because permits are served in order, whatever blocks the head blocks every waiter, so this queue-level reason is correct for a waiter at any position. EachSearchPermitFuturereads the handle, and a drop guard (WaitBlockReasonRecorder) on the wait span recordsblocked_onwhen the wait ends — granted or cancelled — gated on the wait being long (≥1ms) so instant grants stay unlabeled. Attributing on the wait side (not from the granted permit) is what lets the long, deadline-cancelled waits — the ones we care about — carry a reason even though they never get a permit.Stacked on #6588.
🤖 Generated with Claude Code