rpc: report psrpc request delivery, expiry and claim outcomes - #1687
Draft
u9g wants to merge 1 commit into
Draft
Conversation
|
psrpc gained a RequestObserver for three server-side lifecycle events that the
interceptor chain cannot see, because in each case the handler is never
invoked: a request read off the bus, a request dropped for arriving past its
expiry, and the settlement of the claim handshake.
PSRPCMetricsObserver implements it, so every service already passing it through
WithServerObservability picks these up with no call-site change. New series:
livekit_psrpc_requests_received_total{service,method}
livekit_psrpc_requests_expired_total{service,method}
livekit_psrpc_claim_total{service,method,outcome}
livekit_psrpc_claim_wait_time_ms{service,method,outcome}
outcome is granted, lost_to_peer or abandoned. The abandoned case also logs at
warn: the caller has already returned ErrNoResponse upstream, so it is
otherwise unrecorded on either side.
Together these separate two failures that are currently identical from every
vantage point -- a caller ErrNoResponse with a matching abandoned claim means
the request was delivered and bid for, while one with no delivery event means
it never arrived. lost_to_peer on a queue RPC means more than one member
received the same request.
Lifecycle series carry no role label; they are server-side by construction.
u9g
force-pushed
the
r1-psrpc-request-observer
branch
from
July 27, 2026 20:30
b895600 to
63f1e75
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — depends on livekit/psrpc#118.
go.modcurrently pins a pseudo-version of that branch so this compiles; it must be re-pinned to a tagged psrpc release before merge.Why
Three server-side lifecycle events in psrpc are invisible to the interceptor chain, because in each case the handler is never invoked:
The third is the one that matters. When a caller gets
ErrNoResponse— no server bid within the selection window — nothing is recorded on either side. The request exists in no log, metric or trace, so "the request never arrived" and "a server bid too late" are indistinguishable, despite having different causes and different fixes.Change
PSRPCMetricsObserverimplements the newpsrpc.RequestObserver. psrpc wires it automatically for any observer passed tomiddleware.WithServerMetrics, so every service already usingWithServerObservabilitypicks this up with no call-site change.abandonedand pre-dispatch expiry also log at warn, alongside the existingserver error/client errorlines inrpc/logging.go.Notes on the shape:
rolelabel — they are server-side by construction, and a constant label would be misleading.lost_to_peeris expected on broadcast RPCs; on a queue RPC it means more than one member received the same request.Tests
TestRequestObserverMetricsgathers from the registry and asserts each series actually registers and increments — not merely that the methods compile. A silent regression here would be invisible in production, since these are the only signals for a request whose handler never runs.