Skip to content

rpc: report psrpc request delivery, expiry and claim outcomes - #1687

Draft
u9g wants to merge 1 commit into
mainfrom
r1-psrpc-request-observer
Draft

rpc: report psrpc request delivery, expiry and claim outcomes#1687
u9g wants to merge 1 commit into
mainfrom
r1-psrpc-request-observer

Conversation

@u9g

@u9g u9g commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Draft — depends on livekit/psrpc#118. go.mod currently pins a pseudo-version of that branch so this compiles; it must be re-pinned to a tagged psrpc release before merge.

Why

Three server-side lifecycle events in psrpc are invisible to the interceptor chain, because in each case the handler is never invoked:

  • a request read off the bus
  • a request dropped for arriving past its expiry
  • settlement of the claim handshake

The third is the one that matters. When a caller gets ErrNoResponse — no server bid within the selection window — nothing is recorded on either side. The request exists in no log, metric or trace, so "the request never arrived" and "a server bid too late" are indistinguishable, despite having different causes and different fixes.

Change

PSRPCMetricsObserver implements the new psrpc.RequestObserver. psrpc wires it automatically for any observer passed to middleware.WithServerMetrics, so every service already using WithServerObservability picks this up with no call-site change.

livekit_psrpc_requests_received_total{service,method}
livekit_psrpc_requests_expired_total{service,method}
livekit_psrpc_claim_total{service,method,outcome}         # granted | lost_to_peer | abandoned
livekit_psrpc_claim_wait_time_ms{service,method,outcome}

abandoned and pre-dispatch expiry also log at warn, alongside the existing server error / client error lines in rpc/logging.go.

Notes on the shape:

  • Lifecycle series carry no role label — they are server-side by construction, and a constant label would be misleading.
  • Histogram buckets are 1ms–3s: a granted claim settles in single-digit ms, an abandoned one runs to the caller's selection timeout (1s by default), so the mass sits at both ends.
  • lost_to_peer is expected on broadcast RPCs; on a queue RPC it means more than one member received the same request.

Tests

TestRequestObserverMetrics gathers from the registry and asserts each series actually registers and increments — not merely that the methods compile. A silent regression here would be invisible in production, since these are the only signals for a request whose handler never runs.

@changeset-bot

changeset-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 63f1e75

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

psrpc gained a RequestObserver for three server-side lifecycle events that the
interceptor chain cannot see, because in each case the handler is never
invoked: a request read off the bus, a request dropped for arriving past its
expiry, and the settlement of the claim handshake.

PSRPCMetricsObserver implements it, so every service already passing it through
WithServerObservability picks these up with no call-site change. New series:

  livekit_psrpc_requests_received_total{service,method}
  livekit_psrpc_requests_expired_total{service,method}
  livekit_psrpc_claim_total{service,method,outcome}
  livekit_psrpc_claim_wait_time_ms{service,method,outcome}

outcome is granted, lost_to_peer or abandoned. The abandoned case also logs at
warn: the caller has already returned ErrNoResponse upstream, so it is
otherwise unrecorded on either side.

Together these separate two failures that are currently identical from every
vantage point -- a caller ErrNoResponse with a matching abandoned claim means
the request was delivered and bid for, while one with no delivery event means
it never arrived. lost_to_peer on a queue RPC means more than one member
received the same request.

Lifecycle series carry no role label; they are server-side by construction.
@u9g
u9g force-pushed the r1-psrpc-request-observer branch from b895600 to 63f1e75 Compare July 27, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant