refactor(net, relay): reshape the GOAWAY API and move migration into Reconnect - #2542
refactor(net, relay): reshape the GOAWAY API and move migration into Reconnect#2542kixelated wants to merge 15 commits into
Conversation
|
@kixelated proposing we just go with this one as the path forward: I close #2490, and port over the one piece that is not here yet as a separate commit. That piece is the draining-route deprioritization from your original review note. Right now What the commit does: on GOAWAY receipt the subscriber bumps its own routes to I can rebase it onto (Written by claude-opus-4.8) |
There was a problem hiding this comment.
Sorry @kixelated, your pull request is larger than the review limit of 150000 diff characters
WalkthroughImplemented GOAWAY support across protocol encoders, transport sessions, reconnect handling, and relay shutdown. Added URI validation, draft-specific timeout and request-ID processing, session APIs, subscription gating, redirect policies, predecessor draining, and relay-wide shutdown coordination. Added documentation plus unit, integration, broadcast, cluster migration, failover, timeout, and mock transport tests. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… the URI The GOAWAY codecs existed but predated the draft-18/19 churn and had no bounds on the New Session URI: - draft-18 (#1559) requires a trailing Request ID on the control stream; encode appends 0 (no per-request tracking) and decode tolerates a lenient peer omitting it. draft-19 (#1623) removed the field again. - Enforce the 8,192-byte New Session URI cap on the IETF wire (all drafts) and add the same cap to moq-lite, rejected from the length prefix alone so a hostile length cannot force unbounded buffering. The moq-lite draft now specifies the cap (matching moq-transport). - js/net: mirror all of the above, with byte-layout tests locking the draft-18 Request ID parity against the Rust encoder. Test plan: cargo test -p moq-net --lib goaway (11 tests incl. the new cap and draft-18/19 layout tests); bun test on ietf.test.ts + lite/goaway.test.ts (95); kramdown-rfc parses the draft. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the GOAWAY lifecycle on both wires, replacing the receive-side stubs
(lite logged and ignored; IETF failed with Unsupported):
- Send: Session::drain() claims the one-GOAWAY-per-session slot and
returns a Drain; Drain::start(uri) / start_with_timeout(uri, t) fire
the frame and return a Draining whose complete() force-closes with the
new GoawayTimeout code (33) when the deadline expires.
- Receive: Session::goaway() resolves with GoawayReceived { uri, timeout }
and Session::is_going_away() is a cheap flag. Duplicate GOAWAYs keep
the first payload (an observer may already be acting on its URI) and
are logged.
- Request gating: after a received GOAWAY, new SUBSCRIBE / FETCH / TRACK /
announce-interest opens are rejected with GoingAway (32); PROBE is
silently skipped; existing subscriptions keep flowing.
- Wire channels per version: lite-04+ uses the dedicated Goaway control
stream; IETF draft-14-16 ride the shared control stream through the
adapter; draft-17+ use the SETUP uni streams (ours to send, the peer's
to receive). The receive reader stays alive after decoding, since
closing the peer's SETUP stream mid-session is a protocol violation a
strict peer punishes right in the middle of the drain.
- The lite sender waits for its FIN to be acknowledged before dropping
the Goaway stream: Writer's Drop resets, and on real QUIC a reset
racing the FIN discards the unacked GOAWAY frame (caught by the
real-transport tests; the in-memory mock cannot lose data). Same dance
as send_setup.
- Session::closed() now surfaces the application close code and reason
when the transport carries them (quinn's Display drops both), so a
peer can distinguish a GoawayTimeout force-close from a network error.
All plumbing follows the driver architecture: the send paths are TaskSet
children racing the drain trigger against transport close (nothing
spawned, nothing blocking clean shutdown), and Draining::complete polls
under a kio waiter rather than tokio::select.
Test plan: cargo test -p moq-net (544 lib + 8 goaway integration),
-p moq-relay, -p moq-native (incl. 6 real-QUIC/WebTransport GOAWAY
tests) all green; clippy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two consumers of the new moq-net GOAWAY support: Upstream (cluster): when a peer sends GOAWAY, the relay dials the replacement (the redirect URI when safe, else the original URL) while the old session keeps serving, then drains the old session in the background (the GOAWAY's own deadline, else --cluster-drain-timeout, default 10s) and force-closes it with GoawayTimeout. The replacement shares the same origin, so its announcements attach as routes to the broadcasts the old session served and the origin hands live tracks over at a group boundary when the old routes detach. Downstream sessions never see a GOAWAY. A redirect is followed only when its scheme is at least as secure as the current connection's (no plaintext downgrade; unknown schemes rank lowest, fail-safe), and the cluster JWT carries onto the redirect unless it brings its own. Constraining the redirect host to known peers is a deliberate follow-up, marked in the code. Downstream (--drain-timeout): the first shutdown signal broadcasts a drain: every accepted session (QUIC/WebTransport and the WebSocket fallback) sends an empty-URI GOAWAY (reconnect to me) and is force-closed after the window; a second signal, or the window elapsing, exits. Clients on pre-GOAWAY versions (moq-lite-03 and earlier) are closed immediately; there is no wire message to warn them with. Test plan: cargo test -p moq-relay (148 lib incl. new scheme-tier and redirect-resolution unit tests + 12 smoke); clippy clean. Cluster migration integration tests land in the next commit on this branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…test moq-net (in-memory mock transport, deterministic, no sleeps): - send/receive across the full GOAWAY version matrix (lite-04/05, moq-transport-14/16/17/18/19), covering all three wire channels - wire timeout on draft-17+, and the GOAWAY_TIMEOUT force-close observed by an overstaying peer - drain claim exclusivity + release-on-drop; drain() is None pre-lite-04 - regression: a duplicate GOAWAY (injected as raw wire frames; the public API can't send two) keeps the first payload - request gating: a new subscribe after GOAWAY never reaches the wire (delivered as a resume-model stall), while the existing subscription keeps flowing The mock transport and harness are revived from the earlier GOAWAY branch; the harness now spawns each side's Driver as soon as its handshake resolves, since draft-17+ handshakes each block on the peer driver's SETUP. moq-relay (real TCP transports): cluster_migrates_on_upstream_goaway stands up two sibling upstreams sharing one origin, drains sibling A with a redirect to B, and asserts the cluster reconnects, delivery resumes contiguously at the next group, no announce churn leaks to the cluster origin, and the old session drains away. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…knobs - doc/concept/layer/moq-lite.md: GOAWAY send/receive lifecycle on the current API, the route-based seamless resume story, version support, and the js/net wire-only status. - doc/bin/relay/config.md: the top-level drain_timeout (shutdown drain) and cluster.drain_timeout (upstream GOAWAY drain window) keys. Changelogs are release-plz generated from the commit subjects, so no manual CHANGELOG edits ride along. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the `Drain`/`Draining` typestate with the crate's own Producer/Consumer idiom, matching `bandwidth` on the same struct: session.send_goaway() -> Option<goaway::Producer> // taken once session.recv_goaway() -> goaway::Consumer // peek/poll/recv `Producer::send` consumes the handle, so "one GOAWAY per session" is a move rather than an atomic claim with release-on-drop. The driver now owns the force-close deadline, so `Session::closed()` is the single completion signal and `Draining` no longer holds a `Session` clone alive while waiting for a peer that may never leave. Also drops `Session::is_going_away` (`recv_goaway().peek()`), the public `Version::has_goaway`, and the `Duration::ZERO` magic value, and uses `String` rather than `Arc<str>` for the URI: there is at most one GOAWAY per session, so the cheap-clone win was optimizing a once-per-connection allocation while costing a conversion at every FFI boundary. Two conformance fixes, per moq-transport draft-19 sect 10.4: - A client naming a redirect URI is refused with ProtocolViolation. A client cannot instruct a server to open connections, and a conformant server MUST close the session over it. - A duplicate GOAWAY closes the session instead of being logged and discarded. Keeping the first payload leaves an observer acting on a URI the peer has since replaced, with no way to tell. The moq-lite draft specifies both, plus scheme continuity and sticky redirects, which were previously unspecified there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Migration belongs in `Reconnect`, not in each application: it already owns the connect/backoff lifecycle, and everything native (libmoq and therefore py/swift/kt/go, moq-gst, moq-video) reaches it for free. On a GOAWAY from a healthy session the loop dials the replacement while the old session keeps serving, so the old routes stay attached and live tracks hand over at a group boundary. There is no backoff sleep: a handover is not a failure. A session that ends within `backoff.initial` counts as a failed attempt however it ended, GOAWAY included. That one rule bounds redirect churn: an A-redirects-to-B-redirects-to-A loop escalates through backoff and gives up at `backoff.timeout`, so no separate churn counter is needed. The redirect target is sticky for the life of the loop. A redirect is an assignment, not a detour, so reverting to the configured URL would redial the node that just told us to leave. A fresh `Reconnect` starts from the configured URL again. `Redirect` is the trust policy, defaulting to `Follow` like every HTTP client. It refuses scheme downgrades and, unlike the previous relay-side check, refuses a redirect that widens reachability (a public endpoint pointing at loopback, a private range, or IPC). `unix` is no longer ranked as an upgrade over network transports. The old session drains inside the loop rather than in a detached task, so dropping the `Reconnect` handle tears it down too instead of leaving an orphan holding the connection open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`run_remote_once` no longer hand-rolls the GOAWAY dance. The reconnect
loop owns it, deleting the inline retry loop, the hardcoded backoff, the
churn window/limit constants, and the relay's own redirect resolver
(now `moq_native::Redirect`, reachable as `--client-goaway-*`).
`Connection` becomes a builder, matching the `Web::with_shutdown` added
alongside it. This keeps the change additive: the required `shutdown`
pub field would have broken every external constructor and routed the PR
to `dev`.
`--drain-timeout` takes a humantime duration ("10s", "500ms") like its
neighbour `--cluster-linger`, rather than bare seconds.
`--cluster-drain-timeout` is gone; the upstream handover window is now
`--client-goaway-handover` on the reconnect loop that performs it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cargo doc -D warnings runs only in CI, so a stale intra-doc link from the Drain removal passed just check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`cluster_diamond_goaway_seamless_failover` lost six groups across the handover: the subscriber received 0, 1, 2 and then jumped to 9. Root cause: the "a session ending within `backoff.initial` counts as a failed attempt" rule was applied to the *handover* as well as to the backoff bookkeeping. `backoff.initial` is 1s, and a cluster peer that GOAWAYs shortly after the handshake is inside that window, so the migration took the failure path: the old session was dropped instead of being kept alive, and the loop slept a full second before dialing the replacement. Nothing served the subscription across that gap, so every group published during it was lost. Deterministic, and invisible to a delay inserted on the healthy path because that path never ran. Split the two concerns. The handover is now unconditional: the old session keeps serving until it closes or overstays its window, whatever the backoff scores this session as. Only the bookkeeping branches, so an early GOAWAY still records a failed attempt and sleeps the backoff, and two peers bouncing us between them still escalate and give up. The redirect loop costs time rather than data. The tests set `--client-goaway-handover` to 2s, replacing the `ClusterConfig::drain_timeout` override they lost when that knob moved to the reconnect loop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rs/moq-net/src/lite/goaway.rs (1)
24-48: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winEnforce the URI cap on encode as well.
decode_msgrejects oversized values, butencode_msgwrites them. A localsend_goaway()with a URI over 8,192 bytes therefore succeeds locally and makes a compliant peer terminate the session for a protocol violation. Validate the byte length before encoding and share a named cap constant with the decoder and tests.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-net/src/lite/goaway.rs` around lines 24 - 48, Update the GoAway message encoding flow in encode_msg to reject URIs whose byte length exceeds 8,192 before writing them, returning the existing invalid-value encode error. Define a named URI-cap constant and reuse it in both decode_msg and encode_msg, ensuring tests reference the same constant rather than duplicating the literal.
🧹 Nitpick comments (8)
rs/moq-native/tests/broadcast.rs (1)
2475-2498: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNo coverage for the new
Reconnectmigration path.These tests exercise raw session-level GOAWAY only. The behavior this layer adds (
Redirect::resolvepolicy,Status::Migrating, sticky redirect URL,Draininghandover, the healthy-vs-immediate backoff split) is untested here. A test that drivesClient::reconnectagainst a server that GOAWAYs with a redirect to a second server would cover the interesting part. Want me to draft one?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-native/tests/broadcast.rs` around lines 2475 - 2498, Add coverage for the Reconnect migration path alongside the existing GOAWAY tests by driving Client::reconnect against a server that sends GOAWAY with a redirect to a second server. Assert Redirect::resolve policy, Status::Migrating, sticky redirect URL, Draining handover, and distinct healthy-versus-immediate backoff behavior.rs/moq-native/src/reconnect.rs (1)
114-162: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd inline tests for
Redirect::resolve/scheme_tier.These are the security boundary for peer-supplied URIs (scheme downgrade, reachability widening,
SameHost), and the branches are cheap to pin down. Note one gap worth a case: whencurrenthas an unclassified scheme (tier 0, e.g.unix:), any target passes the downgrade check.As per coding guidelines: "Keep Rust tests inline as
#[cfg(test)] mod tests".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-native/src/reconnect.rs` around lines 114 - 162, Add an inline #[cfg(test)] mod tests beside Redirect::resolve and scheme_tier, covering empty or ignored redirects, malformed URIs, scheme downgrades, reachability widening, SameHost mismatches, and accepted redirects. Include a case where current uses an unclassified tier-0 scheme such as unix: and any target passes the downgrade check, while asserting scheme_tier rankings directly where useful.Source: Coding guidelines
rs/moq-relay/src/shutdown.rs (1)
27-31: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider a getter instead of a public
drain_timeoutfield.A public field lets any holder of a cloned handle mutate its own drain window, silently diverging from the value
mainconfigured. A private field plus an accessor keeps the surface hard to misuse.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-relay/src/shutdown.rs` around lines 27 - 31, Make Shutdown::drain_timeout private and add a read-only accessor method for retrieving the configured Duration. Update existing consumers to call the accessor instead of accessing the field directly, preserving the value configured by main across cloned handles.Source: Coding guidelines
rs/moq-relay/src/web.rs (1)
160-166: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
with_shutdownpanics if any state clone exists.
routes()clonesself.stateinto the router, soweb.routes()followed byweb.with_shutdown(...)aborts the process for an embedder. A panicking builder method is easy to misuse; consider holdingshutdownonWeband constructingWebStatelazily inroutes()/serve(), so ordering never matters.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-relay/src/web.rs` around lines 160 - 166, Refactor the Web builder around with_shutdown so it no longer relies on Arc::get_mut or panics after routes() has cloned state. Store the shutdown value directly on Web, then apply it when constructing WebState lazily in routes() or serve(), preserving shutdown behavior regardless of whether with_shutdown is called before or after routes().rs/moq-relay/tests/goaway_cluster.rs (1)
267-270: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the orphaned drain comment.
"Short drain so the test observes teardown quickly." no longer describes any code here (the drain configuration was dropped), leaving a misleading dangling comment.
♻️ Proposed cleanup
let mut cluster_config = ClusterConfig::default(); cluster_config.connect = vec![upstream_url.to_string()]; - // Short drain so the test observes teardown quickly. - let mut client_config = moq_native::ClientConfig::default();As per coding guidelines: "Comments and documentation must describe the current behavior, not historical migration context or obsolete behavior."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-relay/tests/goaway_cluster.rs` around lines 267 - 270, Remove the obsolete “Short drain” comment immediately after the cluster_config.connect assignment in the test setup, leaving the active ClusterConfig initialization and connection configuration unchanged.Source: Coding guidelines
rs/moq-relay/src/config.rs (1)
62-73: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a config-merge regression test for
drain-timeout.The TOML tests cover similar
Option<Duration>fields, butrs/moq-relay/src/config.rshas no regression for the new--drain-timeout/MOQ_DRAIN_TIMEOUTflag leavingdrain_timeoutintact when the CLI flag is absent, or being overridden when present.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-relay/src/config.rs` around lines 62 - 73, The config merge tests around the relay configuration must add regression coverage for drain_timeout: verify a TOML-configured value remains intact when --drain-timeout is absent, and verify the CLI value overrides it when provided. Reuse the existing Option<Duration> TOML merge test pattern and reference the drain_timeout field and its CLI/environment configuration.Source: Coding guidelines
js/net/src/lite/goaway.ts (1)
31-37: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider a shared named constant for the 8,192-byte cap.
The same literal and near-identical check now live in
js/net/src/ietf/goaway.ts(and in the Rust decoders). A shared exported constant (e.g.MAX_NEW_SESSION_URI_BYTES) keeps the limit and the error wording in one place.As per coding guidelines: "Avoid using magic numbers; use named constants instead."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@js/net/src/lite/goaway.ts` around lines 31 - 37, Replace the hard-coded 8,192-byte limit in the Goaway decoder with a shared exported named constant, such as MAX_NEW_SESSION_URI_BYTES, and reuse that constant in both the lite and IETF goaway implementations. Update the associated error wording to reference the shared limit while preserving the existing byte-length validation behavior.Source: Coding guidelines
rs/moq-native/src/lib.rs (1)
47-47: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider
#[non_exhaustive]on the newly publicGoawayConfigandRedirect.Both become part of the published surface here.
Redirectis a policy enum likely to gain variants, andGoawayConfigis an additive config struct with public fields and aDefault. Existing exports (Backoff) share this shape, so this is a consistency call rather than a blocker.As per coding guidelines: "For Rust config structs with public fields, use
#[non_exhaustive]plusDefaultor a constructor; mark public enums that may gain variants as#[non_exhaustive]."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rs/moq-native/src/lib.rs` at line 47, Mark the newly public GoawayConfig struct and Redirect enum as #[non_exhaustive] in their definitions within the reconnect module. Preserve GoawayConfig’s existing Default implementation and public API behavior, and leave the re-export in lib.rs unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@doc/concept/layer/moq-lite.md`:
- Around line 143-153: The lifecycle documentation should use the current GOAWAY
API instead of the removed drain flow: replace the sending-side references to
session.drain(), Drain, start, and Draining with session.send_goaway(),
Producer::send(Goaway), and Session::closed(), and update the receiving-side
flow to use session.recv_goaway() and Session::closed(). Preserve the documented
behavior for rejecting new requests, retaining existing subscriptions, and
reconnecting through a replacement session.
In `@rs/moq-native/src/client.rs`:
- Around line 63-67: Update the doc comment for Client::goaway to advertise the
actual flattened CLI flag names defined by GoawayConfig, replacing the
inaccurate --client-goaway-* pattern once the finalized flag names are
confirmed.
In `@rs/moq-native/src/reconnect.rs`:
- Around line 90-93: Update the documentation for Reconnect::status to describe
all current connection states, including Migrating, instead of claiming the loop
only alternates between Connected and Disconnected. Keep the documentation
aligned with the status behavior introduced by the Migrating variant.
- Around line 337-378: Ensure the reconnect loop continues polling an existing
Draining predecessor while connection attempts and backoff sleeps are pending,
so its handover deadline is enforced even when reconnects fail. Update the
immediate-redirect retry path around run_session and client.connect to race or
otherwise service draining alongside connect/backoff waits. Before assigning a
new Draining in the Ended::Goaway branch, retire the existing predecessor
instead of silently overwriting it.
- Around line 176-202: The GoawayConfig CLI fields register incorrect names and
scalar defaults that overwrite TOML values. In rs/moq-native/src/reconnect.rs
lines 176-202, set explicit client-goaway long names, use the
MOQ_CLIENT_GOAWAY_* environment variables, change redirect and handover to
Option<T> without default_value, apply fallback defaults at the use sites around
lines 348 and 359, and add a regression test confirming CLI defaults do not
override TOML configuration. In rs/moq-native/src/client.rs lines 63-67, update
the documentation to advertise the --client-goaway-* flags.
- Around line 165-173: Update is_local to classify IPv6 link-local addresses,
IPv4-mapped IPv6 addresses, and the unspecified IPv4 address 0.0.0.0 as local.
Preserve the existing domain, Unix-socket, loopback, private, and link-local
checks, and apply the IPv4 classification to mapped addresses before returning
the IPv6 result.
In `@rs/moq-net/src/goaway.rs`:
- Around line 90-99: The GOAWAY send path must reject oversized URIs and prevent
message-length truncation. In rs/moq-net/src/goaway.rs lines 90-99, update
Producer::send to return Error::ProtocolViolation when goaway.uri exceeds 8,192
bytes; in rs/moq-net/src/ietf/adapter.rs lines 545-576, update send_goaway to
convert body.len() with try_into() instead of an unchecked u16 cast, handling
conversion failure as an error.
In `@rs/moq-net/src/ietf/session.rs`:
- Around line 512-538: Update the post-GOAWAY drain loop in the SETUP stream
handler to preserve duplicate-GOAWAY enforcement: when the decoded message ID is
ietf::GoAway::ID, record it through goaway.record(...) and propagate its error
instead of only logging and ignoring it. Keep the existing discard-and-log
behavior for other message types.
In `@rs/moq-net/src/ietf/subscriber.rs`:
- Around line 90-92: Update run_subscribe_namespace to check the shared
going_away state before allocating a request ID or opening the
SUBSCRIBE_NAMESPACE stream; return Error::GoingAway when the peer has sent
GOAWAY, while preserving the existing request flow otherwise.
In `@rs/moq-net/tests/support/mock.rs`:
- Around line 286-307: Update accept_uni and accept_bi to create the
close_notify.notified() future before awaiting rx.recv(), then check the close
signal after the channel returns None using the same notified-before-check
pattern as closed(). Preserve returning the received stream or pair immediately,
and return close_error() once shutdown is confirmed without introducing a
post-close notification race.
---
Outside diff comments:
In `@rs/moq-net/src/lite/goaway.rs`:
- Around line 24-48: Update the GoAway message encoding flow in encode_msg to
reject URIs whose byte length exceeds 8,192 before writing them, returning the
existing invalid-value encode error. Define a named URI-cap constant and reuse
it in both decode_msg and encode_msg, ensuring tests reference the same constant
rather than duplicating the literal.
---
Nitpick comments:
In `@js/net/src/lite/goaway.ts`:
- Around line 31-37: Replace the hard-coded 8,192-byte limit in the Goaway
decoder with a shared exported named constant, such as
MAX_NEW_SESSION_URI_BYTES, and reuse that constant in both the lite and IETF
goaway implementations. Update the associated error wording to reference the
shared limit while preserving the existing byte-length validation behavior.
In `@rs/moq-native/src/lib.rs`:
- Line 47: Mark the newly public GoawayConfig struct and Redirect enum as
#[non_exhaustive] in their definitions within the reconnect module. Preserve
GoawayConfig’s existing Default implementation and public API behavior, and
leave the re-export in lib.rs unchanged.
In `@rs/moq-native/src/reconnect.rs`:
- Around line 114-162: Add an inline #[cfg(test)] mod tests beside
Redirect::resolve and scheme_tier, covering empty or ignored redirects,
malformed URIs, scheme downgrades, reachability widening, SameHost mismatches,
and accepted redirects. Include a case where current uses an unclassified tier-0
scheme such as unix: and any target passes the downgrade check, while asserting
scheme_tier rankings directly where useful.
In `@rs/moq-native/tests/broadcast.rs`:
- Around line 2475-2498: Add coverage for the Reconnect migration path alongside
the existing GOAWAY tests by driving Client::reconnect against a server that
sends GOAWAY with a redirect to a second server. Assert Redirect::resolve
policy, Status::Migrating, sticky redirect URL, Draining handover, and distinct
healthy-versus-immediate backoff behavior.
In `@rs/moq-relay/src/config.rs`:
- Around line 62-73: The config merge tests around the relay configuration must
add regression coverage for drain_timeout: verify a TOML-configured value
remains intact when --drain-timeout is absent, and verify the CLI value
overrides it when provided. Reuse the existing Option<Duration> TOML merge test
pattern and reference the drain_timeout field and its CLI/environment
configuration.
In `@rs/moq-relay/src/shutdown.rs`:
- Around line 27-31: Make Shutdown::drain_timeout private and add a read-only
accessor method for retrieving the configured Duration. Update existing
consumers to call the accessor instead of accessing the field directly,
preserving the value configured by main across cloned handles.
In `@rs/moq-relay/src/web.rs`:
- Around line 160-166: Refactor the Web builder around with_shutdown so it no
longer relies on Arc::get_mut or panics after routes() has cloned state. Store
the shutdown value directly on Web, then apply it when constructing WebState
lazily in routes() or serve(), preserving shutdown behavior regardless of
whether with_shutdown is called before or after routes().
In `@rs/moq-relay/tests/goaway_cluster.rs`:
- Around line 267-270: Remove the obsolete “Short drain” comment immediately
after the cluster_config.connect assignment in the test setup, leaving the
active ClusterConfig initialization and connection configuration unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4a3c3cf2-7026-4894-b97a-26515ede72b7
📒 Files selected for processing (41)
doc/bin/relay/config.mddoc/concept/layer/moq-lite.mddrafts/draft-lcurley-moq-lite.mdjs/net/src/ietf/goaway.tsjs/net/src/ietf/ietf.test.tsjs/net/src/lite/goaway.test.tsjs/net/src/lite/goaway.tsrs/moq-native/src/client.rsrs/moq-native/src/lib.rsrs/moq-native/src/reconnect.rsrs/moq-native/tests/broadcast.rsrs/moq-net/src/client.rsrs/moq-net/src/error.rsrs/moq-net/src/goaway.rsrs/moq-net/src/ietf/adapter.rsrs/moq-net/src/ietf/goaway.rsrs/moq-net/src/ietf/session.rsrs/moq-net/src/ietf/subscriber.rsrs/moq-net/src/lib.rsrs/moq-net/src/lite/goaway.rsrs/moq-net/src/lite/publisher.rsrs/moq-net/src/lite/session.rsrs/moq-net/src/lite/subscriber.rsrs/moq-net/src/lite/version.rsrs/moq-net/src/server.rsrs/moq-net/src/session.rsrs/moq-net/src/version.rsrs/moq-net/tests/goaway.rsrs/moq-net/tests/support/harness.rsrs/moq-net/tests/support/mock.rsrs/moq-net/tests/support/mod.rsrs/moq-relay/src/cluster.rsrs/moq-relay/src/config.rsrs/moq-relay/src/connection.rsrs/moq-relay/src/lib.rsrs/moq-relay/src/main.rsrs/moq-relay/src/shutdown.rsrs/moq-relay/src/web.rsrs/moq-relay/src/websocket.rsrs/moq-relay/tests/goaway_cluster.rsrs/moq-relay/tests/smoke.rs
moq-net: - Cap the New Session URI at 8,192 bytes in `goaway::Producer::send`, the one chokepoint both wires funnel through. Every decoder in the tree already rejects longer, so sending one only got the session closed by the peer. - Compute the IETF control-stream size prefix with `try_into` instead of `as u16`. That stream is shared with every other in-flight request, so a wrapping cast would desynchronize framing for all of them. - Enforce a duplicate GOAWAY on the draft-17+ SETUP stream too. The shared control stream (draft-14-16) already closed over it, so the two paths disagreed and the comment described only one of them. - Gate `run_subscribe_namespace` on the IETF wire after a received GOAWAY, matching the lite side and the SHOULD NOT in draft-19 sect 10.4. Announce-interest could still open a stream. - Fix a lost wakeup in the mock transport: `notify_waiters` stores no permit, so `accept_uni`/`accept_bi` registering after a close already fired would park until the test's outer timeout. moq-native: - `is_local` missed IPv4-mapped forms (`::ffff:127.0.0.1`), IPv6 link-local (`fe80::/10`), and unspecified addresses, so a peer could redirect us at the local host through any of them. - `GoawayConfig` fields are `Option<T>` without clap defaults, per the TOML-merge rule: bare scalars let the CLI re-parse clobber a TOML-configured value. Defaults move to accessors, with the regression test the guide requires for a new flag. - A draining predecessor is now retired before a new one replaces it, and stays polled across the backoff sleep. It was previously only polled from `run_session`, so a replacement that took several rounds to reach left the old session holding its connection past the window. docs: the moq-lite concept page still documented the removed `drain()`/`Drain`/`Draining` flow; the relay config page still described the removed `cluster.drain_timeout` and seconds-valued `drain_timeout`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`RouteCost::charged` saturated at `u64::MAX`, but the sum is re-encoded as a QUIC varint when the announcement is forwarded, and a varint tops out at 2^62-1. A peer may legally advertise that largest value on lite-06, so adding this link's price pushed the total out of range and the downstream encode failed with BoundsExceeded, breaking the announce stream. Saturate at the ceiling the wire can actually carry instead, so whatever a peer advertises, what we forward still encodes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A GOAWAY left the peer's routes at their original cost, so the origin kept preferring them. `route_order` ranks on (announce, cost, hop count, hash), so a draining route that tied on cost stayed primary through the hop/hash tie-break and served new groups right up until the session closed. Because resume is group-granular, whatever group was in flight at the force-close was lost. The handover was meant to happen before that, not at it. Routes fed by a session now move to `broadcast::DRAIN_COST` when its peer drains, so the origin hands live subscriptions to any other route at the next group boundary. Deprioritized, not withdrawn: a broadcast reachable only over the draining session keeps being served, which is what makes the sender's deadline a handover window rather than a cutoff. Both wires do this in their per-source task, whose `kio::wait` already races in-memory pollables. That is what makes the two symmetric and what keeps a peer that drains and then goes quiet from being missed: reacting on the next announce would never fire, and racing the announce read itself would mean cancelling a partially-buffered transport read. `DRAIN_COST` is the largest cost a varint can carry rather than `u64::MAX`, since a draining route is still announced downstream. The drain lives on `broadcast::Dynamic` because that is the handle each wire's per-source task already holds, so nothing keeps a second producer clone alive just to reach the route. `GoingAway` gains the poll, reusing the GOAWAY channel it already sits next to rather than adding a second one. Co-Authored-By: Kyle Sletmoe <sletmoe@amazon.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`SameHost` is documented as letting a peer move us between ports or schemes but not to another host, which is also what its name says. It compared `Url::authority()`, which includes the port, so it refused exactly the handoff it advertises: a relay pointing at a sibling process on the same box. Compare the host instead, and cover the redirect policy properly: scheme downgrade refusal, the empty/malformed/ignored fallbacks, and the SameHost pin. That is the security-relevant path and it had no direct tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rs/moq-net/src/model/broadcast.rs`:
- Around line 52-58: Enforce MAX_COST in every public route construction path,
including Route::new and Route::with_cost, so no Route::cost can exceed the wire
ceiling; clamp or reject oversized values consistently with the existing API
contract. Add a regression test covering Route::new().with_cost(u64::MAX) and
verify the resulting route cannot carry a cost above MAX_COST.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 34855a26-0d98-493c-b47b-faf44b62d867
📒 Files selected for processing (16)
doc/bin/relay/config.mddoc/concept/layer/moq-lite.mddrafts/draft-lcurley-moq-lite.mdrs/moq-native/src/client.rsrs/moq-native/src/reconnect.rsrs/moq-net/src/goaway.rsrs/moq-net/src/ietf/adapter.rsrs/moq-net/src/ietf/session.rsrs/moq-net/src/ietf/subscriber.rsrs/moq-net/src/lite/announce.rsrs/moq-net/src/lite/subscriber.rsrs/moq-net/src/model/broadcast.rsrs/moq-net/src/model/origin.rsrs/moq-net/tests/goaway.rsrs/moq-net/tests/support/mock.rsrs/moq-relay/tests/goaway_cluster.rs
🚧 Files skipped from review as they are similar to previous changes (10)
- doc/concept/layer/moq-lite.md
- rs/moq-native/src/client.rs
- rs/moq-net/tests/goaway.rs
- rs/moq-relay/tests/goaway_cluster.rs
- rs/moq-net/src/ietf/adapter.rs
- rs/moq-net/src/lite/subscriber.rs
- rs/moq-net/src/goaway.rs
- rs/moq-net/tests/support/mock.rs
- rs/moq-net/src/ietf/session.rs
- rs/moq-native/src/reconnect.rs
| /// The highest value [`Route::cost`] can take, and where cost accumulation | ||
| /// saturates. | ||
| /// | ||
| /// The ceiling is the wire's, not the model's: lite-06 carries the cost as a QUIC | ||
| /// varint, which tops out at 2^62-1, so a larger value could be selected on but | ||
| /// never forwarded. | ||
| pub const MAX_COST: u64 = (1 << 62) - 1; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Enforce MAX_COST at every public route entry point.
Route::cost is public and Route::with_cost accepts any u64, so callers can still create a cost above this ceiling. Such a route can reach Lite-06 forwarding and fail varint encoding, contradicting the new “capped” contract. Clamp or reject oversized costs when routes enter the model and add a regression test using Route::new().with_cost(u64::MAX).
Also applies to: 87-87
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rs/moq-net/src/model/broadcast.rs` around lines 52 - 58, Enforce MAX_COST in
every public route construction path, including Route::new and Route::with_cost,
so no Route::cost can exceed the wire ceiling; clamp or reject oversized values
consistently with the existing API contract. Add a regression test covering
Route::new().with_cost(u64::MAX) and verify the resulting route cannot carry a
cost above MAX_COST.
Yeah I'm working on it. We might not even need the DRAIN_COST thing if we switch to preferring the newer route, which is generally the correct one. |
The handover window took the peer's deadline outright whenever it named one, so `--goaway-handover` only applied to a GOAWAY that carried none. A peer could name an hour and we would hold the old session open for it. Take whichever deadline comes first instead. The peer's is a promise about when it force-closes, so waiting past it just holds a dead session; ours is the cap on how long we keep one around at all, and it should hold regardless of what the peer asks for. `GoawayConfig::handover` now takes the peer's deadline and returns the effective window, so the policy lives in one place rather than in a caller that has to remember to apply it. Never released, so no deprecation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Summary
An API reshape of #2490, built on @ksletmoe-aws's five commits so the GOAWAY
implementation, the codec work, and the test suite keep their authorship. Their
commits are unmodified apart from a rebase onto current
main. #2490 is nowclosed in favor of this, and its remaining piece (draining-route
deprioritization) is ported here.
What changed
moq-net:
Drain/Drainingbecomegoaway::Producer/ConsumerEvery call site in #2490 was the same one-liner,
drain().start*(uri).complete().await,so the three-type typestate plus a claim released on drop was machinery with no
users.
Producer::sendconsumes the handle, making "one GOAWAY per session" amove rather than an atomic claim. The consumer matches
bandwidth::Consumeronthe same struct, giving a sync
peek, a composablepoll, and a cloneablehandle the reconnect loop can hold.
The driver now owns the force-close deadline, so
Session::closed()is the onecompletion signal. That drops
Drainingentirely, along with the footgun whereit held a
Sessionclone alive while waiting for a peer that might never leave.Also gone:
Session::is_going_away(nowrecv_goaway().peek()), publicVersion::has_goaway,startvsstart_with_timeout, and theDuration::ZERO= "no deadline" magic value.
Goaway::new()/Goaway::redirect(uri)/.with_timeout(d)construct the message. URIs areString: there is at most oneGOAWAY per session, so
Arc<str>was optimizing a once-per-connection allocationwhile costing a conversion at every FFI boundary.
moq-net: two conformance fixes
Both are MUST-level in moq-transport draft-19 sect 10.4:
ProtocolViolation. Clientscannot instruct servers to open connections, and a conformant server MUST close
the session over it. feat(net, relay): GOAWAY with graceful drain and cluster migration #2490's API let a client send one and get killed for it.
feat(net, relay): GOAWAY with graceful drain and cluster migration #2490's keep-first reasoning was sound (an observer may already be acting on the
first URI) but leaves that observer acting on a URI the peer has replaced, with
no way to tell.
sendconsumes the handle even when it returnsErr: naming a URI as a client isa bug to fix, not a runtime condition to recover from.
moq-native:
Reconnectperforms the migrationThe lifecycle belongs in the reconnect loop, not in each application.
Reconnectalready owns connect/backoff, and everything native reaches it (libmoq and
therefore py/swift/kt/go, moq-gst, moq-video), so they all get migration for free.
On a GOAWAY from a healthy session it dials the replacement while the old session
keeps serving, so old routes stay attached and live tracks hand over at a group
boundary (#2241). No backoff sleep: a handover is not a failure.
A session ending within
backoff.initialcounts as a failed attempt however itended, GOAWAY included. That one rule replaces #2490's
CHURN_WINDOW/CHURN_LIMIT:an A-redirects-to-B-redirects-to-A loop escalates through backoff and gives up at
backoff.timeout. The residual slow ping-pong (each leg healthy for seconds) isleft unguarded deliberately, since a long-lived session may legitimately migrate
many times and any fixed cap would eventually be wrong.
Redirects are sticky for the life of the loop. draft-19 scopes its MUST to
"the new session" and is silent beyond it, but a redirect is an assignment, not a
detour: reverting to the configured URL would redial the node that just told us
to leave. A fresh
Reconnectstarts from the configured URL again, which is theright granularity and needs no extra bookkeeping.
Redirectis the trust policy (--client-goaway-redirect), defaulting toFollowlike every HTTP client. Beyond #2490's scheme-downgrade check it refusesredirects that widen reachability: a public endpoint pointing at loopback, a
private range, or IPC. #2490 ranked
unixin the top security tier, so anauthenticated upstream could aim the relay at a local socket and have it count as
an upgrade.
The old session drains inside the loop rather than in a detached task, so dropping
the
Reconnecthandle tears it down too.moq-relay
run_remote_onceis nowclient.reconnect(url), deleting the inline retry loop,hardcoded backoff, churn constants, and the relay's own redirect resolver.
Connectionbecomes a builder, matching theWeb::with_shutdown#2490 addedalongside it. This keeps the change additive and resolves #2490's open question
about
mainvsdevon the merits rather than by blast radius.--drain-timeouttakes a humantime duration ("10s", "500ms") like its neighbour--cluster-linger.--cluster-drain-timeoutis gone: the upstream handoverwindow is
--client-goaway-handover, on the loop that actually performs it.moq-native: the handover window is our deadline, not the peer's
--goaway-handoveris a cap, applied whether or not the GOAWAY carries adeadline of its own: whichever comes first wins. The peer's deadline is a
promise about when it force-closes, so waiting past it just holds a dead
session; ours is the cap on how long we keep one around at all, and a peer
naming an hour should not be able to talk us into honoring it.
An earlier revision let the peer's deadline win outright, which left the flag
applying only to a GOAWAY that named none.
GoawayConfig::handovernow takesthe peer's deadline and returns the effective window, so the policy lives in one
place instead of in a caller that has to remember to apply it.
moq-net: a draining session's routes are deprioritized
Ported from #2490 (@ksletmoe-aws's design), reshaped. This was the piece missing
from the earlier revision, and it is the one the original review asked for.
route_orderranks on(announce, cost, hop count, hash), so leaving a drainingroute at its original cost let it stay primary through the hop/hash tie-break and
serve new groups right up until the session closed. Resume is group-granular, so
whatever group was in flight at the force-close was lost. The handover was meant
to happen before that, not at it.
Routes fed by a session now move to
broadcast::DRAIN_COSTwhen its peer drains.Deprioritized, not withdrawn: a broadcast reachable only over the draining
session keeps being served, which is what makes the sender's deadline a handover
window rather than a cutoff. The two mechanisms compose, they don't overlap.
Three things differ from #2490's version:
announce decode against the GOAWAY signal on moq-lite, which means cancelling
a partially-buffered transport read whenever the signal wins; whether that
loses bytes depends on per-backend
readcancel-safety we don't control. OnIETF it only checked the flag when a message arrived, so an idle announce
stream never deprioritized at all. Both wires now do it in the per-source task
they already run, whose
kio::waitraces only in-memory pollables. Symmetric,and a peer that drains then goes quiet is handled on both.
DRAIN_COSTis the largest varint, notu64::MAX. A draining route isstill announced downstream, and lite-06 carries the cost as a QUIC varint,
which caps at 2^62-1.
u64::MAXwould fail to encode on the way out.broadcast::Dynamic. That is the handle each wire's per-sourcetask already holds, so nothing keeps a second producer clone alive (and past
its close-on-last-drop) just to reach the route.
moq-net: a pre-existing route-cost overflow
Found while sizing
DRAIN_COST, and independent of GOAWAY.RouteCost::chargedsaturated at
u64::MAX, but the sum is re-encoded as a varint when forwarded. Apeer may legally advertise the largest varint there is, so adding this link's
price pushed the total out of range and the downstream encode failed with
BoundsExceeded, breaking the announce stream. Reachable onmaintoday by anylite-06 peer. Now saturates at the ceiling the wire can carry.
moq-native:
Redirect::SameHostcompared the authoritySameHostis documented (and named) as allowing a move between ports or schemesbut not to another host. It compared
Url::authority(), which includes the port,so it refused exactly the handoff it advertises. Surfaced by the redirect tests
below.
drafts
moq-lite specified the receiver-side "validate against local policy" rule but not
duplicate handling or whether a client may name a URI, so an implementer had no
way to be conformant. Added both, plus sender-side scheme continuity (matching
draft-19) and the sticky-redirect recommendation. The drain rule above is in as a
SHOULD, and the cost-saturation ceiling is now spelled out (the draft said
"saturating" without naming the cap, which is the bug above).
Resolved: the diamond failover regression
An earlier revision lost six groups across the handover in
cluster_diamond_goaway_seamless_failover. Root-caused and fixed, not silenced.The "a session ending within
backoff.initialcounts as a failed attempt" rulewas applied to the handover as well as to the backoff bookkeeping.
backoff.initialis 1s, and a cluster peer that GOAWAYs shortly after thehandshake falls inside that window, so migration took the failure path: the old
session was dropped rather than kept alive, and the loop slept a second before
dialing the replacement. Nothing served the subscription across that gap.
The two concerns are now split. The handover is unconditional, so the old session
keeps serving until it closes or overstays its window. Only the bookkeeping
branches: an early GOAWAY still records a failed attempt and sleeps the backoff,
so two peers bouncing us between them still escalate and give up. A redirect loop
costs time, not data.
Worth noting for anyone reviewing the earlier revision's reasoning: my first
hypothesis (a pre-existing "replacement far ahead of the group cache" gap) was
wrong. The control experiment settled it — @ksletmoe-aws's original commits pass
this test on current
main, so the regression was mine.I also claimed at one point that #2473 conflicted behaviorally here, having
merged onto it and seen all three GOAWAY cluster tests hang. That was my own bug,
not an interaction: the experiment ran against the pre-fix code, whose early-GOAWAY
path tore the old session down, and #2473's stricter route selection turned the
resulting gap into a hang instead of a skip. Now that #2473 has landed, this
branch is rebased on it and all three pass.
Test plan
just checkclean (workspace tests, clippy,cargo doc -D warnings, taplo,remark);
just drafts checkparses.just rs loom5/5, run because this touchesmodel/.goaway_drains_routes_{moq_lite_04,moq_transport_19}(the peer sendsnothing after the GOAWAY, so these only pass if the subscriber acted on the
signal itself),
drain_migrates_best_route, plus drain unit tests and theredirect-policy tests.
test that covers it, so none of them are vacuous.
just test smoke-full. The js/net codecchanges from feat(net, relay): GOAWAY with graceful drain and cluster migration #2490 are decode-only and untouched here.
Still open
run_remoteon top ofReconnect'sown. Functional but redundant; collapsing it means giving cluster peers an
unlimited
backoff.timeout.Status::Migratingdoes not cross the libmoq FFI boundary (OnStatusonlyreports
Connected). Additive in Rust; needs a decision on whether C consumersshould see it, which would touch
moq.h,cpp/obs, anddoc/lib/*.a mirror since
@moq/netshares no code withmoq-net.(Written by Claude Opus 5)