feat(net): fail over across redundant publishers via per-peer route selection

[kixelated]

Addresses #2461 (the mesh-propagation half; the remaining pieces are listed at the bottom, so this deliberately does not auto-close the issue).

Root cause

Active/active (1+1) source failover never worked across a mesh because the standby route was never presented to the node that needed it, and could not have been served even if it were:

1. A relay announces only its single best route per path, filtered per peer. The announce loop reads the front's one advertised route (broadcast.route()) and skips the announce entirely when that chain contains the peer's exclude_hop (prepare_active_hops). So relay B, actively carrying pubA → relayA → relayB, advertised nothing to relay A even while holding a perfectly good standby pubB. Relay A's origin table then had nothing to reselect onto when pubA died: permanent freeze, exactly the repro in Active/active source failover: a standby publisher's route is never presented to the relay serving the active source (mesh), so a dead source is not failed over — even with moq-lite-06 cost routing #2461. Cost routing (feat(net): route by cumulative cost on lite-06 announcements #2424) could not help because pricing cannot rank a route that is never propagated.
2. Serving was not exclusion-aware. Track requests always dispatch to the front's global active source. Had relay B advertised its standby to relay A, relay A's failover subscription would have been served from relay B's active source, whose data flows through relay A: a subscription cycle.
3. Content identity was enforced inconsistently. A restart with a different first hop on one session is a deliberate teardown (entry.publisher != publisher), but the same event across two sessions silently attached as a splice-compatible standby in attach_source, with no identity check at all.

The fix

One selection function, used for both announcing and serving: the best route whose hop chain excludes the requesting peer.

• Per-peer announce selection (lite/publisher.rs): the front mirrors its full route table (metadata only, preference-ordered, serving route first) onto the spliced broadcast (broadcast::Producer::set_routes, crate-private). Each announce stream selects from the table with its peer's exclude_hop, so a peer the serving chain flows through receives the best standby instead of nothing. The winner changing per peer rides the existing restart machinery; swinging into/out of existence rides the existing filtered-announce arms, which already handled per-peer subsetting.
• Data-plane split horizon (model/origin.rs): sessions resolve broadcasts through a crate-private origin::Consumer::excluding(peer). The peer's identity arrives in its SETUP via a new additive Origin parameter (0x5), declared by each endpoint at session start, so the exclusion applies even to sessions that never open an announce stream. The shared front is handed out only when no attached route is tainted for the requester, since the front picks a source per track and re-picks on failover, and resolving registers that peer on the front (ExclusionGuard, refcounted, released when its last broadcast::Consumer clone drops) so route selection keeps avoiding routes through them for as long as they read it, falling back only when no clean route remains; otherwise the lookup returns the best clean source directly. When only tainted routes exist the subscription is unroutable, and resolving distinguishes that from an unannounced path (Resolved::{Found, Excluded, Missing}) so a dynamic handler is never asked to route around the exclusion. Together with truthful advertised chains this is path-vector loop freedom: any would-be cycle surfaces the requester's own id inside a candidate chain, where the filter removes it — at any cycle length, which plain immediate-peer split horizon cannot guarantee.
• First-hop content identity, enforced at the front (model/origin.rs): attach_source now rejects a source whose first hop differs from the incumbent's. The rejected source parks invisibly and takes over the path once the incumbent ends (first wins, successor succeeds it), instead of silently splicing unrelated content into live subscribers. The session-level restart rule was already first-hop-keyed; this makes the cross-session path agree with it. run_source applies the same check to route updates that change their own first hop.
• Cost stays honest per peer: the actively-carrying zero-cost discount now applies only when the advertised route is the one actually being served (SentRoute.serving). A standby advertised to an excluded peer keeps its accumulated cost, because serving that peer means opening a fresh ingest. (This also fixes a busy-spin the naive version would have had: the demand watch armed on any nonzero sent cost, expecting the recompute to discount it to zero, which a standby never does.)
• A track a joining standby cannot carry yet (model/origin.rs, model/broadcast.rs): a standby wins dispatch the moment it attaches (one local hop beats two remote ones), which is before a real publisher has created every track. serve_track used to re-splice onto the newcomer, count the rejection as a strike, retry the same source three times back to back, and abort the logical track as Unroutable, tearing down a subscription the incumbent route was serving; the aborted entry then stayed in spliced.tracks forever, so the name was dead for the life of the front. Whether a source carries a given track is a per-track property, so a refusal now rules that source out of that track only: FrontState::serve_route prefers the active source, falls back to the next-best route this track has not been refused by, and only a sweep in which every attached source refuses spends a strike (the sweep then clears the refusals, so a publisher that has since created the track is re-tried). The served route leaving the table closes a sweep too, which is the failover case. broadcast::Consumer::track drops an aborted spliced entry the way the plain lookup already reclaims a closed one. This is a pre-existing bug (serve_track is otherwise untouched here, and it reproduces on main), but it is what blocked the end-to-end drill, so it lands with the mechanism it blocks.
• moq --origin <id> (moq-cli): pins the process origin id (default: fresh random per run). Redundant 1+1 publishers share an id to declare their content interchangeable; a crash-restarted encoder keeps getting a fresh id and is correctly treated as new content. The relay already had this via --cluster-id. Sharing an id is a promise that both publishers produce exactly the same tracks with aligned group sequences (now spelled out in doc/bin/cli.md); independent publishers must use distinct ids, and the newcomer waits as a replacement.

With this, the #2461 repro works end to end: relay B advertises [pubB] to relay A; pubA dies; relay A splices to the standby at a group boundary; the restart that propagates keeps the first hop, so no downstream node tears anything down; relay A's subscription to relay B is dispatched to pubB even while relay B's own table still holds the stale via-relay-A entry (no loop window).

Wire / draft

Two wire changes: a SETUP Origin parameter (0x5) declaring each endpoint's Hop ID at session setup (additive: SETUP receivers ignore unknown parameter IDs, so it is backward-compatible with deployed lite-05 peers), and lite-06-wip drops ANNOUNCE_REQUEST's per-stream Exclude Hop in its favor (lite-06 is unreleased and negotiates only by explicit opt-in, so this breaks no published contract and the PR still targets main). Lite-04/05 keep the field on the wire and behave exactly as before. The normative changes are specified in drafts/draft-lcurley-moq-lite.md in this PR: the Origin parameter, per-subscriber advertisement selection, the serve-by-the-same-rule requirement, shared-first-hop interchangeability (redundant publishers MAY share a Hop ID), and the serving-only scope of the carrying discount. drafts/draft-lcurley-moq-relay-hops.md (the IETF moq-transport extension) is updated to match: the RELAY_HOPS Setup Option value now declares the endpoint's own Hop ID, replacing the EXCLUDE_HOP parameter, with the same per-session selection, serve-by-the-same-rule, and shared-first-entry interchangeability semantics.

Behavioral changes to existing public APIs (no signature changes)

• origin::Producer::create_broadcast: a second source at a path with a different first hop no longer joins as a standby; it parks until the incumbent ends. Sources with the same first hop (or both hopless/local) behave exactly as before.
• What a given peer observes on its announce stream can now include a route today's code would have suppressed entirely (that is the feature).

Cross-package sync

• js/net: mirrored the wire changes. AnnounceRequest carries excludeHop only on draft-04/05, and the client declares its session origin via the SETUP Origin parameter so relays keep suppressing reflected announces on lite-06. No selection/dispatch logic to mirror: JS is a leaf with no multi-route origin table.
• moq-ffi / bindings: no surface change. Exposing an origin-pin knob to bindings publishers is a natural follow-up but not needed for the core mechanism.
• moq-relay: no change; --cluster-id already pins the relay origin.
• Docs: doc/bin/cli.md gains --origin and a "Redundant Publishers (1+1)" section.

Tests

• test_publisher_mismatch_parks: different first hop parks, takes over on incumbent end as unannounce + announce (also regression-covers the orphaned-leaf bug found while writing it: a parked source now re-resolves its tree leaf per attach attempt, since the incumbent's teardown can prune it).
• test_exclusion_holds_when_a_tainted_route_attaches_later: the peer resolves while the table is clean, a cheaper route back through them attaches after, and the front stays off it until the peer's last handle drops.
• test_per_track_fallback_respects_exclusion: the clean route is active but cannot carry the track, so the front's fallback is the peer's own route; the peer is pinned to the clean source and never reaches it.
• test_exclusion_survives_failover_onto_a_tainted_route: the peer resolves while a clean route is active, that route dies, and its subscription still never migrates onto the route through itself.
• test_excluded_path_never_reaches_the_dynamic_handler: an all-tainted announced path is Unroutable with a live handler registered, while a genuinely missing path still falls back to it.
• test_dispatch_excludes_requester: tainted-for-the-peer active → peer pinned to the clean source, everyone else on the shared front.
• test_dispatch_all_tainted_unroutable: all routes through the requester → Unroutable.
• excluded_peer_receives_the_standby (wire): peer in the active chain receives the standby chain at its undiscounted cost while an ordinary peer receives the active chain at the warm discount.
• retracts_when_route_swings_through_peer (wire): route swinging into the peer's chain → ANNOUNCE_END; swinging back → fresh ANNOUNCE_START.
• standby_attach_announces_to_excluded_peer (wire): a relay carrying via its peer initially advertises nothing to them; a same-publisher standby attaching later goes out as a fresh announce, and the active source dying afterward is invisible on that stream (the standby is already advertised). Unit shape of the e2e drill's finding 1.
• test_standby_missing_track_keeps_incumbent: the standby refuses the track more times than the strike budget (a publisher whose demuxer has not reached it yet); the live subscription keeps flowing from the incumbent instead of aborting, and splices onto the standby once the incumbent dies and the standby has the track.
• test_unservable_track_retried_by_a_later_request: a track every source refuses still aborts as Unroutable, but a later request re-asks the sources instead of resolving from the cached corpse.
• test_standby_join_splices_live_subscriber: a local same-first-hop standby joining a front that is carrying from a peer splices the live subscription at the group boundary (finding 2, with the aligned content that a shared origin id requires).
• test_dispatch_excludes_requester now also asserts the via-peer source receives no track request at all: a track request on that source is exactly what a session would forward upstream as a SUBSCRIBE, so this is the "never subscribe back to the peer the data came from" guarantee.
• announce_request_exclude_hop_by_version + js mirrors: the field round-trips on lite-04/05 and is absent from lite-06's wire image.
• Existing multi-source tests updated to share first hops, which is now the documented precondition for standby splicing. That includes broadcast_route_migration (caught by CI, not just check, which skips the moq-native integration binaries): it encoded the pre-identity contract of silently migrating across different first hops, and now stamps a shared publisher id, which is the behavior the migration feature promises.

Not in this PR

Follow-ups toward the full #2461 acceptance test: checking in the two-relay integration drill (the issue authors offered their cluster_failover.sh; it needs a timeline longer than the QUIC idle timeout, see below), an origin-pin knob for the bindings/gateways (moq-ffi, gst, rtmp/srt ingest), and subscriber multi-homing (option B in the issue).

Failover latency

A relay cannot tell a dead publisher from a quiet one: it does not know whether frames are supposed to be flowing for a given broadcast, so it waits for the transport to declare the peer gone. With an ungracefully killed publisher (no CONNECTION_CLOSE) that is the QUIC idle timeout, 30s by default, and the failover cannot start before it. Lowering that default is worth doing but is a separate change with its own history; --server-quic-idle-timeout moves it in the meantime.

(Written by Fable 5, extended by Opus 5)

[sourcery-ai]

Sorry @kixelated, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The change adds optional fixed origin IDs to the CLI and documents redundant publisher operation. Lite SETUP now carries endpoint origins, while ANNOUNCE_REQUEST Exclude Hop handling is limited to supported versions. Broadcast state retains complete ordered route tables with change polling. Origin routing filters excluded paths, preserves exclusions across consumers, and parks sources with different publisher identities. Lite publisher announcements select routes per peer, track serving paths, apply serving-only demand discounts, and recompute on route changes. Tests cover failover, split-horizon dispatch, route selection, and announcement retraction.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title matches the main change: per-peer route selection for redundant publisher failover.
Description check	✅ Passed	The description is strongly related to the changeset and covers the main fixes and supporting updates.

✨ Finishing Touches

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/moq-lite-duplicate-announcements-34ef49

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@doc/bin/cli.md`:
- Around line 75-76: Update the Redundant Publishers reference in the --origin
documentation to use the heading’s actual fragment, `#redundant-publishers-11`, so
the in-page link resolves and satisfies MD051.

In `@drafts/draft-lcurley-moq-lite.md`:
- Line 298: Replace the em dash in the advertisement constraint sentence with a
colon or other permitted punctuation, preserving the existing meaning and
wording.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e9b13394-e2cb-4291-bed4-03501ed712af

📥 Commits

Reviewing files that changed from the base of the PR and between ff306e5 and baeb69b.

📒 Files selected for processing (7)

• doc/bin/cli.md
• drafts/draft-lcurley-moq-lite.md
• rs/moq-cli/src/args.rs
• rs/moq-cli/src/main.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/model/broadcast.rs
• rs/moq-net/src/model/origin.rs

[t0ms]

TL;DR

Thanks for this — the per-peer selection design reads well and the model/wire unit tests
are convincing. We picked up the PR to run the end-to-end two-relay drill you list under
"Not in this PR", and wanted to share results early:

• ✅ Unit layer is solid. test_route_failover, test_dispatch_excludes_requester,
  test_dispatch_all_tainted_unroutable, test_publisher_mismatch_parks and the wire
  advertisement tests all pass on our build.
• ❌ End-to-end, a single-homed subscriber does not yet fail over in a real
  two-moq-relay mesh: the standby route is still never presented to the relay serving
  the active source, so there is nothing to reselect onto when the active publisher dies
  — the same symptom Active/active source failover: a standby publisher's route is never presented to the relay serving the active source (mesh), so a dead source is not failed over — even with moq-lite-06 cost routing #2461 describes. The per-peer selection landed, but the standby
  announcement never reaches the peer.
• ⚠️ A reproducible Unroutable teardown when a shared---origin standby publisher
  joins a relay that is already carrying the broadcast from its peer (details below).

We don't think either is a reason to hold the PR (the mechanism it lands is a prerequisite
and is well-tested); they're the next e2e step.

Setup note: this PR predates #2469

#2473 branched before #2469 ("linger a broadcast across an ungraceful source loss",
merged to main overnight). The two interact: the failover splice needs the front to
linger across the active source's loss, which is exactly what #2469 adds — without it
the front closes synchronously on source detach and there is nothing to splice into (and
moq export still dies with json: dropped). We merged main into the PR branch locally
(one small conflict in run_source's loop, resolved by keeping the per-peer arms and
threading source.is_finished() into the linger-aware detach_source); the full
moq-net suite — both the linger tests and this PR's failover tests — stays green. All the
e2e results below are on that merged build, so the linger is present and is not the
cause. Recommend rebasing #2473 on main.

Drill

[cluster_failover.sh](https://github.com/user-attachments/files/30343689/cluster_failover.sh) cluster_failover.sh (attached) builds the exact topology your root-cause section
describes — relay B actively carrying pubA → relayA → relayB while also holding a local
standby pubB, both publishers sharing --origin:

pubA ──▶ relayA(:4443) ◀──cluster──▶ relayB(:5443) ◀── pubB
          ▲   ▲                          ▲
        sub1 sub2                       sub3     (sub3 makes relayB carry via relayA)

Timeline: bring everything up; at t=10 pubB joins relay B; at t=22 kill pubA; watch
sub1 on relay A. PASS = sub1 keeps growing after t=22.

Finding 1 — the standby route never reaches the relay serving the active source

Across every topology we tried (relay B dials A only; full mesh A↔B; and with/without a
subscriber on relay B to force it to carry), relay A never held a second route for the
broadcast. With debug logging, relay A shows only its local pubA route; there is no
remote-hop (relay B) route, no reselect, nothing to promote. When pubA dies, sub1
freezes permanently:

t,sub1,sub3,event
21,24435112,...,
22,24844388,...,KILL_pubA
...            (frozen)
43,24844388,...,

Our reading: the per-peer advertisement selection you added decides what to
advertise when a relay advertises a route to a peer — but a relay only opens an
announce stream for a broadcast toward a peer that has expressed announce-interest in
it. The node serving the active source (relay A) is fully satisfied by its local pubA, so
it never solicits the broadcast from relay B, so relay B never advertises the pubB
standby to relay A. Chicken-and-egg: relay A won't learn the standby until it needs it, and
it can't fail over because it never learned the standby exists. (Cross-relay on-demand
forwarding works fine — control test: a subscriber on relay B with no local publisher pulls
pubA across the mesh at full rate — so this is specifically about pre-positioning a
standby, not about the data plane.)

Is the intended design that the active-serving relay proactively holds all peer-advertised
routes for broadcasts it serves (i.e. the interest is unconditional across the cluster), or
is a subscriber-multi-homing / explicit-standby-subscription step still expected? That's
the piece our drill can't make happen from the outside today.

Finding 2 — Unroutable teardown when a shared-origin standby joins a carrying relay

With pubA and pubB sharing --origin 424242, a subscriber on relay B (sub3,
which relay B is serving by pulling pubA across the mesh) is torn down the instant
pubB joins relay B:

WARN moq_net::lite::subscriber: track info failed broadcast=red.hang track=4.ts err=remote error: code=30
Error: moq: unroutable

code=30 is Error::Unroutable. So when a local source appears whose first hop equals the
first hop of the route relay B is already carrying from its peer (the shared-origin
"interchangeable" case this PR is built around), relay B's dispatch for its existing local
subscriber resolves to Unroutable rather than splicing onto the new local source. With
independent origin ids sub3 instead survives until pubA dies (then the pre-#2469
teardown applies). This smells like an edge in the exclusion/excluding() dispatch when a
local same-first-hop source coexists with a peer route carrying that same first hop.

What works (so this is balanced)

• All the model/wire unit tests above.
• Cross-relay on-demand forwarding (control): sub-on-relayB pulls pubA across the mesh,
  ~21.8 MB in 18 s, no loss.
• The --origin CLI knob and doc/bin/cli.md "Redundant Publishers (1+1)" section are
  clear and were easy to use.

Questions

1. Intended announce-interest model for standbys (Finding 1): unconditional cluster-wide
   holding of peer routes, or does subscriber multi-homing (option B in Active/active source failover: a standby publisher's route is never presented to the relay serving the active source (mesh), so a dead source is not failed over — even with moq-lite-06 cost routing #2461) remain the
   single-homed path?
2. Is Finding 2 (Unroutable on shared-origin local-standby join at a carrying relay) a
   known edge, or worth a repro test?
3. Would you like cluster_failover.sh as a checked-in integration drill (e.g. under
   test/)? It already prints a PASS/FAIL and takes MOQ/RELAY/SRC/OID env knobs.

[coderabbitai]

🧹 Nitpick comments (1)

rs/moq-net/src/model/origin.rs (1)

618-647: 🩺 Stability & Availability | 🔵 Trivial

Confirm the shared-origin Unroutable teardown is covered.

Reviewers observed an Unroutable teardown for a subscriber when a shared-origin standby joined a relay already carrying the broadcast from its peer (not seen with distinct origin IDs). That surfaces here: when the active chain contains the requester, resolution falls to dispatch(Some(origin)) and yields None (→ Unroutable) if every candidate route is tainted. The new tests cover the excluded-peer and all-tainted cases in the abstract, but not the shared-origin standby-join scenario the reviewers hit. Want me to draft a regression test reproducing that topology, or open an issue to track it?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rs/moq-net/src/model/origin.rs` around lines 618 - 647, Add a regression test
for consume_broadcast covering a shared-origin standby joining a relay already
carrying the broadcast from its peer. Build the topology with the shared origin,
relay, active peer route, and standby subscriber, then assert the standby
remains routable and does not receive an Unroutable teardown; update
consume_broadcast or dispatch only if the test exposes the described all-tainted
failure.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rs/moq-net/src/model/origin.rs`:
- Around line 618-647: Add a regression test for consume_broadcast covering a
shared-origin standby joining a relay already carrying the broadcast from its
peer. Build the topology with the shared origin, relay, active peer route, and
standby subscriber, then assert the standby remains routable and does not
receive an Unroutable teardown; update consume_broadcast or dispatch only if the
test exposes the described all-tainted failure.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1a36e294-d829-47ad-8854-c8fbb192f01f

📥 Commits

Reviewing files that changed from the base of the PR and between 7c976cd and 0165ec9.

📒 Files selected for processing (7)

• doc/bin/cli.md
• drafts/draft-lcurley-moq-lite.md
• rs/moq-cli/src/args.rs
• rs/moq-cli/src/main.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/model/broadcast.rs
• rs/moq-net/src/model/origin.rs

[kixelated]

Our reading: the per-peer advertisement selection you added decides what to advertise when a relay advertises a route to a peer — but a relay only opens an announce stream for a broadcast toward a peer that has expressed announce-interest in it. The node serving the active source (relay A) is fully satisfied by its local pubA, so it never solicits the broadcast from relay B, so relay B never advertises the pubB standby to relay A. Chicken-and-egg: relay A won't learn the standby until it needs it, and it can't fail over because it never learned the standby exists. (Cross-relay on-demand forwarding works fine — control test: a subscriber on relay B with no local publisher pulls pubA across the mesh at full rate — so this is specifically about pre-positioning a standby, not about the data plane.)

That's not true, we always announce-interest to every peer.

One of the problems is that we currently announce the best route to a peer, and we purposely skip doing it if the peer is in the hop list. If there's two valid routes (relay with 2 connections, each advertising a different "best" route), then we can fail to publish the better route to the peer.

That being said, it doesn't explain the failure you're seeing, as relay A/B should both try to announce to each other.

Is the intended design that the active-serving relay proactively holds all peer-advertised routes for broadcasts it serves (i.e. the interest is unconditional across the cluster), or is a subscriber-multi-homing / explicit-standby-subscription step still expected? That's the piece our drill can't make happen from the outside today.

I think we should only advertise a single route to a peer. Not only does this prevent explosive growth, but it also matches how SUBSCRIBE cannot choose a route. Like if we advertised 10 routes to a peer... it has no way of picking which one to use.

Finding 2 — Unroutable teardown when a shared-origin standby joins a carrying relay

With pubA and pubB sharing --origin 424242, a subscriber on relay B (sub3, which relay B is serving by pulling pubA across the mesh) is torn down the instant pubB joins relay B:

WARN moq_net::lite::subscriber: track info failed broadcast=red.hang track=4.ts err=remote error: code=30
Error: moq: unroutable

code=30 is Error::Unroutable. So when a local source appears whose first hop equals the first hop of the route relay B is already carrying from its peer (the shared-origin "interchangeable" case this PR is built around), relay B's dispatch for its existing local subscriber resolves to Unroutable rather than splicing onto the new local source. With independent origin ids sub3 instead survives until pubA dies (then the pre-#2469 teardown applies). This smells like an edge in the exclusion/excluding() dispatch when a local same-first-hop source coexists with a peer route carrying that same first hop.

Yeah, we should splice if the first hop is shared.

2. Is Finding 2 (Unroutable on shared-origin local-standby join at a carrying relay) a
   known edge, or worth a repro test?
3. Would you like cluster_failover.sh as a checked-in integration drill (e.g. under
   test/)? It already prints a PASS/FAIL and takes MOQ/RELAY/SRC/OID env knobs.

Unit tests for both.

[kixelated]

@t0ms both publishers MUST be publishing identical content when using the same origin/name. If they're not, like it's two different publishers slurping the same TS feed, then the subscribers MUST get a signal to reinitialize on route change. A few ways of doing this:

1. Use different broadcast names (active/passive on receiver side).
2. Use a different origin. But it's not great because different subscribers will get different content based on their geo.
3. Ensure identical content. Otherwise subscriptions can't be seamlessly migrated.

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

doc/bin/cli.md (1)

102-108: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Do not document failover as guaranteed before the two-relay path is fixed or covered by end-to-end testing.

The supplied PR context reports a two-relay drill where no standby route arrived and playback froze after the active publisher stopped. Either resolve that path and add a passing regression test, or qualify this documentation instead of promising uninterrupted viewer failover.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@doc/bin/cli.md` around lines 102 - 108, Qualify the failover claims in the
“Redundant Publishers (1+1)” documentation so uninterrupted viewer continuity is
not presented as guaranteed. Alternatively, only retain the current guarantee
after fixing the two-relay standby-route path and adding a passing end-to-end
regression test covering active-publisher failure.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@doc/bin/cli.md`:
- Around line 110-115: Update the relay splice timing statement in the
sharing-id documentation so it consistently states that failover or splicing
occurs at a group boundary, matching the earlier documented behavior. Preserve
the surrounding requirements about identical tracks, content, and aligned group
sequences.

In `@drafts/draft-lcurley-moq-lite.md`:
- Around line 293-295: Clarify that origin or peer-identity filtering applies
only when a nonzero, declared identity exists: in
drafts/draft-lcurley-moq-lite.md:293-295 update announcement selection, :306-307
update subscription-source selection, and :704-705 update ANNOUNCE_REQUEST
behavior; in drafts/draft-lcurley-moq-relay-hops.md:168-174 state that an empty
RELAY_HOPS value disables peer-identity filtering.
- Line 308: Update the paragraph’s “Exclude-Hop filter” reference to
“declared-origin filter,” preserving the explanation that the filter prevents
subscription cycles by removing candidates containing the subscriber’s own Hop
ID. Do not refer to the removed wire field.

In `@js/net/src/lite/announce.test.ts`:
- Around line 109-123: Update the AnnounceRequest test to assert reader.done()
after each decode for both draft-04/05 iterations and the draft-06 case,
ensuring the entire encoded frame is consumed while preserving the existing
excludeHop assertions.

---

Outside diff comments:
In `@doc/bin/cli.md`:
- Around line 102-108: Qualify the failover claims in the “Redundant Publishers
(1+1)” documentation so uninterrupted viewer continuity is not presented as
guaranteed. Alternatively, only retain the current guarantee after fixing the
two-relay standby-route path and adding a passing end-to-end regression test
covering active-publisher failure.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1292563c-b1c7-46c4-a666-4b6a3d600018

📥 Commits

Reviewing files that changed from the base of the PR and between 0165ec9 and fdcab8f.

📒 Files selected for processing (18)

• doc/bin/cli.md
• drafts/draft-lcurley-moq-lite.md
• drafts/draft-lcurley-moq-relay-hops.md
• js/net/src/lite/announce.test.ts
• js/net/src/lite/announce.ts
• js/net/src/lite/connection.ts
• js/net/src/lite/setup.test.ts
• js/net/src/lite/setup.ts
• rs/moq-cli/src/args.rs
• rs/moq-native/tests/broadcast.rs
• rs/moq-net/src/client.rs
• rs/moq-net/src/lite/announce.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/lite/session.rs
• rs/moq-net/src/lite/setup.rs
• rs/moq-net/src/lite/version.rs
• rs/moq-net/src/model/origin.rs
• rs/moq-net/src/server.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• rs/moq-cli/src/args.rs
• rs/moq-net/src/model/origin.rs

[t0ms]

thanks for adopting the drill and fixing both findings at the model layer — on the rebased head the live two-relay drill still doesn't fail over, under lite-05 and lite-06-wip

relay A (serving the local active source) never opens an announce-interest toward relay B for the broadcast, so relay B's excluded_peer_receives_the_standby path never actually fires on the wire so the model tests aren't yet exercising the live standby-pre-positioning / splice path.

is the intended design that an active-serving relay unconditionally holds peer-advertised standby routes for broadcasts it serves, or is subscriber-multi-homing still the single-homed path?

sizes.csv
relayA.log
relayB.log

[t0ms]

Finding 2 still reproduces live despite its new passing model test (test_standby_join_splices_live_subscriber): the shared---origin sub3 on the carrying relay still dies Unroutable code=30 when the local standby joins — i.e. the model test and the live dispatch diverge

[kixelated]

Picked this up and ran your drill against the PR head with two real moq-relay processes (synthetic TS source, moq_net at debug), plus a control run against main. Short version: finding 1 isn't a bug and the drill can't observe the failover; finding 2 is real, is not caused by this PR, and is now fixed here.

Finding 1: the standby does reach the peer

Relay A's log, the instant pubB joins relay B:

DEBUG moq_net::lite::subscriber: announce broadcast=red.hang hops=2

Relay B selects the standby for the excluded peer and sends it; relay A attaches it as a second route with no parking warning. Before pubB joins, relay B logs no advertisable route for this peer exclude_hop=11111 and sends nothing, which is correct. Announce-interest is unconditional, so there is no chicken-and-egg.

What the drill actually hits is its own timeline. kill on the publisher never sends CONNECTION_CLOSE, so relay A holds pubA's session until the QUIC idle timeout (DEFAULT_IDLE_TIMEOUT, 30s). Relay A logs nothing at all between the kill and connection closed err=timed out 30s later. The drill kills at t=22 and grades at t=43, so it grades 21s into a 30s timeout: no build can pass it.

Removing that and finding 2 (video-only source, so the standby has its one track ready at connect; window extended past the timeout):

build	result
this PR	PASS, sub1 resumes at t=53, 31s after the kill
main, identical drill	FAIL, frozen for the full 68s window

So the mesh half of #2461 does work end to end here.

On the detection latency: a relay can't do better than the QUIC keep-alive today. It has no idea whether frames are supposed to be flowing for a given broadcast, so it can't treat silence as failure the way an application that knows its own cadence could; it has to wait for the transport to declare the peer gone. Lowering the default idle timeout (10s or so) is probably the right call and we may do it, but it has bitten us before, so it's a separate change rather than something to rush into this PR. If you want a faster drill in the meantime, --server-quic-idle-timeout on the relays moves it.

Finding 2: real, pre-existing, fixed in this PR

Reproduced exactly as you described, and the root cause is one layer below routing.

The standby wins dispatch the moment it attaches (one local hop beats two remote ones), which is before a real publisher has created every track: moq import announces its broadcast on connect and only creates each track once its demuxer reaches it. In the drill pubB had 0.avc3 but not yet 0.aac:

16:55:45.714  subscriber: track info failed track=0.aac err=remote error: code=30
16:55:45.715  (x3, all within 1ms)
16:55:45.715  origin_impl: aborting unservable track name=0.aac
16:55:45.715  publisher: subscribed error track=0.aac err=unroutable

serve_track re-spliced onto the newcomer, counted the rejection as a strike, retried the same source three times back to back with no fallback to the incumbent, and aborted the logical track. Worse, the aborted entry stayed in spliced.tracks (which, unlike the plain track lookup beside it, had no liveness check), so 0.aac was dead for the life of that front, on relay B and on pubB's own front. That's why relay A's eventual failover 30s later still only recovered video.

Two notes on attribution: serve_track is untouched by this PR, and the control run reproduces the same t=10 teardown on main (it surfaces as json: dropped rather than unroutable). And the model test passing while the field diverged is exactly because the test's standby accepts the track request immediately, which a real publisher doesn't.

The fix, in this PR: whether a source carries a given track is a per-track property, so a refusal rules that source out of that track only. FrontState::serve_route picks the front's active source unless this track's refusals rule it out, then the next-best route. Only a sweep in which every attached source refuses spends a strike, and the sweep clears the refusals so a publisher that has since created the track gets re-tried. The served route leaving the table closes a sweep too, which is the failover case. And broadcast::Consumer::track now drops an aborted spliced entry the way the plain lookup already reclaims a closed one, so the verdict belongs to the sources attached at the time rather than to the name forever.

Two regression tests cover it: test_standby_missing_track_keeps_incumbent (standby refuses more times than the strike budget; the subscription keeps flowing from the incumbent, and splices onto the standby once the incumbent dies and the standby has the track) and test_unservable_track_retried_by_a_later_request (a track every source refuses still aborts, but a later request re-asks instead of getting the cached corpse).

With both in, your original drill (audio+video, window extended past the idle timeout) passes end to end: sub3 survives pubB's join and keeps growing, and sub1 fails over. Relay logs show exactly one source refused track name=0.aac and zero unroutable.

Your questions

1. Announce-interest is already unconditional across the cluster, and we deliberately advertise a single route per peer: SUBSCRIBE can't pick among routes, so advertising N of them gives the peer nothing to do with them, and it grows explosively in a real mesh. Subscriber multi-homing (option B) is still open, but it isn't needed for the single-homed path.
2. Covered by the two unit tests above.
3. Yes, please, and thank you for the drill. It found a real bug we would not have found from unit tests. Worth checking in with the timeline made explicit about the idle timeout, so a future reader doesn't rediscover the 30s window the hard way; happy to take it as a PR against test/.

(written by Opus 5)

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@drafts/draft-lcurley-moq-lite.md`:
- Line 1322: Update the Hop IDs and origin-based announcement filter paragraph
to remove the claim that publishers are not required to honor the filter.
Preserve the privacy disclaimer by stating that the filter is for loop
avoidance, is not access control, and must not be relied upon to hide
broadcasts; keep the normative subscription requirement consistent with the rule
referenced near the Origin Parameter section.

In `@rs/moq-net/src/model/broadcast.rs`:
- Around line 431-439: Update set_routes to explicitly clear state.route with
the crate’s no-route value when routes is empty, incrementing route_epoch when
this changes, while preserving the existing active-route update for non-empty
tables. Add a regression test covering a non-empty to empty transition and
verifying Consumer::route() and poll_route_changed() no longer expose the
detached route.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b5ea7fc3-aa19-4546-84ed-010eba791385

📥 Commits

Reviewing files that changed from the base of the PR and between fdcab8f and 2916172.

📒 Files selected for processing (21)

• doc/bin/cli.md
• drafts/draft-lcurley-moq-lite.md
• drafts/draft-lcurley-moq-relay-hops.md
• js/net/src/lite/announce.test.ts
• js/net/src/lite/announce.ts
• js/net/src/lite/connection.ts
• js/net/src/lite/setup.test.ts
• js/net/src/lite/setup.ts
• rs/moq-cli/src/args.rs
• rs/moq-cli/src/main.rs
• rs/moq-native/tests/broadcast.rs
• rs/moq-net/src/client.rs
• rs/moq-net/src/lite/announce.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/lite/session.rs
• rs/moq-net/src/lite/setup.rs
• rs/moq-net/src/lite/version.rs
• rs/moq-net/src/model/broadcast.rs
• rs/moq-net/src/model/origin.rs
• rs/moq-net/src/model/resume.rs
• rs/moq-net/src/server.rs

🚧 Files skipped from review as they are similar to previous changes (16)

• rs/moq-net/src/lite/version.rs
• js/net/src/lite/connection.ts
• js/net/src/lite/setup.test.ts
• js/net/src/lite/announce.test.ts
• doc/bin/cli.md
• rs/moq-cli/src/main.rs
• rs/moq-net/src/server.rs
• rs/moq-cli/src/args.rs
• js/net/src/lite/announce.ts
• rs/moq-net/src/lite/session.rs
• rs/moq-net/src/lite/setup.rs
• rs/moq-native/tests/broadcast.rs
• js/net/src/lite/setup.ts
• rs/moq-net/src/lite/announce.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/model/origin.rs

[t0ms]

Confirmed on cc11cbaf, and you were right on both counts.

Finding 1 — withdrawn, it was our drill. A hard kill sends no
CONNECTION_CLOSE, so the relay keeps serving the dead source until the QUIC
idle timeout expires and logs nothing in between. Our drill killed at t=22 and
graded at t=43 — 21 s into a 30 s timeout, so it could not have passed on any
build, including one with a perfect reselect. Once the window is derived from the
detection budget instead of hard-coded, the failover is there:

CHECK 1 failover: sub1 at kill(t=32)=5293892 -> t=81=8647624
PASS: sub1 resumed at t=65, 33s after the kill (+3353732 bytes)

30–33 s across runs, i.e. one idle timeout — detection dominates and the reselect
itself is essentially free.

Finding 2 — fixed. sub3 now survives the shared-origin standby joining the
carrying relay, zero Unroutable.

One caveat worth a look before this lands

The standby join is not transparent. sub3 is on relayB, which is only carrying
the broadcast via relayA, and the instant pubB attaches locally it stops dead
for 8–9 s before recovering at full rate. Per-second bytes, pubB joins at t=10:

  9    1512836   175028
 10    1674892   162056  pubB_join
 11    1764944    90052
 12    1764944        0
 ...                     <- 8s of nothing
 19    1764944        0
 20    1778480    13536
 21    1903876   125396
 22    2083228   179352

Consistent across every run on this head. Much better than being torn down, but a
viewer on a relay that merely carries the broadcast still loses 8–9 s because a
redundant publisher appeared somewhere else — which is the opposite of what
adding redundancy is supposed to do, and in a 1+1 setup the standby attach is a
routine event. I have not dug into whether the splice is re-requesting tracks
from scratch or waiting on the new source's first keyframe/catalog.

Separate, possibly unrelated

Killing tsp first in a tsp | moq import pipeline makes the import see EOF and
shut its broadcast down cleanly. On that graceful path sub1's export ts
does not fail over at all — it exits with TS track layout changed after PAT/PMT was emitted: '0.avc3' removed, even though the standby is announced and the
relay unannounces immediately rather than waiting out any timeout. Graceful
departure of an active source is the common production case (SIGTERM on an
encoder, a rolling restart), so it seems worth its own look. Happy to open an
issue if it is not already covered.

Drill

Submitted as #2545 — just test failover, with the timeline and kill semantics
documented since both of those are what led us to report a bug that was not there.
It depends on moq --origin from this PR, so it wants to land with or after it.

[t0ms]

Retested both of my post-merge notes on merged main (with ffa02c31a in the build). One of
them was mine to withdraw.

Retracting the 8–9 s standby-join stall

It still reproduces, but it is my harness, not your routing. Both of my publishers replay
independent copies of the same clip from its start, so the standby's media timeline lags the
active one by exactly the join delay; when the relay splices sub3 onto the local standby, the
exporter is handed timestamps in the past and writes nothing until the new source passes the last
one it emitted. Scaling the join delay settles it — the stall follows with slope 1:

standby joins at	measured stall
t=4	< 2 s
t=10	9 s
t=20	18 s

Your switch is in fact immediate: relay B logs subscribe started for all three tracks against
the new local source in the same millisecond the standby's session is accepted. Sorry for the
noise — that is the second time an unsynchronised drill has cost you attention, and I have
written the rule into the harness so it stops happening: a redundancy drill whose sources start
independently measures its own clock skew unless the feeds are timestamp-aligned. #2545 now
defaults the standby to joining at t=4 and only warns when a stall exceeds what the offset
explains.

For the record, of the four things I reported across this work, two were real (the exporter's
fatal json: dropped, and the Unroutable teardown) and two were mine.

The graceful source exit does stand, and I have a cleaner repro

My first observation of this was contaminated: I had killed tsp mid-write, so the importer was
reading a truncated stream. Rebuilt without that confound — pubA publishes a finite clip, so
tsp reaches EOF and moq import finishes cleanly, and the standby joins at t=2 so the timeline
offset above is negligible. On merged main, pubA ends at t=25 with the standby announced since
t=2:

 24    4061740    174679
 25    4127164     65424  <- pubA's clip ends, clean finish
 26    4127164         0
 ...                         24s of nothing
 49    4127164         0
Error: TS track layout changed after PAT/PMT was emitted: '0.avc3' removed

Relay A's log shows why, and it is not a reselect that failed — no reselect is attempted:

subscribe canceled (idle) track=0.avc3
subscribe canceled (idle) track=0.aac
subscribed complete id=2 ... track=0.aac
subscribed complete id=1 ... track=0.avc3
subscribed cancelled id=0 ... track=catalog.json

The tracks complete, so the broadcast is treated as finished rather than as a source that left,
and the shared --origin standby is never consulted. The subscriber then dies on the catalog
losing a track.

I can see the argument that this is correct semantics — a finished broadcast is finished, and
finish() means what it says. But it does mean 1+1 redundancy protects against the harder
failure (the host disappears) and not the easier, far more common one: SIGTERM to an encoder, a
container rescheduled, a rolling restart. In those cases the operator has a healthy hot standby
and viewers still drop. If the distinction is worth drawing, the ingredient that seems to be
missing is for a graceful detach of a shared-origin source to be a reselect trigger rather than
a completion, at least while another source for that origin is announced.

Happy to leave it here if you consider it working as intended — just flagging it, since it is the
case a broadcaster will hit first. Repro script available if useful.