feat(libsy): stage router with handoff notes, per-tier prompts, and LLM fallback by sabhatinas · Pull Request #170 · NVIDIA-NeMo/Switchyard

sabhatinas · 2026-07-28T22:06:15Z

A stage router cascade algorithm: tool signals decide most turns for free, and only the ones they can't fall through to the LLM classifier. If LLM classifier can not decide, default target is used as fall open.

The cascade

tool signals → capability classifier (optional) → picker's default tier

Each step decides or abstains, and the next one gets the turn. The last never abstains, so a turn is never left unrouted.

Turn	Decided by	Recorded as
signals confident	signals	`override` / `tests_passed` / `dimensions`
signals unsure, classifier answers	classifier	`llm-classifier`
signals unsure, classifier can't tell or absent	picker default	`fall_open`

A turn the signals settle never reaches the classifier, so it costs nothing. StageRouter wraps the cascade and implements Algorithm, so FallThrough stays an internal detail.

The two tiers are capable and efficient; their targets are whatever the deployment names them.

Changes to `llm_class`

Abstains when it can't tell — unusable verdict, timeout, or the judge's own abstain — instead of quietly forcing the capable tier. The composition picks the fallback, so a judge failure now errs cheap under efficient_first rather than expensive.
Its own cascade closes with the capable tier, so the standalone route behaves exactly as before.
recent_turn_window is now a TaskClassifierConfig knob. None keeps today's newest-user-message-only judging; Some(n) widens to the client instructions, the opening task, and the last n turns after it.
trim_messages selects by reference and clones only what survives, rather than copying the whole transcript to keep a window of it.

Changes to tool signals

Signals are read off the decoded conversation, not the raw body — three per-format parsers become one.
Found live: the endpoint records no inbound wire format, so the old parser saw no signals and every turn routed as if the conversation had none.
Recording the format is not the fix — the LLM client reads that field to pick the upstream format.
The 40 signal tests are unchanged, still written against real wire bodies, now decoded through the codec.

Changes to `llm_class`

Abstains when it can't tell — unusable verdict, timeout, or the judge's own abstain — instead of forcing the capable tier. The composition picks the fallback, so a judge failure errs cheap under efficient_first.
Its own cascade closes with the capable tier, so the standalone route is unchanged.
recent_turn_window is now a TaskClassifierConfig knob: None judges the newest user message alone, Some(n) adds the instructions, opening task, and last n turns.
trim_messages clones only what survives instead of copying the transcript.

Changes to `fall_through`

A post-decision processor block. Two passes: every processor sees the decision, then the outbound request is offered for rewriting. Order matters — the second pass sees the state the first settled.
Event::ModelRequest now fires and carries the decision. Declared but never emitted before; it is the only hook that sees both, so a processor can act on the chosen target without keeping state between turns.
The tier label now comes from any classifier in the cascade, not just the one that scored first, so a terminal fallback no longer leaves the call untiered.
DefaultTarget — a terminal classifier that always picks one target. Used by this router and the capability route to close their cascades now that classifiers can abstain.

Configuration

Thresholds are validated at construction, and the fall-back target is resolved once from the picker mode rather than per turn.

[routes.stage]
id = "switchyard/stage"
type = "stage_router"
capable_target = "opus"
efficient_target = "nemotron"
picker = "efficient_first"          # tier an undecided turn falls back to
confidence_threshold = 0.5          # agreement the signals need to decide
recent_turn_window = 3              # recent tool results the signals read
capable_system_prompt = "Diagnose the root cause quickly and act decisively."
efficient_system_prompt = "Work efficiently. Do not re-read files you have examined."

[routes.stage.handoff_notes]
escalation_note = "The previous model was stalling; pick up the diagnosis."
# deescalation_note = "..."                 # optional capable→efficient note
# only_on_wrong_signal_escalation = true    # default

[routes.stage.classifier]                   # omit for a signals-only router
target = "judge"
base_threshold = 0.5
recent_turn_window = 3                      # conversation turns the judge reads

Handoff notes


Fires on	signal-driven escalation; hand-back to efficient if configured
Placement	end of the last user turn, after the tool result
State	none — describes this turn's signals, like the Python router
Lifetime	forwarded request only, never in history

Tier system prompts


Applies on	every turn that target serves, whichever step of the cascade picked it
Placement	ahead of the client's own instructions, so the framing is read first
Keyed by	target name, not tier — any router can use it, not just this one
Lifetime	forwarded request only, never in history

Follow-ups left as TODOs

Session affinity should bind on the post-decision hook (llm_class.rs), the way the per-target system prompt does. Affinity is a fact about the decision, not a classification, so recording it there would let a pinned session skip the cascade outright on the next turn. It would also make the pin survive composition: a cascade that uses this classifier as one member — the stage router does — never runs the affinity wired inside it, because only the inner classifier is consulted. That is why session_affinity in a stage_router's [classifier] block parses today but has no effect.

Testing

tests/stage_router.rs — multi-turn sessions through the assembled router: one note per signal-driven turn and none otherwise, no classifier call when the signals settled, a fresh classifier answer each turn, an unusable verdict landing on the picker default, the configured window reaching the judge, and the correct record of who decided.
tests/prompts.rs — the request-text helpers on a plain cascade with no stage routing, which is what proves they are reusable: per-target prompts, one target's prompt not leaking onto another, the prompt following the target whichever classifier picked it, and a note reaching the model in the conversation.
The server tests build a stage_router route with notes, tier prompts, and a classifier, and pin that a critical tool error escalates on the signals alone — the case that was silently broken.
Validated against a live server (NVIDIA Inference Hub, Opus 4.7 capable / Nemotron-3-Super efficient) over 20 turns, run twice: signals-only, and with the classifier. All 10 error turns escalate identically in both configs, so the classifier is genuinely never consulted when the signals resolve a turn; the 10 clean turns fall open to the picker default without a classifier, and split on the judge's own reading with one.
All green, plus lint and docs.

Refs SWITCH-1120, SWITCH-982, SWITCH-1110.

coderabbitai · 2026-07-28T22:13:02Z

Walkthrough

Adds a public StageRouter that routes between strong and weak targets using tool signals, optional judge fallback, handoff notes, and tier-specific prompts. Request processors now receive a final ModelRequest replay before the routed model call.

Changes

Stage Routing

Layer / File(s)	Summary
Request processor replay `crates/libsy/src/algorithms/fall_through.rs`	`FallThrough` now replays `ModelRequest` after the request and decision events, with updated event-ordering tests.
Tier change and handoff notes `crates/libsy/src/algorithms/util/handoff_notes.rs`, `crates/libsy/src/algorithms/util/stage_router.rs`	Handoff notes are configured and injected directly into outbound requests when resolved routing changes tiers; state tracking and note behavior are covered by unit tests.
Per-tier prompt processing `crates/libsy/src/algorithms/util/tier_prompts.rs`, `crates/libsy/src/algorithms/util.rs`	A processor appends configured strong or weak system prompts during `ModelRequest` processing.
StageRouter cascade and public API `crates/libsy/src/algorithms/stage.rs`, `crates/libsy/src/algorithms.rs`, `crates/libsy/src/lib.rs`, `crates/libsy/tests/stage_router_handoff.rs`	Adds the configurable signal cascade, optional LLM judge fallback, fall-open behavior, public exports, construction tests, and integration coverage for routing, notes, judge calls, and prompts.

Estimated code review effort: 4 (Complex) | ~60 minutes

Poem

I’m a rabbit routing through the hay,
Strong and weak tiers hop into play.
Notes appear when pathways bend,
Prompts follow each turn to the end.
Judge or signal, the cascade knows—
Then safely falls where default flows.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 93.88% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main addition: a new stage router with handoff notes, tier prompts, and LLM fallback.

_{Comment @coderabbitai help to get the list of available commands.}

nachiketb-nvidia · 2026-07-29T00:22:43Z

+}
+
+#[async_trait]
+impl Algorithm<SharedState> for StageRouter {


we should use this directly in config.rs

messiaen · 2026-07-29T13:59:38Z

+///
+/// Errors if either threshold in `config` is outside `[0.0, 1.0]` or a tier is
+/// missing from `targets`.
+pub fn stage_router(


Do we want a factory function like this or a algorithm::Stage wrapper around FallThrough?
LlmTaskClassifier uses a wrapper. I'm guessing that is easier to configure on the server size?

@nachiketb-nvidia what are your thoughts?

I prefer not have function,

something like this is better

struct StageRouter { inner: FallThrough<...> } impl Algorithm for StageRouter { ... # use inner directly here }

Using a function means we move Fallthrough from an internal implementation detail to public API (since it is crossing crates right now)

Got it, I am pushing this change

sabhatinas · 2026-07-29T17:20:43Z

                .await?;
        }

+        // 5. Offer the outbound request to the processors, paired with the decision that


@messiaen for clarity, I added this to append system prompts for instance. operations that do not need to go through routing algorithm and depend on routing decision can be here?

sabhatinas · 2026-07-29T17:21:46Z

-            .collect::<Vec<_>>();
+        // Task-based routing judges the newest user message alone. A configured
+        // window widens that to the surrounding conversation.
+        let mut messages = match self.recent_turn_window {


@ayushag-nv please review this new method for recent window trimming.

sabhatinas · 2026-07-29T17:23:12Z

-                &self.efficient_target
-            }
-            _ => &self.capable_target,
+        // Judge output is untrusted, so only a complete, valid, non-abstained verdict that


@messiaen @ayushag-nv please should review this change as well. When an algorithm does not have a clear decision, there should be an ambigious decision so that cascades and fallbacks can move the needle.

Looks correct to me

sabhatinas · 2026-07-29T17:36:54Z

+}
+
+#[async_trait]
+impl<S: Send> Classifier<S> for DefaultTarget {


No preference on how we are setting default target when all algorithms fail to provide a score. Please help review.

This seems reasonable.

sabhatinas · 2026-07-29T18:06:55Z

+        // 5. Offer the outbound request to the processors, paired with the decision that
+        //    routed it. This is the only hook that sees both, so it is where a
+        //    decision-dependent rewrite belongs.
+        for processor in &self.processors {


@linj-glitch please grab this for your latching mechanism with escalation router. For confirmation count and session affinity, this will be the path for next request to skip all classifiers.

Sounds good!

cc: @messiaen for confirmation

A decision event contains a request.
And we want to process the decision also, no?

Yeah we want to pass Decision event here, request event implies it's preprocessing I think

I would imagine we should not touch decision here.

If Im reading it right, decision is not mutable intentionally so.

nvm, applying changes per your comment. passing in decision is enough.

…apability-classifier fallback Signed-off-by: Sabhatina Selvam <sabhatinas@nvidia.com>

messiaen

adding comments still reviewing. But looks good overal

messiaen · 2026-07-29T18:47:59Z

+        // 5. Offer the outbound request to the processors, paired with the decision that
+        //    routed it. This is the only hook that sees both, so it is where a
+        //    decision-dependent rewrite belongs.
+        for processor in &self.processors {


A decision event contains a request.
And we want to process the decision also, no?

messiaen · 2026-07-29T18:48:24Z

+}
+
+#[async_trait]
+impl<S: Send> Classifier<S> for DefaultTarget {


This seems reasonable.

messiaen · 2026-07-29T18:48:46Z

+        // 3. Resolve the target and publish the decision. A tier is a fact about the
+        //    target, not about which classifier happened to name it first, so any member
+        //    of the cascade may answer — a terminal fallback that decides a turn would
+        //    otherwise leave the call untiered.


don't need the paragraphs here.

messiaen · 2026-07-29T18:52:35Z

+        // 5. Offer the outbound request to the processors, paired with the decision that
+        //    routed it. This is the only hook that sees both, so it is where a
+        //    decision-dependent rewrite belongs.
+        for processor in &self.processors {


Yeah we want to pass Decision event here, request event implies it's preprocessing I think

messiaen · 2026-07-29T18:53:08Z

                let kind = match event {
                    Event::Request(_) => "request",
                    Event::Decision { .. } => "decision",
+                    Event::ModelRequest { .. } => "model_request",


Decision has the request already.

messiaen · 2026-07-29T18:57:27Z

-                &self.efficient_target
-            }
-            _ => &self.capable_target,
+        // Judge output is untrusted, so only a complete, valid, non-abstained verdict that


Looks correct to me

messiaen · 2026-07-29T18:58:46Z

+            return Classification::Ambiguous(vec![Score {
+                target: self.capable_target.clone(),
+                confidence: 0.0,
+            }]);


should we just return an empty set of scores here?

Is it different from returning confidence = 0?

sabhatinas requested a review from a team as a code owner July 28, 2026 22:06

nachiketb-nvidia reviewed Jul 29, 2026

View reviewed changes

sabhatinas force-pushed the sabhatinas/switch-982-handoff-notes-in-stage-classifier branch from e1154ce to 72d8fe6 Compare July 29, 2026 00:43

messiaen reviewed Jul 29, 2026

View reviewed changes

sabhatinas force-pushed the sabhatinas/switch-982-handoff-notes-in-stage-classifier branch 3 times, most recently from e577cfe to 219f2ca Compare July 29, 2026 17:18

sabhatinas commented Jul 29, 2026

View reviewed changes

sabhatinas force-pushed the sabhatinas/switch-982-handoff-notes-in-stage-classifier branch from 219f2ca to 36706fd Compare July 29, 2026 17:32

sabhatinas commented Jul 29, 2026

View reviewed changes

sabhatinas force-pushed the sabhatinas/switch-982-handoff-notes-in-stage-classifier branch 4 times, most recently from 1b09552 to 94b68ac Compare July 29, 2026 19:15

feat(libsy): stage router with handoff notes, per-tier prompts, and c…

f4f62df

…apability-classifier fallback Signed-off-by: Sabhatina Selvam <sabhatinas@nvidia.com>

sabhatinas force-pushed the sabhatinas/switch-982-handoff-notes-in-stage-classifier branch from 94b68ac to f4f62df Compare July 29, 2026 19:31

messiaen reviewed Jul 29, 2026

View reviewed changes

Uh oh!

Conversation

sabhatinas commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The cascade

Changes to llm_class

Changes to tool signals

Changes to llm_class

Changes to fall_through

Configuration

Handoff notes

Tier system prompts

Follow-ups left as TODOs

Testing

Uh oh!

coderabbitai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

messiaen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

sabhatinas commented Jul 28, 2026 •

edited

Loading

Changes to `llm_class`

Changes to `llm_class`

Changes to `fall_through`

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading