Skip to content

refactor(libsy): extract load_judge_config and drop reasoning fallback in judge verdict parsing - #204

Merged
linj-glitch merged 1 commit into
mainfrom
linj/switch-1054-llm-judge-changes
Jul 30, 2026
Merged

refactor(libsy): extract load_judge_config and drop reasoning fallback in judge verdict parsing#204
linj-glitch merged 1 commit into
mainfrom
linj/switch-1054-llm-judge-changes

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Two related judge changes: extracts the prompt+schema loading logic into a shared load_judge_config helper (used by both the capability classifier and the escalation judge), and drops the reasoning-content fallback in parse_json_verdict. The fallback was added to handle gateways that return verdicts in reasoning blocks, but the judge is now configured to run without thinking, so the completion always carries the verdict — reading from reasoning was masking truncated replies instead of surfacing them as errors. Part 3 of breaking up PR #172.

Summary by CodeRabbit

  • Bug Fixes
    • Improved task-routing judge requests by consistently including required system guidance.
    • Fixed verdict parsing to read responses from the correct content and avoid treating reasoning text as a verdict.
    • Improved handling of structured response schemas used by judge evaluations.
  • Tests
    • Added coverage for verdicts containing reasoning-only content and updated validation of judge request formatting.

@linj-glitch
linj-glitch requested a review from a team as a code owner July 30, 2026 17:52
…k in judge verdict parsing

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch
linj-glitch force-pushed the linj/switch-1054-llm-judge-changes branch from 59ea9c4 to 6577642 Compare July 30, 2026 17:56
@linj-glitch
linj-glitch changed the base branch from linj/switch-1054-classifier-changes to main July 30, 2026 17:57
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Changes

Judge evaluation flow

Layer / File(s) Summary
Shared judge configuration loader
crates/libsy/src/algorithms/util/llm_judge.rs, crates/libsy/src/algorithms/util.rs
Adds shared schema parsing and prompt rendering, then re-exports the loader within the crate.
System message request construction
crates/libsy/src/algorithms/llm_class.rs
Places the judge prompt at the start of LlmRequest.messages and updates the corresponding test expectations.
Completion verdict parsing
crates/libsy/src/algorithms/util/llm_judge.rs
Parses verdicts from completion content, ignores reasoning blocks, and adjusts classification extraction.
Estimated code review effort: 3 (Moderate) ~20 minutes

Poem

I’m a rabbit with prompts in my pack,
A system message leads the stack.
Schemas bloom bright, verdicts come through,
Reasoning hops aside from view.
Tests thump softly: the path is true.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main refactor and the verdict-parsing change in this PR.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/libsy/src/algorithms/llm_class.rs (1)

850-850: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert the prepended system-message contract.

The count alone passes if the first message has the wrong role or prompt. Assert that message zero is Role::System and equals judge.config.system_prompt.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/libsy/src/algorithms/llm_class.rs` at line 850, Extend the assertions
around judge_request.llm_request.messages to validate the prepended message
itself: assert message zero has Role::System and its prompt/content equals
judge.config.system_prompt, while retaining the existing length assertion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/libsy/src/algorithms/llm_class.rs`:
- Line 850: Extend the assertions around judge_request.llm_request.messages to
validate the prepended message itself: assert message zero has Role::System and
its prompt/content equals judge.config.system_prompt, while retaining the
existing length assertion.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c7b0fa6b-c8eb-46ca-a043-ba9a8640c536

📥 Commits

Reviewing files that changed from the base of the PR and between f3ff7ff and 6577642.

📒 Files selected for processing (3)
  • crates/libsy/src/algorithms/llm_class.rs
  • crates/libsy/src/algorithms/util.rs
  • crates/libsy/src/algorithms/util/llm_judge.rs

@linj-glitch
linj-glitch merged commit 6b9a168 into main Jul 30, 2026
18 checks passed
@linj-glitch
linj-glitch deleted the linj/switch-1054-llm-judge-changes branch July 30, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants