Agent test UI: routing cases, history, registration, scope planning, … - #480
Open
yuyixg wants to merge 5 commits into
Open
Agent test UI: routing cases, history, registration, scope planning, …#480yuyixg wants to merge 5 commits into
yuyixg wants to merge 5 commits into
Conversation
…metrics Follows the backend commit in BotSharp. One commit for the same reason: the four agent-test pages and the shared helper each carry changes from several of the features below, and the llmJudge UI work was already sitting uncommitted here. Case editor ----------- Case type (Routing or Agent) with the rules a routing case has to satisfy checked as you type, sharing one helper with the backend so the two cannot drift -- and if they ever do, the backend wins and the save fails, which is the safe direction. Entry agent is a dropdown, not an id field. The id is a guid nobody types from memory and a typo was only caught on save. The dropdown own "Clear selection" is how you fall back to the suite agent, and the payload sends null rather than an empty string for that. Authored history rows (role plus text, reorderable, since the order IS the conversation) with a note that they are not driven through the model and never appear in the agent chain. A registration panel: priority, severity, batch, cross-cutting, involved agents, business domain, expected outcome and a last-reviewed date. The batch help text shows the batch the case will actually run in, since it is derived rather than chosen. Last reviewed has a Today button -- a date field nobody can be bothered to type is a date field that stays empty, and an empty reviewed date is indistinguishable from a case reviewed long ago. An agentChain assertion turns the Target column into a mode dropdown, because that field carries the comparison mode for this one type. Switching an assertion type now resets a mode left behind by the previous type, which would otherwise be saved as the name of a function to assert was called. Case list --------- Type and history columns, so the type a case runs under is visible without opening every one of them, and a copy button. Copying navigates straight into the copy: it lands disabled and named "(copy)", and neither is something anyone leaves as it is. Fixed a bug this branch had introduced: the enable toggle rebuilt a full-replace PUT payload field by field, so a single click erased caseType, entryAgentId and history. It now spreads the case, which cannot drop a field added later. Run detail ---------- Routing accuracy per model, shown as a percentage next to passed/total -- with a handful of routing cases, how much the figure is worth trusting matters more than the figure. Latency, tokens and cost per model, alongside the unit costs the cost was computed from. Unknown pricing renders as unknown, never as zero, which would read as a claim that the model is free. The agent chain on each result, visible without expanding, since "which agent actually answered" is the first thing looked at when a routing case goes red. Per result the agent-call time is shown next to the wall clock, because the percentiles are built from the former. Scope planning -------------- A panel on the index page: name the changed agents, or declare a platform-wide change, and see which cases that needs to run. Deliberately separate from triggering a run -- folding it into a run button would mean the only way to see the plan is to have already paid for it. Included and excluded cases appear in one table with the rule that decided each. Splitting them into two panels would make it possible to read only the reassuring one, and the exclusions are the half that matters: an excluded case produces no result to notice. Translations ------------ Every new string in both dictionaries, no missing keys either way. Case type labels use their own keys rather than the bare words: "Routing" already exists in zh.json meaning the routing-graph screen, so reusing it would have mistranslated the option. .env is deliberately not included -- it points at a local host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
main gained "replace agent test styles" (88568bc) after PR SciSharp#478 was merged: the four agent-test pages were rewritten against a new 880-line stylesheet, +2321/-1055. This branch had added its features to the pages as they were before that, so three of them conflicted. Resolved by taking main's version of all three wholesale and re-applying each addition in the new idiom, rather than untangling the conflict markers. A mechanical merge would have left the new features wearing the old Bootstrap classes inside restyled pages -- balanced markup that looks broken. Every addition now uses the ats-* classes: ats-card / ats-card-section for the two new cards, ats-input, ats-textarea, ats-switch, ats-badge with ats-tone-*, ats-btn-icon for the row actions, and the grid/col-span layout the rest of the page uses. Two things the conflict surfaced rather than caused: - One nesting mistake of mine, caught by reading the structure rather than by the build: the cards share a single flex column, and my first attempt closed it after the new Registration card and opened two invented wrappers. Tags balanced, so it compiled -- the remaining cards would just have lost their column spacing. - The restyled markup still carried the pre-judge llmJudge wording ("Always fails in P1"), whose keys this branch had already replaced, and which is no longer true: llmJudge is scored by LlmAgentTestJudge. Pointed both usages at the corrected strings instead of reviving the stale keys, and dropped the warning tone from a hint that now describes normal behaviour. Four help strings picked up their apostrophes again, so they reuse the keys the dictionaries already hold rather than adding near-duplicates that differ by punctuation. Verified: the UI builds, no missing translation keys in either dictionary, and every feature this branch adds is present in the merged files -- case type, entry agent picker, history rows, registration panel, involved-agent picker, agentChain mode picker, copy button, the enable-toggle payload fix, the type and history columns, routing accuracy, the latency/token/cost table, agent chain badges, per-result model time, and the scope panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A checkbox column and a select-all in the run history table, mirroring the case table above it, plus a delete on each finished row and a bulk delete for the selection. The button count excludes runs that are still executing, and says so next to itself, because the server refuses those -- a count that included them would promise a delete that never happens. On a live row the only action offered is Cancel: deleting a run that is still going would not stop it, so the other button would not do what it says. Skipped runs are surfaced one by one rather than folded into the success message. A toast saying "deleted 2" while a third row stays put is how someone concludes the button is broken. Reloading the list drops selections for runs that are gone, so a later delete cannot post ids the server can only answer "already deleted" for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scope panel was written before main's agent-test restyle landed, and git merged it cleanly because nothing about it conflicted textually -- so it kept its Bootstrap classes (card-body, row g-3, col-md-5, form-label, table table-sm, badge bg-*) inside a page that had moved to ats-*. That is precisely the half-old, half-new result the conflict resolution avoided in the three files that did conflict; this one slipped through by not conflicting. Rewritten in the ats-* idiom and moved to /page/agent-test/scope. The styling was the reported problem, but the page is the better home for a second reason: planning a scope spans every suite, so it never belonged on a page titled "Test Suites" -- it is a different job, done at a different time, and it now reads that way. Reached from a link on the suites card, since nothing else in the UI points at that route. An anchor rather than a button with goto(), so middle-click, open-in-new-tab and screen-reader semantics come for free. The route also had to be named in svelte.config.js. Prerendering here runs with crawl: false and an explicit entries list, so a new page is invisible to the build until it is listed -- the build fails outright rather than quietly shipping without it, which is how this surfaced. My first attempt at the link comment claimed the crawler only follows anchors; with crawling off there is no crawler at all, and the comment now says what actually registers the route. The results table keeps both halves of the decision in one list, included first. Splitting them into two panels would make it possible to read only the reassuring one, and the exclusions are the half that matters: an excluded case produces no result to notice. Verified: the UI builds, scope.html is prerendered, no missing translation keys in either dictionary, and no Bootstrap classes remain on either page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two columns on the case table, so the decision of what needs running can be made from the list instead of by opening every case. The priority cell also shows the batch the case will actually run in, and that is the part that matters. Cross-cutting overrides priority and forces batch 1, so a P2 safety case runs FIRST -- a bare "P2" would read as "runs last", which is the opposite of the truth and precisely the call this column exists to inform. Tones are chosen so the list can be skimmed rather than read. P0 and S0 are loud: P0 is the stop-loss batch where one failure halts the evaluation, and S0 is zero tolerance -- data leakage, an unauthorised action, a missed escalation. P1 stays neutral rather than amber, because it is the default every untriaged case carries and a wall of warning colour would say nothing at all. S2 is quiet and must never look as loud as the other two. The scope page shows severity as a badge now too, from the same helper, so a severity means the same thing wherever it appears. No backend change: the case list already returns these fields. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…metrics
Follows the backend commit in BotSharp. One commit for the same reason: the four agent-test pages and the shared helper each carry changes from several of the features below, and the llmJudge UI work was already sitting uncommitted here.
Case editor
Case type (Routing or Agent) with the rules a routing case has to satisfy checked as you type, sharing one helper with the backend so the two cannot drift -- and if they ever do, the backend wins and the save fails, which is the safe direction.
Entry agent is a dropdown, not an id field. The id is a guid nobody types from memory and a typo was only caught on save. The dropdown own "Clear selection" is how you fall back to the suite agent, and the payload sends null rather than an empty string for that.
Authored history rows (role plus text, reorderable, since the order IS the conversation) with a note that they are not driven through the model and never appear in the agent chain.
A registration panel: priority, severity, batch, cross-cutting, involved agents, business domain, expected outcome and a last-reviewed date. The batch help text shows the batch the case will actually run in, since it is derived rather than chosen. Last reviewed has a Today button -- a date field nobody can be bothered to type is a date field that stays empty, and an empty reviewed date is indistinguishable from a case reviewed long ago.
An agentChain assertion turns the Target column into a mode dropdown, because that field carries the comparison mode for this one type. Switching an assertion type now resets a mode left behind by the previous type, which would otherwise be saved as the name of a function to assert was called.
Case list
Type and history columns, so the type a case runs under is visible without opening every one of them, and a copy button.
Copying navigates straight into the copy: it lands disabled and named "(copy)", and neither is something anyone leaves as it is.
Fixed a bug this branch had introduced: the enable toggle rebuilt a full-replace PUT payload field by field, so a single click erased caseType, entryAgentId and history. It now spreads the case, which cannot drop a field added later.
Run detail
Routing accuracy per model, shown as a percentage next to passed/total -- with a handful of routing cases, how much the figure is worth trusting matters more than the figure.
Latency, tokens and cost per model, alongside the unit costs the cost was computed from. Unknown pricing renders as unknown, never as zero, which would read as a claim that the model is free.
The agent chain on each result, visible without expanding, since "which agent actually answered" is the first thing looked at when a routing case goes red. Per result the agent-call time is shown next to the wall clock, because the percentiles are built from the former.
Scope planning
A panel on the index page: name the changed agents, or declare a platform-wide change, and see which cases that needs to run. Deliberately separate from triggering a run -- folding it into a run button would mean the only way to see the plan is to have already paid for it.
Included and excluded cases appear in one table with the rule that decided each. Splitting them into two panels would make it possible to read only the reassuring one, and the exclusions are the half that matters: an excluded case produces no result to notice.
Translations
Every new string in both dictionaries, no missing keys either way. Case type labels use their own keys rather than the bare words: "Routing" already exists in zh.json meaning the routing-graph screen, so reusing it would have mistranslated the option.
.env is deliberately not included -- it points at a local host.