Skip to content

Fix eval explorer comparisons and complete Kimi low coverage - #43

Merged
justsml merged 4 commits into
mainfrom
agent/eval-explorer-headlines
Jul 29, 2026
Merged

Fix eval explorer comparisons and complete Kimi low coverage#43
justsml merged 4 commits into
mainfrom
agent/eval-explorer-headlines

Conversation

@justsml

@justsml justsml commented Jul 29, 2026

Copy link
Copy Markdown
Owner

What changed

  • require meaningful row and task coverage before model profiles enter headline charts
  • order Anthropic, OpenAI, and Google first, then other companies alphabetically
  • classify root exploit-hunter/network-lab results as Network Labs
  • omit radar axes without scored coverage instead of drawing misleading blank spokes
  • complete the missing Kimi K3 Native low-reasoning model-tool tail with an 8,192-token ceiling
  • retain the zero-cost native-temperature failure diagnostic and its external failure research
  • regenerate the checked-in explorer dataset with the new results

Why

Sparse canaries and mixed-suite outliers were distorting headline token, time, and cost charts. Network browser rows were also misclassified because the category matcher required a trailing slash. The historical Kimi report referenced a clean completion run that had never been checked in.

Kimi completion

The corrected native profile uses temperature=0.6, maxTokens=8192, and low reasoning. The nine-row tail cost $1.200962, used 34/408 tool calls, scored 198/218 deterministically, and completed all 9 judges. Seven rows passed; two valid quality failures remain visible.

Validation

  • pnpm -s run run:vitest run tests/security-chat/eval-explorer-build-data.test.ts
  • node --check evals/explorer/app.js
  • pnpm eval:explorer

@justsml justsml changed the title Fix eval explorer headline comparisons Fix eval explorer comparisons and complete Kimi low coverage Jul 29, 2026
@justsml
justsml marked this pull request as ready for review July 29, 2026 01:50
@justsml
justsml merged commit 91742ea into main Jul 29, 2026
6 of 7 checks passed
@justsml
justsml deleted the agent/eval-explorer-headlines branch July 29, 2026 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant