Skip to content

Break tri-comparison precision ties by validated count; codify as a system-wide rule - #1943

Merged
squid-protocol merged 1 commit into
mainfrom
feat/tri-comparison-precision-tiebreak
Aug 20, 2026
Merged

Break tri-comparison precision ties by validated count; codify as a system-wide rule#1943
squid-protocol merged 1 commit into
mainfrom
feat/tri-comparison-precision-tiebreak

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Summary

Found while investigating why rust's tri-comparison chart shows zero badges despite having the highest, ledger-validated accuracy: _winner_or_tie() in tests/tools/tri_comparison_chart.py had no tie-break at all -- any 2+-way tie on rate_pct returned "tie", blank badge, full stop.

Rust's real case: Func/Class Precision now ties 3-way at 100% (GitGalaxy 1927/1927, tree-sitter 1775/1775, ctags 1774/1774) once GitGalaxy's macro-body-only claims were ledger-validated (#1941). Each tool is simply never wrong about what it itself claims, at very different claim counts -- so a rate-only comparison awards nobody, despite GitGalaxy having demonstrably found more of the validated-real total.

Fix

Break a rate tie using each tied tool's absolute matched_consensus (its count of validated-correct occurrences), not the raw rate alone. This is the same "more evidence should count for something" principle as the existing sample-size fix (a 2-sample 100% cell no longer silently outranks an 80-sample 98.75% cell) -- just pointed the other direction: a 1927-sample validated 100% shouldn't lose to a 1774-sample validated 100% just because both cleared the same bar.

Only ever reached after the caller's has_open_question() check (already gated before _winner_or_tie is called), so every count used in the tie-break is already either unquestioned or ledger-validated -- this is not a reversion to the "just trust the bigger number" anti-pattern the verification system exists to prevent.

Verified

  • Direct unit check against rust's real numbers: gitgalaxy now wins the tie.
  • Guard case 1: a genuine unbreakable tie (equal rate AND equal count) still returns "tie".
  • Guard case 2: the original sample-size-bug scenario (2/2 @ 100% vs. 79/80 @ 98.75%) is unaffected -- it was never a tie to begin with, so this path isn't touched.
  • Full 45-language chart regenerated (tri_comparison_chart.py --all --write, using tree-sitter-language-pack + a locally-built universal-ctags): diff is exactly rust's two new badges (Func Precision, Class Precision) plus the summary tally moving 4→6 GitGalaxy wins / 18→16 ties. No other language's badges changed. tri_comparison_ledger.json's diff is routine last_reconciled_at timestamp churn from the same regen run, no verdict/status content changed.

Also: codifies this as a system-wide CLAUDE.md rule

The "badges require verification, not just the higher number" principle (originally proven necessary on 2026-08-19, landed via PR #1868) was only ever documented inside tri_comparison_chart.py's own docstring and the tri-comparison-ledger-sweep skill -- never promoted to a repo-wide instruction. Added a new CLAUDE.md section generalizing it to any comparative-correctness claim in this repo (a badge, a README claim, a PR description), plus this PR's new tie-break rule, so future sessions apply the same standard without re-deriving it from a specific tool's source.

🤖 Generated with Claude Code

_winner_or_tie() returned "tie" (no badge) whenever 2+ tools shared the
top rate_pct, with no tie-break at all. Real case this misses: rust's
Func/Class Precision now ties 3-way at 100% (GitGalaxy 1927/1927,
tree-sitter 1775/1775, ctags 1774/1774) once GitGalaxy's macro-body-only
claims are ledger-validated (#1941) -- each tool is simply never wrong
about what it itself claims, at very different claim counts, so a
rate-only comparison awards nobody despite GitGalaxy having demonstrably
found more of the validated-real total.

Break a rate tie using each tied tool's absolute matched_consensus (its
count of validated-correct occurrences), not the raw rate alone. This is
the same "more evidence should count for something" principle as the
existing sample-size fix (a 2-sample 100% cell no longer silently
outranks an 80-sample 98.75% cell), just pointed the other direction: a
1927-sample validated 100% shouldn't lose a badge to a 1774-sample
validated 100% just because both cleared the same bar. Only ever reached
after the caller's has_open_question() check, so every count used here
is already either unquestioned or ledger-validated -- not a reversion to
the "just trust the bigger number" anti-pattern this system exists to
prevent.

Regenerated the full 45-language chart: rust now shows a real GitGalaxy
badge on both Func and Class Precision (summary tally 4->6 languages,
ties 18->16); no other language's badges changed. Verified the tie
logic directly against rust's real numbers plus two guard cases: a
genuine unbreakable tie (equal rate AND equal count) still returns
"tie", and the original sample-size-bug scenario (2/2 vs. 79/80) is
unaffected since it was never a tie to begin with.

Also adds a new CLAUDE.md section generalizing "badges require
verification, not just the higher number" (previously only documented
inside tri_comparison_chart.py's own docstring and the
tri-comparison-ledger-sweep skill) into a repo-wide principle for any
comparative-correctness claim, plus the new tie-break rule.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit cb063ff into main Aug 20, 2026
28 checks passed
@squid-protocol
squid-protocol deleted the feat/tri-comparison-precision-tiebreak branch August 20, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant