Break tri-comparison precision ties by validated count; codify as a system-wide rule - #1943
Merged
Merged
Conversation
_winner_or_tie() returned "tie" (no badge) whenever 2+ tools shared the top rate_pct, with no tie-break at all. Real case this misses: rust's Func/Class Precision now ties 3-way at 100% (GitGalaxy 1927/1927, tree-sitter 1775/1775, ctags 1774/1774) once GitGalaxy's macro-body-only claims are ledger-validated (#1941) -- each tool is simply never wrong about what it itself claims, at very different claim counts, so a rate-only comparison awards nobody despite GitGalaxy having demonstrably found more of the validated-real total. Break a rate tie using each tied tool's absolute matched_consensus (its count of validated-correct occurrences), not the raw rate alone. This is the same "more evidence should count for something" principle as the existing sample-size fix (a 2-sample 100% cell no longer silently outranks an 80-sample 98.75% cell), just pointed the other direction: a 1927-sample validated 100% shouldn't lose a badge to a 1774-sample validated 100% just because both cleared the same bar. Only ever reached after the caller's has_open_question() check, so every count used here is already either unquestioned or ledger-validated -- not a reversion to the "just trust the bigger number" anti-pattern this system exists to prevent. Regenerated the full 45-language chart: rust now shows a real GitGalaxy badge on both Func and Class Precision (summary tally 4->6 languages, ties 18->16); no other language's badges changed. Verified the tie logic directly against rust's real numbers plus two guard cases: a genuine unbreakable tie (equal rate AND equal count) still returns "tie", and the original sample-size-bug scenario (2/2 vs. 79/80) is unaffected since it was never a tie to begin with. Also adds a new CLAUDE.md section generalizing "badges require verification, not just the higher number" (previously only documented inside tri_comparison_chart.py's own docstring and the tri-comparison-ledger-sweep skill) into a repo-wide principle for any comparative-correctness claim, plus the new tie-break rule. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Found while investigating why rust's tri-comparison chart shows zero badges despite having the highest, ledger-validated accuracy:
_winner_or_tie()intests/tools/tri_comparison_chart.pyhad no tie-break at all -- any 2+-way tie onrate_pctreturned"tie", blank badge, full stop.Rust's real case: Func/Class Precision now ties 3-way at 100% (GitGalaxy 1927/1927, tree-sitter 1775/1775, ctags 1774/1774) once GitGalaxy's macro-body-only claims were ledger-validated (#1941). Each tool is simply never wrong about what it itself claims, at very different claim counts -- so a rate-only comparison awards nobody, despite GitGalaxy having demonstrably found more of the validated-real total.
Fix
Break a rate tie using each tied tool's absolute
matched_consensus(its count of validated-correct occurrences), not the raw rate alone. This is the same "more evidence should count for something" principle as the existing sample-size fix (a 2-sample 100% cell no longer silently outranks an 80-sample 98.75% cell) -- just pointed the other direction: a 1927-sample validated 100% shouldn't lose to a 1774-sample validated 100% just because both cleared the same bar.Only ever reached after the caller's
has_open_question()check (already gated before_winner_or_tieis called), so every count used in the tie-break is already either unquestioned or ledger-validated -- this is not a reversion to the "just trust the bigger number" anti-pattern the verification system exists to prevent.Verified
gitgalaxynow wins the tie."tie".tri_comparison_chart.py --all --write, using tree-sitter-language-pack + a locally-built universal-ctags): diff is exactly rust's two new badges (Func Precision, Class Precision) plus the summary tally moving 4→6 GitGalaxy wins / 18→16 ties. No other language's badges changed.tri_comparison_ledger.json's diff is routinelast_reconciled_attimestamp churn from the same regen run, no verdict/status content changed.Also: codifies this as a system-wide CLAUDE.md rule
The "badges require verification, not just the higher number" principle (originally proven necessary on 2026-08-19, landed via PR #1868) was only ever documented inside
tri_comparison_chart.py's own docstring and thetri-comparison-ledger-sweepskill -- never promoted to a repo-wide instruction. Added a new CLAUDE.md section generalizing it to any comparative-correctness claim in this repo (a badge, a README claim, a PR description), plus this PR's new tie-break rule, so future sessions apply the same standard without re-deriving it from a specific tool's source.🤖 Generated with Claude Code