Make the benchmark CI guard robust against run-to-run noise#48
Merged
Conversation
Korijn
force-pushed
the
claude/benchmark-guard
branch
from
00ac88f to
d5dfadb
Compare
The benchmark job compared a single master run against a single PR run with --benchmark-compare-fail=mean:5%, a gate that sits below the noise floor of GitHub-hosted runners (observ measured 10-30% swings between runs of identical code). Replace it with the design observ landed in fork-tongue/observ#181 and #187: * Disable GC during timed rounds (--benchmark-disable-gc) and add the median column, which is what the guard gates on. * Pin PYTHONHASHSEED=0 in CI so dict/set timings are comparable between processes. * Run master and PR benchmarks interleaved, 3 runs each (A/B/A/B/A/B), instead of once each. * Replace --benchmark-compare-fail with benchmarks/compare_runs.py, which only fails a benchmark when the fastest PR run is more than 25% slower than the slowest master run (per-benchmark medians), so a lucky or unlucky process on either side can't produce a false positive. Results are also written to the GitHub job summary. The 25% threshold is deliberately a tripwire for gross accidental regressions, not a precision instrument; smaller deltas should be measured with repeated local runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2
Korijn
force-pushed
the
claude/benchmark-guard
branch
from
d5dfadb to
ee50a23
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap PR 2 (Track A) — port of observ #181 + #187. Rebased onto
masterafter #47 merged.Why
The benchmark job compared a single master run against a single PR run with
--benchmark-compare-fail=mean:5%. That gate sits below the noise floor of GitHub-hosted runners: on observ, whole runs of identical code differed by 10–30%, producing fail → fail → pass across attempts of the same commit. A gate below the noise floor doesn't catch regressions — it breeds re-run-until-green rituals.What this changes
--benchmark-disable-gcinaddopts) and the median column added — the guard gates on medians.PYTHONHASHSEED=0pinned in CI so dict/set timings are comparable between processes.benchmarks/compare_runs.py(taken from observ'sbench/compare_runs.py) replaces--benchmark-compare-fail: a benchmark only fails when the fastest PR run is more than 25% slower than the slowest master run (per-benchmark medians), so a lucky or unlucky process on either side can't produce a false positive. Results are also appended to the GitHub job summary as a markdown table.The 25% threshold is deliberately positioned as a tripwire for gross accidental regressions (an accidental copy in a hot path, an algorithmic slip), not a precision instrument — both the workflow and the script say so in comments. Changes too small to trip it should be measured with repeated local runs.
Verification
Ran the exact pipeline locally: two
--benchmark-saveruns of the pointer benchmarks withPYTHONHASHSEED=0, thencompare_runs.py --baseline '.benchmarks/*/*_master?.json' --branch '.benchmarks/*/*_branch?.json' --threshold 0.25— table renders, exit 0 on parity, and ruff check/format pass on the new script.🤖 Generated with Claude Code
https://claude.ai/code/session_016Mu9vEBwU4fQLi8ZgkgdS2