Skip to content

Add evidence-based benchmark regression limits#239

Merged
Widthdom merged 1 commit into
mainfrom
agent/issue-225-evidence-thresholds
Jul 25, 2026
Merged

Add evidence-based benchmark regression limits#239
Widthdom merged 1 commit into
mainfrom
agent/issue-225-evidence-thresholds

Conversation

@Widthdom

Copy link
Copy Markdown
Owner

CLOSES #225

Summary

Replace the single 200% BenchmarkDotNet regression threshold with an evidence-based, per-benchmark-group policy and a repository-owned comparison script.

  • compare the current BenchmarkDotNet mean with the median of compatible trusted gh-benchmarks history
  • distinguish PASS, WARNING, FAIL, and WARMUP in the GitHub Actions job summary
  • publish a failed run as a reviewed baseline only through an explicit main-only workflow-dispatch option
  • keep English and Japanese performance, developer, and testing documentation aligned

Acceptance criteria

  1. Measure normal variance per group: documented seven compatible hosted Ubuntu runs and each group's observed maximum slowdown (4.7% to 42.7%).
  2. Document rationale: warning limits round the observed maximum up to 5 percentage points and add a 5-point guard band; failure limits are twice the warning limits.
  3. Detect meaningful regressions: failure limits are 20%, 40%, 80%, 40%, and 100%, all substantially below the retired 200% threshold.
  4. Avoid noise-driven failures: use the median of up to ten compatible runs, require five samples, and fingerprint benchmark definitions plus global.json.
  5. Report warning/failure results: write a per-benchmark Actions summary table and emit ::warning / ::error annotations.
  6. Define reset/update process: document fingerprint-based warmup, baseline-revision bumps, trusted-main publication, and runner-change remeasurement.
  7. Test boundaries: cover exact warning/failure boundaries, intentional publication, definition/revision/base filtering, unmatched benchmarks, and rejection of 200% policies.

Validation

  • python3 -m unittest discover -s scripts/tests -p 'test_*.py' -v — 13 passed
  • dotnet test FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj -c Release — 2,044 passed, 1 expected environment-dependent skip
  • dotnet format FolderDiffIL4DotNet.sln --verify-no-changes --no-restore --verbosity minimal
  • parsed .github/workflows/benchmark-regression.yml with Ruby YAML
  • validated benchmark-regression-policy.json with jq
  • ran the new gate against the latest hosted BenchmarkCombinedReport artifact and published gh-benchmarks history — all 12 benchmarks passed
  • git diff --check

@Widthdom
Widthdom marked this pull request as ready for review July 25, 2026 17:48
@Widthdom
Widthdom merged commit 4dfce22 into main Jul 25, 2026
14 checks passed
@Widthdom
Widthdom deleted the agent/issue-225-evidence-thresholds branch July 25, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace the 200% performance-regression threshold with evidence-based limits

1 participant