feat: add python line and branch coverage governance to ci - #317
feat: add python line and branch coverage governance to ci#317forrestjgq wants to merge 2 commits into
Conversation
Add scripts/coverage_gate.py and make targets that measure line and branch coverage for the raven package and enforce three gates: a monotonic ratchet against an audited baseline, 90% coverage on the executable lines a pull request changes, and a baseline that cannot be lowered relative to the target branch. The CI unit job runs the suite once with coverage and reuses the report for every gate, uploading xml, json, and html artifacts on every run. Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
|
Claude finished @forrestjgq's task in 1m 32s —— View job Review: python coverage governance CINo blockers; this can merge.
Both findings from the earlier review round were fixed in the follow-up commit
The coverage-baseline.json was also regenerated against the Ubuntu/3.12 CI run itself so the ratchet reproduces at delta 0.00, matching the commit's stated intent. No new issues found in the delta: gate logic (ratchet, diff, baseline-check), Makefile targets, and |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| - name: Install dependencies | ||
| run: uv sync --locked --all-extras --dev | ||
| run: uv sync --frozen --all-extras --dev --python ${{ matrix.python }} |
There was a problem hiding this comment.
[blocker] This silently removes the only CI check that verifies uv.lock is up to date with pyproject.toml. The previous step used uv sync --locked ..., which fails if the lockfile is stale; switching to --frozen (needed to add --python ${{ matrix.python }}) makes uv sync accept a stale lockfile without error. Grepping the repo, no other job or Makefile target uses --locked for this purpose (Makefile's install-deps already used --frozen before this PR, but this was the one CI step enforcing lockfile freshness). After this change, a PR that edits a dependency in pyproject.toml without running uv lock will no longer be caught by CI.
Suggest keeping the freshness check, e.g. uv sync --locked --all-extras --dev and setting the interpreter via uv python pin/UV_PYTHON env instead of the --python flag, or adding a separate uv lock --check/--locked step.
| def parse_changed_lines(diff: str) -> dict[str, set[int]]: | ||
| """Return new-side hunk lines; deleted lines never enter the denominator.""" | ||
| changed: dict[str, set[int]] = {} | ||
| path: str | None = None | ||
| for line in diff.splitlines(): | ||
| if line.startswith("+++ b/"): |
There was a problem hiding this comment.
[nit] path is never reset/cleared between files, and only updated on +++ b/... lines. A deleted file's diff shows +++ /dev/null, which doesn't match, so its hunks would silently be attributed to whatever path was last set from an earlier file. Currently harmless because the only caller (git_diff_inputs) filters with --diff-filter=ACMR (no deletions ever reach this function), but it's a latent bug if parse_changed_lines is ever reused on a raw diff. Consider setting path = None when the new-file line isn't +++ b/....
The bootstrap baseline was generated on macOS; the Ubuntu 3.12 CI run produces slightly lower line and branch coverage, so the ratchet failed by more than its 0.05pp tolerance. Replace it with the candidate emitted by that CI run so the ratchet reproduces at delta 0.00. Also restore uv sync --locked on the unit job, since dropping it for --frozen removed the only CI check that uv.lock matches pyproject.toml, and reset the parser path on a +++ /dev/null target so a deleted file's hunks cannot be misattributed if parse_changed_lines runs on an unfiltered diff. Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
2 participants
Summary
Adds Python coverage governance to CI so existing coverage cannot silently
regress and new code stays well tested, without pretending the project is
already near 100%.
scripts/coverage_gate.pyplusmake coverage*targets compute line andbranch coverage for the
ravenpackage and drive four gates:.github/coverage-baseline.json(0.05pp tolerance for rounding noise);(comments, deletions, and non-executable lines are excluded);
branch coverage relative to the target branch.
every gate, uploads xml/json/html artifacts on every run (
if: always()),and keeps diff coverage as a PR-only gate.
pyproject.tomlconfigurescoverage.py(branch mode,ravensource, threedocumented omissions); generated reports are gitignored.
than auto-raising, so the diff-coverage gate is what guards new code while the
ratchet only blocks regression.
The production-file pathspec uses git glob magic (
:(glob)raven/**/*.py) sotop-level modules such as
raven/__init__.pyare not silently dropped from diffcoverage; a regression test covers this.
Type
Verification
uv run --frozen --python 3.12 pytest tests/test_coverage_gate.py -q-> 13 passeduv run --frozen --python 3.12 --extra dev ruff check scripts/coverage_gate.py tests/test_coverage_gate.py-> All checks passeduv run --frozen --python 3.12 --extra dev ruff format --check scripts/coverage_gate.py tests/test_coverage_gate.py-> already formattedRelevant tests pass locally
Relevant lint / type checks pass locally
User-facing docs or screenshots are updated when needed
Risk
No runtime
ravencode changes; this is CI and tooling only. The new gates canfail PRs that lower coverage, which is the intended behavior. Rollback is a
plain revert of this commit (or disabling the coverage steps in the unit job).
Related Issues
N/A