fix(validate): say which engine produced the coverage, and whose skips they are - #660
Merged
Conversation
…s they are
`evolith validate` with no flag runs the native evaluator. On a freshly
initialised satellite it decides 41 of 159 rules where `--engine opa`
decides 133, and nothing in the output said so. Every skip was published
and the totals were honest — the reader still concluded their repository
was the problem, because the denominator that moved was the engine's.
Three additions, smallest first:
- `ValidationResult.engine` carries which evaluator ran. Optional for the
same additive reason as the GT-569 counters; the service always sets it.
- The human report prints it ON the counts line ("… / 159 total — engine:
native"), the markdown report as a `**Engine:**` field, and the JSON
envelope and table payload carry the field. A captured coverage figure
with no engine beside it cannot be compared to another capture.
- A non-blocking `GOV-ENGINE-COVERAGE` advisory, plus a CLI hint, that fire
ONLY on the native engine and ONLY when skips outnumber checks — the
shape a reader misreads. Both say the attribution out loud: the skips
belong to the evaluator, not to the repository.
Deliberately not making the engines agree, and deliberately not switching
the default. 68-validate-engine-verdict-parity.mjs holds the two engines to
agreement on the verdicts they both reach and explicitly not to equal reach;
this reports which reach you got.
Closes #628
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: aarroyo <beyondnet.peru@gmail.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
📊 Bilingual Coverage ImpactPR Changes
Repository Coverage
To create skeletons: node .harness/scripts/generate-es-skeleton.mjs <file.md>Generated by GitHub Actions |
…travel
Three checks caught what the first commit missed, each in a different sense:
- `Test contract (SDK wire fidelity)` — the field must be DECLARED in
WIRE_VALIDATION_RESULT, not merely emitted. That guard exists because a
producer can add a field the SDK type does not describe and no consumer
finds out until it deserializes. Declared optional, like the GT-569
coverage fields: an envelope from a producer that predates it is valid.
- `Validate documentation` — the interface how-to pages are DERIVED from a
live capture, so a new field in the response shape drifts them. Regenerated
with gen-howto: `"engine": "<string>"` in construction and qa.
- `build-and-test` — max-lines. The #628 cases pushed
validate.coverage-report.spec.ts to 334 lines, so they move to their own
validate.engine-disclosure.spec.ts. Better home anyway: GT-569's file pins
that a denominator is reported, this one pins that it is attributed.
Verified in a clean worktree with the full workspace built: contract 47/47,
CLI validate 81/81, exploration 9/9, CLI lint 0 errors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
evolith validatewith no flag runs the native evaluator. On a freshly initialised satellite it decides 41 of 159 rules where--engine opadecides 133, and nothing in the output said so.Every skip was published and both totals were honest — the project's thesis held — and the reader still concluded their repository was the problem, because the denominator that moved was the engine's, not theirs.
What this adds
ValidationResult.engineRulesetValidatorService.validatealways sets itRules: 41 checked / 118 skipped / 0 errored / 159 total — engine: native**Engine:** native— this is the surface people paste into a PR, so it is where an unattributed coverage figure does the most damageGOV-ENGINE-COVERAGEadvisory + CLI hintWhere it stays quiet
The advisory and the hint fire only on the native engine and only when skips outnumber checks — the shape a reader misreads. A native run that decided most of its scope needs no explanation, an
--engine oparun has nowhere to be redirected to, and a row on every run is noise that teaches people to skim past it. Both directions are pinned by tests, including the tie (80/80stays quiet,79/81fires).What this deliberately does not do
It does not make the engines agree, and it does not switch the default.
.harness/scripts/ci/68-validate-engine-verdict-parity.mjsholds the two engines to agreement on the verdicts they both reach and explicitly not to equal reach. That position is unchanged; this reports which reach you got.Verified
In a clean worktree off
main, with a realcore-domainbuild: 81 CLI validate tests pass and 5 new core-domain tests pass. The full core-domain suite is 1989 passed / 11 failed, and all 11 are the three OPA suites failing onpolicy.wasmnot being compiled in a fresh checkout — CI compiles it.Closes #628
🤖 Generated with Claude Code