ci(guardian): publish score check on PR checklist - #1542
Conversation
Guardian evals only wrote a job summary, so the PR checklist showed GitHub's canned "failing after Xm" line. Mirror behavioral reporting: soft-fail the run step, hard-fail on missing results, and publish a dedicated guardian / score Check Run with pass-rate title text. Co-Authored-By: David Cramer <david@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dcramer
added
trigger-evals-guardian
and removed
risk: medium
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit afa1fe3. Configure here.
| name: guardian-evals | ||
| path: packages/junior-evals/guardian-results.json | ||
| if-no-files-found: warn | ||
| if-no-files-found: error |
There was a problem hiding this comment.
Failed evals skip artifact upload
Medium Severity
fail-on-failures: true on the publish step exits non-zero when Guardian cases miss, and the upload step’s if still implies a prior success() status. Failed runs therefore never upload guardian-results.json, even though the results gate already passed. That drops the artifact in the failure case where it is most useful for debugging.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit afa1fe3. Configure here.
1 participant
Guardian evals already computed pass rate text, but the PR checklist only showed GitHub's canned job line (
failing after 2m) because we never published a dedicated Check Run.Mirror the behavioral eval reporter path:
checks: writeguardian / scorewithfail-on-failures: trueso the checklist secondary line shows pass-rate title text (for exampleEval pass rate 94.4% — required 100.0%)Evidence: #1533
Requested by David Cramer.
--
View Junior Session [Sentry]