eval: populate canary source governance report#1099
Conversation
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
📝 WalkthroughWalkthroughThe quality evaluator now accepts precomputed golden retrieval results for Source Governance reporting. The canary workflow supplies that artifact, while RAG-only answer gating remains based on existing RAG results. Documentation and tests reflect the new data flow and validation. ChangesSource governance reporting
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant EvalCanary
participant EvalQuality
participant GoldenRetrievalJSON
participant QualityReport
EvalCanary->>GoldenRetrievalJSON: produce golden retrieval results
EvalCanary->>EvalQuality: run quality evaluation with artifact path
EvalQuality->>GoldenRetrievalJSON: load and validate results
EvalQuality->>QualityReport: compute governance metrics from supplied results
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6adf2c333e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // canary can still supply the preceding golden-retrieval artifact so this | ||
| // report renders its source-governance metadata without rerunning retrieval | ||
| // or changing the answer gate's pass/fail contract. | ||
| const governance = topResultGovernanceCounts(args.sourceGovernanceResults ?? args.retrievalResults); |
There was a problem hiding this comment.
When --source-governance-results is supplied during a full eval:quality run, this line computes the Source Governance table and the top-result threshold checks from the supplied artifact instead of the retrieval results produced by the same run. That means a stale or empty artifact can make --fail-on-threshold pass despite current retrieved top results being outdated/review-required, or fail a clean run based on unrelated data; use the external artifact only when retrievalResults is empty (the --rag-only case) or reject this flag outside --rag-only.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/outstanding-issues.md`:
- Line 81: Update the resolution date in issue `#031`’s row to the actual
merge/resolution date rather than the future date currently recorded, while
preserving the existing auditable outcome and all other row content.
In `@scripts/eval-quality.ts`:
- Around line 602-614: Update the report flow around
summarizeGoldenRetrievalResults and topResultGovernanceCounts so
--source-governance-results is accepted only for --rag-only runs; non-rag-only
runs must always use live retrieval results, preserving existing retrieval
gates. In the governance normalization/validation logic near
topResultGovernanceCounts, reject malformed or unknown document_status,
clinical_validation_status, and extraction_quality values or classify them as
review-required rather than zero-risk. Add coverage for both artifact-scope
rejection and invalid governance values failing closed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a3277cd9-6ff9-420e-9a62-b7bc0c424415
📒 Files selected for processing (6)
.github/workflows/eval-canary.ymldocs/observability-slos.mddocs/outstanding-issues.mdscripts/eval-quality.tstests/eval-canary-workflow.test.tstests/eval-quality.test.ts
|
|
||
| | ID | Type | Summary | Outcome | Resolved | | ||
| | ---- | ----- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | ||
| | #031 | issue | Populate canary Source Governance table | The answer-quality step now consumes the preceding `golden-retrieval.json` only for source-governance reporting. Offline replay of run `30018289898` populated 338 top results, including 202 review-required entries, while retaining zero retrieval cases and no additional threshold failures. Retrieval and ranking behavior are unchanged. | 2026-07-24 | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the actual resolution date.
This archives #031 as resolved on July 24, 2026, which is tomorrow relative to July 23, 2026. Use the actual merge/resolution date so the durable issue ledger remains auditable.
As per coding guidelines, docs/outstanding-issues.md is durable memory and resolved rows retain an auditable outcome.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/outstanding-issues.md` at line 81, Update the resolution date in issue
`#031`’s row to the actual merge/resolution date rather than the future date
currently recorded, while preserving the existing auditable outcome and all
other row content.
Source: Coding guidelines
| sourceGovernanceResults?: GoldenRetrievalResult[]; | ||
| ragResults: RagQualityResult[]; | ||
| sourceMetadataDebtAcceptance?: SourceMetadataDebtAcceptance; | ||
| providerMode?: EvalQualityProviderMode; | ||
| }) { | ||
| const providerMode = args.providerMode ?? "openai"; | ||
| const retrievalSummary = summarizeGoldenRetrievalResults(args.retrievalResults); | ||
| const ragSummary = summarizeRagQualityResults(args.ragResults, providerMode); | ||
| const governance = topResultGovernanceCounts(args.retrievalResults); | ||
| // `--rag-only` intentionally leaves retrieval metrics and gates empty. The | ||
| // canary can still supply the preceding golden-retrieval artifact so this | ||
| // report renders its source-governance metadata without rerunning retrieval | ||
| // or changing the answer gate's pass/fail contract. | ||
| const governance = topResultGovernanceCounts(args.sourceGovernanceResults ?? args.retrievalResults); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Fail closed on artifact scope and governance field values.
Line 614 lets a supplied artifact replace live retrieval governance during non---rag-only runs, changing existing retrieval gate behavior. Also, { document_status: {}, clinical_validation_status: {}, extraction_quality: {} } passes Lines 1219-1221 and is counted as zero-risk because all values are non-null but unrecognized.
Reject --source-governance-results unless --rag-only, and validate/normalize governance fields so invalid or unknown values are rejected or treated as review-required. Add coverage for both cases. The PR contract states retrieval threshold behavior remains unchanged and malformed artifacts fail closed.
Also applies to: 1205-1223
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/eval-quality.ts` around lines 602 - 614, Update the report flow
around summarizeGoldenRetrievalResults and topResultGovernanceCounts so
--source-governance-results is accepted only for --rag-only runs; non-rag-only
runs must always use live retrieval results, preserving existing retrieval
gates. In the governance normalization/validation logic near
topResultGovernanceCounts, reject malformed or unknown document_status,
clinical_validation_status, and extraction_quality values or classify them as
review-required rather than zero-risk. Add coverage for both artifact-scope
rejection and invalid governance values failing closed.
1 participant
Summary
Why
eval:quality --rag-onlyintentionally has no retrieval results, so its Source Governance table was all zero even thoughgolden-retrieval.jsonalready contained the metadata. This reuses that artifact only for reporting; retrieval metrics and threshold behavior remain unchanged.Offline artifact proof
Replayed canary run
30018289898:Verification
npm run test -- tests/eval-quality.test.ts tests/eval-canary-workflow.test.ts— 2 files, 36 tests passednpm run check:github-actions— passednpm run check:rag:fixtures— passed, 36 golden cases / 21 suitesnpm run eval:rag:offline— 21 suites, 307 tests passednpm run verify:cheap— 365 files, 3240 passed / 1 skippednpm run verify:pr-local -- --files ...— passed; build correctly skipped as non-build-affectingChecks not run
Risk and rollback
The remaining risk is workflow-level artifact availability; the preceding golden-retrieval step owns that file and the loader fails closed if it is missing or malformed. Rollback is the single commit in this PR.
RAG impact: no retrieval behaviour change — evaluation artifact reporting and tests only.
Note
Low Risk
Eval reporting and CI wiring only; retrieval ranking and answer threshold logic are unchanged aside from governance display sourcing.
Overview
Fixes the weekly canary answer-quality report showing all zeros in the Source Governance table while answer metrics still reflected high review-required rates.
eval:qualitygains--source-governance-results, which loads the precedinggolden-retrieval.jsonand feedstopResultGovernanceCountsfor reporting only. With--rag-only, retrieval case counts stay at zero and retrieval / top-result threshold gates are not applied from that path—only the governance table is filled.The eval-canary workflow passes
.local/eval-canary/golden-retrieval.jsoninto the answer-quality step so retrieval is not run twice. Docs describe the contract; #31 is closed inoutstanding-issues.md.Reviewed by Cursor Bugbot for commit 6adf2c3. Configure here.
Summary by CodeRabbit
New Features
Documentation
Tests