Test comparison evidence through answer packing

[BigSimmo]

Summary

• replay the admission/discharge comparison document, score, and ordering shape from current-main canary run 30009207429
• assert deterministic answer ranking and cross-document packing retain evidence from both sides

Why

The live answer omitted the specific Admission of Community Patients source even though retrieval found it. This guard proves the deterministic ranking/packing stages do not reproduce that loss, preventing a speculative retrieval-score or comparator change and narrowing follow-up to later live generation/citation selection.

Verification

• npx vitest run tests/rag-fast-path-ordering.test.ts tests/answer-ranking.test.ts tests/cross-document-synthesis.test.ts tests/rag-answer-fallback.test.ts --reporter=dot — 75/75 passed
• npm run eval:rag:offline — 21 suites, 307/307 passed; 36 golden cases validated
• npm run verify:cheap — 365 files, 3,237 passed, 1 skipped; lint and typecheck passed
• git diff --check — passed

Risk and rollback

Test-only. No app, retrieval, ranking, generation, threshold, fixture, or provider behavior changes. Rollback is a plain revert.

RAG impact: no retrieval behaviour change — test-only evidence pins the existing deterministic comparison packing result.

Checks not run

• no additional provider run for this PR; the one separately approved merged-main diagnostic canary is already running
• no build/browser suite because this is a Node unit-test-only change

---

Note

Low Risk
Test-only change with no application, retrieval, ranking, or generation behavior modifications.

Overview
Adds a regression test in answer-ranking.test.ts that replays the document IDs, scores, and ordering from canary run 30009207429 for the query Compare admission and discharge requirements.

The test runs rankAnswerEvidence then buildCrossDocumentSynthesisPlan (comparison mode) and asserts the top five packed results still include Admission Of Community Patients and at least one discharge-related chunk. That pins deterministic ranking and cross-document packing so a live omission of that admission source is not blamed on speculative retrieval-score or comparator tweaks—follow-up stays on later generation or citation selection.

^{Reviewed by Cursor Bugbot for commit 5e947a8. Configure here.}

[coderabbitai]

Warning

Review limit reached

@BigSimmo, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 18 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e6d55545-6306-46b1-9c70-70499de25708

📥 Commits

Reviewing files that changed from the base of the PR and between c24f2e8 and 5e947a8.

📒 Files selected for processing (1)

• tests/answer-ranking.test.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/rag-residual-reproducers

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e947a8808

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Assert that comparison packing actually ran

Because this immediately takes .results from buildCrossDocumentSynthesisPlan without checking that the plan is enabled with reason === "comparison", the guard can still pass if comparison packing is accidentally disabled: the function's disabled path returns the already-ranked input, and this fixture can still satisfy the later admission/discharge checks without proving the balancing step ran. For this RAG-protected answer-ranking guard, assert the plan metadata and pin the discharge document by id before treating it as evidence that packing retained both comparison sides.

AGENTS.md reference: AGENTS.md:L228-L235

Useful? React with 👍 / 👎.