Skip to content

fix(flows): stop byte-exact quote matching from discarding paper builds - #150

Open
ve3bmv wants to merge 1 commit into
LLMQuant:masterfrom
ve3bmv:fix/paper-quote-validation-whitespace
Open

fix(flows): stop byte-exact quote matching from discarding paper builds#150
ve3bmv wants to merge 1 commit into
LLMQuant:masterfrom
ve3bmv:fix/paper-quote-validation-whitespace

Conversation

@ve3bmv

@ve3bmv ve3bmv commented Aug 21, 2026

Copy link
Copy Markdown

Summary

PaperFlow(PaperSemanticCfg).build asserted every model quote as a byte-exact substring of its chunk. Extracting a multi-column PDF interleaves column-gap padding and line-break hyphenation into that text, so a quote copied faithfully off the rendered page is almost never byte-exact, and the research stage raised ValueError before any summary was produced. The module docstring already recorded the symptom on 1706.03762v7 ("the sampled models paraphrased ... produced no summary line"); the underlying cause is the comparison, not the models.

Reproduced 5/5 on arXiv:2506.21775 and on the golden e2e paper 1706.03762v7. An instrumented run then recovered 18/18 quotes whose only difference from their chunk was whitespace, for example the model's "This gives us the cost savings from pre-reconstitution trading." against the extracted "This gives us the cost savings from pre-\n reconstitution tra".

This PR fixes it in two layers.

Whitespace-insensitive matching. quote_matches_chunk_text lives in the knowledge layer that owns the citation contract and is reused at all three check sites (research draft, PaperGlobalSummary.from_draft, and the PaperSemanticResult validator). A paraphrase or an invented sentence still fails.

Graceful degradation in the research stage only. Whitespace tolerance alone left a stochastic failure: an occasional quote diverges by more than whitespace and killed the build again on re-run. A research draft is an internal intermediate and its quote is optional evidence, so an unsupported quote is now dropped while its finding keeps claim, chunk, and page. Fabricated chunk indices and pages still raise, and coverage still fails loudly through min_summary_citations and min_summary_pages. Failure Semantics in the design listed only the out-of-group case for this stage, so dropping the quote follows the documented contract more closely than raising did.

Three points are deliberate and worth a reviewer's judgement:

  • from_draft stays strict. Every failure observed live came from the research stage, but a reducer that re-mangles a non-whitespace character can still raise PaperCitationValidationError. Extending the same degradation to the canonical constructor would change a published-artifact contract, so this PR leaves it alone.
  • quote_matches_chunk_text is exported from quantmind.knowledge. The apex layer needs the predicate that defines the contract; keeping it private would mean a cross-layer private import or a duplicated copy.
  • Prompt text is unchanged. _summary_instructions_hash feeds producer identity, so leaving _RESEARCH_INSTRUCTIONS and _SUMMARY_INSTRUCTIONS untouched keeps artifact IDs stable and avoids re-minting stored summaries.

One test-quality change rides along: the verify_pdf_rag_e2e.py coverage assertion required the literal word "translation", but a faithful summary may report the machine-translation result as BLEU/WMT instead, which is exactly what happened on the first passing run. The group now accepts bleu and wmt.

Related Issue

No existing issue covers this. Filing one first would not add information this body lacks, so the diagnosis is recorded here instead.

Separately, and outside the scope of this PR: the paper-flow job in .github/workflows/e2e.yml has been reporting green while skipping, because it emits ::notice::Skipping Paper Flow V1 E2E because OPENAI_API_KEY is not configured and then guards the run step on that flag (run 31565118318 on 2026-08-12 is one example). That is why a defect reproducible on every two-column paper survived in master. The local run in the Verification section below appears to be the first live execution of this slice in over a month. Configuring the repository secret, or making the skip a visible neutral status, needs repository admin rights and is left to the maintainers.

Verification

bash scripts/verify.sh                      # 431 passed, coverage 85.68% (floor 75%)
python scripts/verify_pdf_rag_e2e.py        # [PASS] paper-flow-v1, citations on 9 source pages

scripts/verify_structure_e2e.py is not applicable: the structure path builds no citations and never compares a quote.

New offline tests cover both layers, success and failure: quote_matches_chunk_text accepts a column-gap-mangled quote and rejects a paraphrase; _validate_research_draft keeps a respaced quote and drops an unsupported one; _build_summary accepts a citation quote that differs from its chunk only in whitespace.

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • bash scripts/verify.sh passes.
  • Every applicable live-network component smoke test passes, or this PR states why none applies.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes.

PaperFlow's semantic build asserted every model quote as a byte-exact
substring of its chunk. Extracting a multi-column PDF interleaves
column-gap padding and line-break hyphenation into that text, so a quote
copied faithfully off the rendered page is almost never byte-exact: the
research stage raised ValueError and the whole build produced no summary.
Reproduced 5/5 on arXiv:2506.21775 and on the golden e2e paper
1706.03762v7; an instrumented run then recovered 18/18 quotes that differ
from their chunk only in whitespace.

Match quotes with whitespace removed on both sides, in the knowledge
layer that owns the citation contract, and reuse it at all three check
sites. A paraphrase or an invented sentence still fails.

Whitespace tolerance alone left a stochastic failure: an occasional quote
diverges by more than whitespace and killed the build again on re-run.
The research draft is an internal intermediate and its quote is optional
evidence, so an unsupported quote now drops while its finding keeps
claim, chunk, and page. Fabricated chunk indices and pages still raise,
and coverage still fails loudly through min_summary_citations and
min_summary_pages. The canonical from_draft contract stays strict.

Prompt text is deliberately unchanged, so the instructions hash, producer
identity, and artifact IDs are stable.

Also widen one e2e coverage keyword group: a faithful summary may report
the machine-translation result as BLEU/WMT without writing the word
"translation". The paper-flow job has been skipping in CI for lack of an
OPENAI_API_KEY secret, so this brittleness was never observed.

scripts/verify.sh passes; scripts/verify_pdf_rag_e2e.py now reports PASS
with citations on 9 source pages.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ve3bmv
ve3bmv force-pushed the fix/paper-quote-validation-whitespace branch from c18d616 to 1482155 Compare August 21, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant