fix(flows): stop byte-exact quote matching from discarding paper builds - #150
Open
ve3bmv wants to merge 1 commit into
Open
fix(flows): stop byte-exact quote matching from discarding paper builds#150ve3bmv wants to merge 1 commit into
ve3bmv wants to merge 1 commit into
Conversation
PaperFlow's semantic build asserted every model quote as a byte-exact substring of its chunk. Extracting a multi-column PDF interleaves column-gap padding and line-break hyphenation into that text, so a quote copied faithfully off the rendered page is almost never byte-exact: the research stage raised ValueError and the whole build produced no summary. Reproduced 5/5 on arXiv:2506.21775 and on the golden e2e paper 1706.03762v7; an instrumented run then recovered 18/18 quotes that differ from their chunk only in whitespace. Match quotes with whitespace removed on both sides, in the knowledge layer that owns the citation contract, and reuse it at all three check sites. A paraphrase or an invented sentence still fails. Whitespace tolerance alone left a stochastic failure: an occasional quote diverges by more than whitespace and killed the build again on re-run. The research draft is an internal intermediate and its quote is optional evidence, so an unsupported quote now drops while its finding keeps claim, chunk, and page. Fabricated chunk indices and pages still raise, and coverage still fails loudly through min_summary_citations and min_summary_pages. The canonical from_draft contract stays strict. Prompt text is deliberately unchanged, so the instructions hash, producer identity, and artifact IDs are stable. Also widen one e2e coverage keyword group: a faithful summary may report the machine-translation result as BLEU/WMT without writing the word "translation". The paper-flow job has been skipping in CI for lack of an OPENAI_API_KEY secret, so this brittleness was never observed. scripts/verify.sh passes; scripts/verify_pdf_rag_e2e.py now reports PASS with citations on 9 source pages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ve3bmv
force-pushed
the
fix/paper-quote-validation-whitespace
branch
from
August 21, 2026 12:42
c18d616 to
1482155
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PaperFlow(PaperSemanticCfg).buildasserted every model quote as a byte-exact substring of its chunk. Extracting a multi-column PDF interleaves column-gap padding and line-break hyphenation into that text, so a quote copied faithfully off the rendered page is almost never byte-exact, and the research stage raisedValueErrorbefore any summary was produced. The module docstring already recorded the symptom on1706.03762v7("the sampled models paraphrased ... produced no summary line"); the underlying cause is the comparison, not the models.Reproduced 5/5 on arXiv:2506.21775 and on the golden e2e paper 1706.03762v7. An instrumented run then recovered 18/18 quotes whose only difference from their chunk was whitespace, for example the model's
"This gives us the cost savings from pre-reconstitution trading."against the extracted"This gives us the cost savings from pre-\n reconstitution tra".This PR fixes it in two layers.
Whitespace-insensitive matching.
quote_matches_chunk_textlives in the knowledge layer that owns the citation contract and is reused at all three check sites (research draft,PaperGlobalSummary.from_draft, and thePaperSemanticResultvalidator). A paraphrase or an invented sentence still fails.Graceful degradation in the research stage only. Whitespace tolerance alone left a stochastic failure: an occasional quote diverges by more than whitespace and killed the build again on re-run. A research draft is an internal intermediate and its quote is optional evidence, so an unsupported quote is now dropped while its finding keeps claim, chunk, and page. Fabricated chunk indices and pages still raise, and coverage still fails loudly through
min_summary_citationsandmin_summary_pages.Failure Semanticsin the design listed only the out-of-group case for this stage, so dropping the quote follows the documented contract more closely than raising did.Three points are deliberate and worth a reviewer's judgement:
from_draftstays strict. Every failure observed live came from the research stage, but a reducer that re-mangles a non-whitespace character can still raisePaperCitationValidationError. Extending the same degradation to the canonical constructor would change a published-artifact contract, so this PR leaves it alone.quote_matches_chunk_textis exported fromquantmind.knowledge. The apex layer needs the predicate that defines the contract; keeping it private would mean a cross-layer private import or a duplicated copy._summary_instructions_hashfeeds producer identity, so leaving_RESEARCH_INSTRUCTIONSand_SUMMARY_INSTRUCTIONSuntouched keeps artifact IDs stable and avoids re-minting stored summaries.One test-quality change rides along: the
verify_pdf_rag_e2e.pycoverage assertion required the literal word "translation", but a faithful summary may report the machine-translation result as BLEU/WMT instead, which is exactly what happened on the first passing run. The group now acceptsbleuandwmt.Related Issue
No existing issue covers this. Filing one first would not add information this body lacks, so the diagnosis is recorded here instead.
Separately, and outside the scope of this PR: the
paper-flowjob in.github/workflows/e2e.ymlhas been reporting green while skipping, because it emits::notice::Skipping Paper Flow V1 E2E because OPENAI_API_KEY is not configuredand then guards the run step on that flag (run 31565118318 on 2026-08-12 is one example). That is why a defect reproducible on every two-column paper survived inmaster. The local run in the Verification section below appears to be the first live execution of this slice in over a month. Configuring the repository secret, or making the skip a visible neutral status, needs repository admin rights and is left to the maintainers.Verification
scripts/verify_structure_e2e.pyis not applicable: the structure path builds no citations and never compares a quote.New offline tests cover both layers, success and failure:
quote_matches_chunk_textaccepts a column-gap-mangled quote and rejects a paraphrase;_validate_research_draftkeeps a respaced quote and drops an unsupported one;_build_summaryaccepts a citation quote that differs from its chunk only in whitespace.Checklist
type(scope): summary.bash scripts/verify.shpasses.