feat(math): preserve mathematical notation across every frontend - #76
Open
erkurtharun wants to merge 11 commits into
Open
feat(math): preserve mathematical notation across every frontend#76erkurtharun wants to merge 11 commits into
erkurtharun wants to merge 11 commits into
Conversation
… math A dollar pair delimits math for every renderer that supports it, so a document saying "costs $100 and $80" was serialized unescaped and read back as an equation spanning the two amounts. Escape a dollar that has a later dollar in the same run, on the same pairing rule the other inert-when- lone delimiters already use; a single dollar opens nothing and stays literal, which leaves the common currency case untouched.
Word carries three things this converter dropped. A `w:vertAlign` run is
super- or subscript, and flattening it changes a value rather than a
presentation: `10^-3` became `10-3`, `H_2 O` became `H2O`. An `m:oMath`
equation vanished entirely, and an `m:oMathPara` took its whole paragraph
with it, because all three namespace filters in the docx frontend admit
only `w:*` and OMML lives in its own namespace.
`Style` gains `vert_align`. It is a value, not a toggle: ECMA-376 s17.7.3
closes the toggle set and `w:vertAlign` is not in it, so it cannot ride the
XOR parity `Toggles` uses, and `on_off` cannot even tell `superscript` from
`subscript` since both are outside the false-set. It resolves through the
ordinary nearest-specification path instead, and the other frontends fill it
too: DrawingML `a:rPr/@baseline` and ODF `style:text-position`.
`Inline::Math` carries LaTeX, rendered between dollars. It is the only inline
whose payload reaches the output unescaped, since `escape_text` escapes every
backslash and would destroy each command, so the producer owns making the
body safe.
`formats::docx::omml` translates the OMML element set. It parses to a small
tree before emitting, because bracing is a question about a node's shape --
`y^{2}` used as another script's base needs braces or LaTeX reads a double
superscript, while `\frac{a}{b}` does not -- and that cannot be decided by
inspecting the emitted string. Property bags are read only through named
lookups and never enumerated: each is optional in the schema, Word omits them
whenever every property takes its default, and an absent bag must resolve
exactly as an empty one does. Defaults follow the spec, including the n-ary
operator defaulting to the integral rather than the sum.
Hostile input is bounded and neutralised. Nesting past 64 levels degrades to
the subtree's text rather than recursing, output is capped per equation
because wrappers multiply through nesting, and text runs are LaTeX-escaped so
a dollar cannot close the span and hand the rest of the document to the
Markdown parser, nor a backslash smuggle in a command. A delimiter separator
is emitted as `\mid` rather than a bare pipe, which would split the table row
the equation sits in.
Emitted commands stay inside the subset KaTeX implements, and a body holding
a dollar takes the longer fence: Markdown math parsers do not honour a
backslash-escaped dollar when scanning for the closing delimiter.
Four bugs, all silent, found by adversarial review of the previous commit and each reproduced before it was fixed. A `w:ins` wrapping exactly one element dropped it. `parse_seq` collapses a one-element sequence to that element, so the `if let Node::Seq(..)` guard on the revision-mark branch never matched and control fell through to `continue`. Tracked changes around a single fraction lost the whole equation. An unmapped n-ary operator became an integral and an unmapped accent became a hat. Well-formed, KaTeX-valid and a different expression than the document's. Both now pass the glyph through, which says what the document said; an unmapped group character draws nothing rather than an underbrace. `m:begChr`, `m:endChr` and `m:sepChr` are author-supplied and reached the LaTeX body without escaping, so a delimiter of `$` put a bare dollar in a payload whose contract says there is none, and one of `\` could open a command. They take the same escaping as any other text now. Degradation was silent everywhere: the crate routes recovery through `log` and this module logged nothing. EPUB and any HTML input carried the same losses and one more. `<sup>` and `<sub>` were flattened, so the same content read `10<sup>-3</sup>` from DOCX and `10-3` from EPUB. MathML was worse than dropped: `<math>` is not a container tag, so the walker descended into it and emitted the presentation tree as symbol soup *and* the `<annotation encoding="application/x-tex">` beside it as visible text, backslashes doubled by the escaper. The annotation is exact where re-deriving LaTeX from the presentation tree is not, so it is taken when present; without one the characters are kept and the shape is lost.
resolve_entity knew about forty names. Everything else fell through to the literal `&name;`, which the Markdown writer then escaped again, so `β` reached the reader as `&beta;` — the reference destroyed rather than merely unresolved. MathML depends on these names more than prose does: `α`, `∑`, `⁢` are how most producers spell their operators. The table is generated from the WHATWG entities.json, restricted to the names that carry a trailing semicolon, which is every name XML admits. Sorted for binary search; the format, separator and combining characters are written as escapes so nothing invisible sits in the source.
The EPUB path only recovered an equation when the producer had embedded a TeX annotation. Most do not, and without one the markup reached the writer as its bare characters: a <mfrac> over 1 and 3 read as "13", which is not a degraded fraction but a different number. Structure is what gets translated. Glyphs are left alone — KaTeX takes Unicode operators and Greek directly and has metrics for them, so mapping them to commands would only be a chance to pick the wrong one. The invisible operators are the exception: they carry no glyph, so passing them through would put unreadable codepoints in front of a reader. Covers the presentation set: scripts (including mmultiscripts), fractions, radicals, under/over with limits and accents, tables, mfenced, semantics, and the token elements. Symbols are trimmed and prose is not, so the space in <mtext>if </mtext> survives as a word boundary. The escaping and Unicode-to-command tables move to shared/latex.rs, which both math frontends now resolve against, with the same rule as before: an unmapped glyph is passed through, never guessed at.
\super, \sub and \nosupersub were not in the dispatcher, so RTF alone kept flattening what the other frontends now preserve: 10\super -3 came out as 10-3, the exact shape of the original report. \upN and \dnN carry an offset in half-points rather than a toggle, so only 0 means the baseline and an absent parameter takes the spec's default of 6. \plain already resets vert_align through Style::PLAIN. A paragraph style may carry the property too, so the stylesheet parser records it in the delta it already builds for bold and italic.
Word 97-2003 never read sprmCIss, and the PPT frontend hardcoded the baseline, so both flattened what every other frontend now keeps. DOC gains sprmCIss (0x2A48: 1 superscript, 2 subscript) and sprmCHpsPos (0x4845), the signed half-point offset Word writes for "raised by" rather than for the checkbox — the same pair as RTF's \super and \up. PPT already walked past the TextCFException position field, a signed percentage of the font size, so reading it is a matter of not skipping it. Both are covered by handmade fixtures with the property in real record bytes, because a unit test over a synthetic grpprl proves the parser and not the path that reaches it.
Neither frontend reached the equation at all, and the loss was total rather than partial: a slide read "Energy: done" and a document "Einstein said and stopped." PowerPoint writes an equation as a14:m inside an mc:AlternateContent whose fallback is a picture of it. The paragraph walker skipped every child outside the a namespace, so not even the fallback text survived. It now resolves the AlternateContent and hands a14:m to the OMML converter, which moves to shared/ because two frontends read it. ODF keeps a formula in a sub-document of its own, referenced by draw:object. Following the reference gives MathML, which the MathML converter already translates; some producers put it inline in the element instead, so both spellings are read. A reference that resolves to nothing still degrades to the frame's alternative text. The TeX-annotation preference moves into the MathML converter so every caller gets it: what the author wrote beats anything derived from the layout, while ODF's StarMath annotation is not LaTeX and is ignored.
The math frontends say they emit LaTeX inside the subset KaTeX implements. Nothing measured that, so the claim held only as far as the reviewer's eye. The gate walks the document model rather than the Markdown — the model carries latex and display directly, so no re-parsing stands between what the converter produced and what is checked — and renders each equation with throwOnError. Every fixture named handmade-math must contribute at least one, because a walk that quietly stops finding anything would otherwise pass while measuring nothing. Verified to fail: emitting an undefined control sequence from the MathML converter breaks it, as it should.
\cs was not in the dispatcher and the stylesheet parser recorded only paragraph styles, so every property a character style carried was lost — bold and italic as much as the script this branch went after. RTF numbers character styles in a space of their own, so \cs15 and \s15 are different styles and need separate maps; the \sbasedon resolution is the same either way and moves into a function both use. A character style applies over the run's own formatting rather than replacing it, which is what the tri-state delta already does.
Found by fuzzing the math XML inside the fixtures and rendering everything
that survived: 9 of 1631 equations came out unrenderable, all from the same
place. A TeX annotation was passed through verbatim, so a broken one became
broken output while a perfectly good presentation tree sat beside it.
The annotation still wins when it can parse. Three ways it certainly cannot
now yield to the presentation tree instead: unbalanced braces, a trailing
script marker with no argument, and a bare `$` — that last one ended the
math span early and handed the rest of the document to the Markdown parser,
which is the same defect the document-text path was already guarded against
and this path bypassed.
A literal `^` or `~` was spelled `\^{}`, which is a text-mode accent LaTeX
rejects in math mode; it is now written as the text it is.
The KaTeX gate runs in strict mode, because a construct KaTeX renders while
warning about it is still not LaTeX.
What remains unrenderable is an annotation whose control sequence the author
misspelled. Telling that from a macro would take a command table, and a
table would reject legitimate LaTeX that is not in it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
anydoc's own design note says the audience is AI agents and the output is
optimised for semantic quality. Mathematical notation was the one place that
promise broke, and it broke by changing values rather than presentation:
10⁻³ mol/L10-3 mol/LH₂OH2OE = mc²(MathML, no TeX annotation)E=mc2⅓as<mfrac><mn>1</mn><mn>3</mn></mfrac>13β&beta;A lost bold is a lost presentation. A lost superscript is a different number,
and no consumer downstream can tell it was ever an exponent.
What changed
A run's position on the baseline is now part of the model.
Stylegainsvert_align, read fromw:vertAlign(DOCX),style:text-position(ODF), thebaseline percentage (PPTX and PPT),
sprmCIssandsprmCHpsPos(DOC), and\super/\sub/\nosupersub/\up/\dn(RTF). It renders as<sup>/<sub>, which GFM already carries.Equations become LaTeX in
$…$/$$…$$. Two converters feed it:shared/omml.rs— Office Math, from DOCX and from thea14:mPowerPointwraps in an
mc:AlternateContent. That AlternateContent was skipped whole,so not even its fallback text survived.
shared/mathml.rs— MathML, for HTML and EPUB, and for the formulasub-document ODF references through
draw:object. Only some MathML carriesa TeX annotation; without one the markup used to arrive as bare characters.
Structure is translated; glyphs are not. KaTeX takes Unicode operators and
Greek directly and has metrics for them, so
αand∑are left as thedocument wrote them — mapping them to commands would only be a chance to pick
the wrong one. The invisible operators (U+2061–2064) are the exception: they
carry no glyph, so passing them through would put unreadable codepoints in
front of a reader. An unmapped glyph is passed through, never guessed at.
Two escaping defects came out of this and are fixed. A paired
$indocument text would open a math span and hand the rest of the document to the
Markdown parser; so would a bare
$inside a TeX annotation. Both are nowescaped or refused.
Named character references were resolved for about forty names; everything
else fell through as literal text and was then escaped again, so
βreached the reader as
&beta;— destroyed rather than merely unresolved.The full HTML5 table is generated from the WHATWG
entities.json. MathMLdepends on these more than prose does.
RTF character styles (
\cs) were not applied at all, so every property onecarried was lost — bold and italic as much as the script this branch went
after.
Verification
cargo fmt,cargo clippy -- -D warnings, and clippy onwasm32-unknown-unknown.measured.
node/katex.test.mjswalks the document model — which carrieslatexanddisplaydirectly, so no re-parsing stands between what theconverter produced and what is checked — and renders every equation with
throwOnErrorandstrict: "error". Verified to fail: emitting an undefinedcontrol sequence breaks it.
mainover all 72 fixtures: 66 byte-identical,6 changed, and all 6 carry mathematics. No panics.
inside the packages. No panics. Rendering everything that survived found 9
unrenderable equations, all from annotations passed through verbatim — that
is the defect the annotation guard above fixes.
entity references, a self-referencing ODF object, a cyclic RTF style. All
bounded; the size and cell caps truncate while staying balanced. 20,005
equations, zero KaTeX failures.
unchanged (
+0.2%DOCX,−7.1%EPUB,−4.6%RTF, byte-identical output).Math-heavy DOCX costs
+1.0%while emitting twice the content.Known limits
10⁻³→10-3). Theloss is at the calamine boundary:
<si>runs collapse intoData::String,and neither the shared-string table nor per-cell indices are public, so
keying on anything else would be a guess. Preserving it means reading XLSX
worksheets directly rather than through calamine — a separate decision.
Telling that from a macro would take a command table, and a table would
reject legitimate LaTeX that is not in it.
Related
PDFs bypass the document model and go through
pdf-inspector, where the sameclass of defect lives: scripts escape their line and reorder. That half is
firecrawl/pdf-inspector#326 —
10 -3 mol/L→10⁻³ mol/L, and565.0 kg/m+ a stray/kgline →565.0 kg/m³. PDF coverage here onlyimproves once that PR lands and a release follows, since this crate pins
pdf-inspector = "0.1.7". The chain was verified locally with a[patch.crates-io]override, which is not part of this branch.