xml5ever: keep question marks in processing instruction data - #779
Open
FadeHack wants to merge 1 commit into
Open
xml5ever: keep question marks in processing instruction data#779FadeHack wants to merge 1 commit into
FadeHack wants to merge 1 commit into
Conversation
The PI after state was throwing away the '?' that got it there whenever that '?' turned out not to be the start of a '?>'. So a processing instruction like <?xml-stylesheet href="style.xsl?v=2"?> came out with the href pointing at the wrong resource, and no error was raised. XML 1.0 section 2.6 says a processing instruction runs up to the first '?>', so a '?' anywhere else is ordinary data and belongs in the data string. That is what libxml2 and Python's minidom do too. The same state had a second problem. On any other character it pushed that character and stayed in the PI after state, so the next '>' ended the instruction even though the character before it was not a '?'. That made <?target a?b>c?> stop early with data "ab". It now goes back to the PI data state, which is what the XML5 draft says to do anyway. At EOF the pending '?' was dropped for the same reason, so an unterminated instruction lost its last character. It is kept now. One xml5lib test, "PI tag with char in PiAfter state", expects the old output. It follows the XML5 draft, whose PI after state appends the '?' when it sees another '?' but silently drops it for anything else. That asymmetry looks like an oversight in the draft rather than something intended, so I skipped that test with a note instead of matching it. Fixes servo#774
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #774
The PI after state was throwing away the
?that got it there whenever that?turned out not to be the start of a?>. So this:came out of the tokenizer as
href="style.xslv=2", pointing at the wrong resource, and nothing was reported.<?target a?b?c?>came out asabc.XML 1.0 section 2.6 says a processing instruction runs up to the first
?>, so a?anywhere else is ordinary data and belongs in the data string. libxml2 and Python's minidom both keep it.What is in here
Three things, all in the same state:
?we now push the pending?into the data and stay in the PI after state, so a run of question marks is preserved except for the one that actually closes the instruction.?and the character, then go back to the PI data state. It used to push the character and stay in the PI after state, which meant the next>closed the instruction even though the character before it was not a?. That made<?target a?b>c?>stop early with dataab. Going back to the PI data state is what the XML5 draft says to do here anyway.?was dropped for the same reason, so an unterminated instruction lost its last character. It is kept now.About the skipped test
One xml5lib test,
PI tag with char in PiAfter state, expects<?xml \t\n ?m?>to produce datam, so it disagrees with this change. It matches the XML5 draft, whose PI after state appends the?when it sees another?but silently drops it for anything else. That asymmetry reads like an oversight in the draft rather than something intended, and following it means the data of a processing instruction can silently differ from what every other XML parser produces.Since the test data lives in a submodule I cannot change here, I skipped that one case with a comment pointing at the issue. Happy to take this a different way if you would rather the draft won, or if you would rather I send a patch to Ygg01/xml5lib-tests first and bump the submodule.
Testing
Added unit tests in
xml5ever/src/tokenizer/mod.rscovering the href case from the issue, interior question marks, runs of question marks, the early>termination, empty data, and the unterminated case. All three fail without the change.cargo test --all,cargo fmt --all -- --checkandcargo clippy --all-features --all-targetsare clean.