Fix #304: Use HTML test document instead of sitemap XML in SubDocumentsFilterTest

[AsifQureshi04]

Summary

Fixes #304.

The SubDocumentsFilterTest was using a sitemap XML file (stormcrawler.sitemap.xml) as test input for JSoupParserBolt, which is an HTML parser. This only worked because MIME type detection was disabled — a fragile workaround.

Changes:

• Created a minimal HTML test document (subdocuments.html) with 6 anchor links
• Rewrote SubDocumentsParseFilter to extract links from the W3C DOM (DocumentFragment) instead of re-parsing raw bytes as XML with XPath
• Removed the detect.mimetype = false hack from the test

Test plan

• SubDocumentsFilterTest passes (7 emitted tuples: 6 subdocuments + 1 parent)
• All other parse filter tests pass (XPathFilterTest, CollectionTaggerTest, CSVMetadataFilterTest)
• Code formatted with mvn git-code-format:format-code

[jnioche]

ParseFilters are not specific to JSoupParseFilter: it is used by all Parsers, including SitemapParserBolt, which is why we had this test.

[AsifQureshi04]

Thanks for the context @jnioche . I understand that ParseFilters are used across all parsers, including SiteMapParserBolt.

I followed the approach you suggested in the issue — using a minimalistic HTML document instead. The rewritten SubDocumentsParseFilter now works with the W3C DOM (DocumentFragment) provided by JSoupParserBolt, rather than re-parsing the raw bytes as XML.

Would you prefer I also add a separate test that demonstrates SubDocumentsParseFilter working with SiteMapParserBolt? Happy to adjust based on your feedback.

[jnioche]

Thanks for the context @jnioche . I understand that ParseFilters are used across all parsers, including SiteMapParserBolt.

I followed the approach you suggested in the issue — using a minimalistic HTML document instead. The rewritten SubDocumentsParseFilter now works with the W3C DOM (DocumentFragment) provided by JSoupParserBolt, rather than re-parsing the raw bytes as XML.

Sorry @AsifQureshi04, I opened the issue 10 years ago (!?!) and needed a bit of time to refresh my memory. Your PR is fine as it is.

Would you prefer I also add a separate test that demonstrates SubDocumentsParseFilter working with SiteMapParserBolt? Happy to adjust based on your feedback.

That's what I was wondering about but actually it is probably not needed and would mean that the document would be reparsed as SitemapParserBolt does not generate a DOM.

Thanks for your contribution to StormCrawler