Fix RoyalRoad parser for the site's Tailwind redesign#2786
Fix RoyalRoad parser for the site's Tailwind redesign#2786possibletoactual wants to merge 1 commit into
Conversation
Royal Road's redesign broke the RoyalRoadParser: chapters silently dropped to one, and metadata/content extraction failed. This repairs it against the new markup while keeping the legacy selectors as fallbacks: - Chapter list: parse the `window.chapters` JSON array (present in every variant) instead of the removed `table#chapters`; fall back to the table. - Chapter content: anchor on `div.chapter-inner` (the `portlet-body` wrapper is gone) and null-guard so one odd chapter can't abort the whole book. - Chapter titles: use the authoritative title from the chapter list rather than the first `<h1>` (now an unrelated "Give Reputation to User" widget). - Author: header `/profile/` link, then `books:author` meta, then legacy. - Cover: `og:image` / `twitter:image`, then legacy `img.thumbnail`. - Description/genres/information page: scrape the full description from the raw page (og:description is truncated; JSON-LD is stripped from the served DOM) and read genres from `a[href*='tagsAdd=']`; rebuild the info page with title, author, genres, and full description. Unit tests unaffected (they cover other methods); eslint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| // Capture book metadata (description + genres) from the RAW html now — the DOM | ||
| // the parser later sees has its <script> tags stripped and, on the redesign, no | ||
| // JSON-LD at all, so the full description must be scraped here. | ||
| RoyalRoadParser.lastBookMeta = RoyalRoadParser.extractBookMeta(tocHtml); |
There was a problem hiding this comment.
Don't store book meta in a static class prop, I'm pretty sure this will cause a bug if you queue 2 RR books at once.
Instance prop this.lastBookMeta or could attach it to the DOM i.e. dom.__royalRoadMeta
There was a problem hiding this comment.
I'm not sure how you'd queue 2 RR books at once? If you open two copies of WebToEpub, they're two instances. With their own statics.
There was a problem hiding this comment.
Ah, yeah you're right, each popup is a separate document.
I still think this is an anti-pattern though, because every method that writes or reads lastBookMeta already has this. So it's the wrong lifetime.. metadata is per-parse, per-book. A static is per realm (popup).
| // Extract the chapter list from the page's `window.chapters = [...]` JSON blob. | ||
| // Returns [] if not present/parseable, so the caller can fall back. | ||
| static chaptersFromWindowChapters(html, baseUrl) { | ||
| let match = html.match(/window\.chapters\s*=\s*(\[[\s\S]*?\])\s*;/); |
There was a problem hiding this comment.
[\s\S]*? is lazy.
what will happen if chapter is named Chapter 1 [Interlude]?
There was a problem hiding this comment.
You should probably have a look at util.locateAndExtractJson()
| // Prefer JSON-LD when present; otherwise locate the full description in the body by | ||
| // matching the og:description as a needle (the redesign's classes are unstable). | ||
| static extractBookMeta(html) { | ||
| let doc = new DOMParser().parseFromString(html, "text/html"); |
There was a problem hiding this comment.
since you parsed it here anyway, you can pass it to getChapterUrls for the fallback extraction, to avoid parasing the DOM twice
| } | ||
| let best = null; | ||
| let bestLen = 0; | ||
| for (let el of doc.querySelectorAll("div, section, article")) { |
There was a problem hiding this comment.
Can you narrow down the search? Iterating over every div, section, article is kinda meh, if there's some heuristic that can be used it would be nice
| // formatting (sanitized downstream by populateInfoDiv). | ||
| nodes.push(heading); | ||
| let descDiv = dom.createElement("div"); | ||
| descDiv.innerHTML = descriptionHtml; |
There was a problem hiding this comment.
No NOT assign un-sanitized HTML to innerHTML. It makes approval for Mozilla and Google stores difficult.
Better yet, do not assign to innerHTML at all.
| ?.getAttribute("content") ?? ""; | ||
| let el = RoyalRoadParser.findDescriptionElement(doc, og); | ||
| if (el != null) { | ||
| descriptionHtml = el.innerHTML; |
| if ((jsonld != null) && !util.isNullOrEmpty(jsonld.description)) { | ||
| descriptionHtml = jsonld.description; | ||
| let holder = doc.createElement("div"); | ||
| holder.innerHTML = jsonld.description; |
3 participants
Royal Road's site redesign broke
RoyalRoadParser: on the new markup the chapter list silently collapses to a single chapter, and author/cover/description/genres extraction fails. This restores the parser against the new markup while keeping the old selectors as fallbacks.What changed
window.chaptersJSON array (present in every page variant) instead of the removedtable#chapters; fall back to the legacy table.div.chapter-inner(theportlet-bodywrapper is gone) and null-guard so a single odd chapter can't throw and abort the whole download.<h1>(now an unrelated "Give Reputation to User" widget)./profile/link →books:authormeta → legacy selector.og:image/twitter:image→ legacyimg.thumbnail.og:descriptionis truncated, so the full description is scraped from the raw page during analyse; genres read froma[href*='tagsAdd=']. The information page is rebuilt with title, author, genres, and full description.PR checklist
UtestRoyalRoadParsercoversremoveOlderChapterNavJunk/makeHiddenElementsVisible, which are unchanged.package.jsonandreadme.md.ExperimentalTabMode.Legacy selectors are preserved as fallbacks, so older/cached page variants continue to work.