fix(CommandPalette): keep astral characters intact when truncating search results - #6817
fix(CommandPalette): keep astral characters intact when truncating search results#6817arsalan507 wants to merge 1 commit into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughHTML truncation now iterates over Unicode code points instead of UTF-16 code units. Highlight truncation budgets also use code-point counts. New tests verify that emoji and other astral characters remain intact, long prefixes receive an ellipsis, and highlighted matches remain present. Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
src/runtime/utils/search.tsParsing error: Unexpected token { test/utils/search.spec.tsParsing error: Unexpected token : Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
commit: |
…arch results `truncateHTMLFromStart` walked the highlighted snippet one UTF-16 code unit at a time. An astral character (emoji, most CJK extension blocks) occupies two code units, so a truncation boundary landing between them sliced the pair in half and emitted an unpaired surrogate, rendering as `<?>`. Iterate by code point instead, and measure the length budget the same way so both sides stay in the same units. Behaviour for BMP-only content is unchanged.
16016db to
6e97151
Compare
…arch results (#365) `truncateHTMLFromStart` walked the string one UTF-16 code unit at a time. An astral character — emoji, most CJK extension blocks — occupies two, so a truncation boundary landing between them sliced the pair in half and emitted an unpaired surrogate that renders as `�`. Deterministic from 7 characters of prefix onward, and every length after. The fix iterates code points and measures the caller's budget in the same units, so the two halves cannot drift apart. Submitted upstream first as nuxt/ui#6817, so `src/runtime/utils/search.ts` stays in sync. Test hardening on top: the original fixture used a single astral character sitting inside both surrogate ranges, so an implementation with an off-by-one range bound passed. The sweep now covers the range edges — U+10000, U+10FFFF, U+20000 (low surrogate U+DC00), U+1F3FF (low surrogate U+DFFF, a skin-tone modifier common in ordinary text) and U+1F600 — and counts surviving characters rather than only scanning for lone surrogates, which an implementation that deletes every astral character satisfied trivially. A new case places astral content after the match, covering the caller's half of the fix; reverting it alone previously left every test green. Behaviour change worth noting in the changelog: an astral character now costs one unit of the truncation budget instead of two, so 20 emoji before a 5-character match keep 13 where main kept 6 and half of a 7th. BMP-only content is byte-identical. Scope: #339 is scoped entirely to `truncateHTMLFromStart` and closes here. #362 (`<mark>` inserted inside an astral character, because Fuse's `indices` are code-unit offsets too) and #364 (grapheme clusters) are separate defects. Closes #339 Co-Authored-By: Arsalan Ahmed <arsalanahmed507@gmail.com>
🔗 Linked issue
None open — filing this directly as it's a small, self-contained bug.
❓ Type of change
📚 Description
truncateHTMLFromStartinsrc/runtime/utils/search.tswalks the highlighted snippet in reverse one UTF-16 code unit at a time (html[i]). An astral character — emoji, most CJK extension blocks, many symbols — occupies two code units. When the truncation boundary lands between them, the surrogate pair is sliced in half and the output carries an unpaired surrogate, which renders as�.This is user-visible in
CommandPaletteanduseContentSearch, where indexed content is more likely to contain emoji than a hand-written command label: a search result whose match sits after an emoji gets a�at the truncation point.Reproduction — sweeping filler lengths 1–40 of
😀before the match, lone surrogates appear from filler length 7 onward and for every length after:Not a security issue. A lone surrogate can only render as a broken glyph or U+FFFD — it cannot manufacture HTML syntax, and no raw
<,>,&,"or'appears outside the<mark>tags the highlighter inserts itself. This is a correctness and appearance bug.🔧 The fix
Two hunks:
truncateHTMLFromStart— iterate withArray.from(html)so a surrogate pair is never split. TheinsideTaglogic is untouched, since<and>are always single code units.Worth calling out explicitly for review: hunk 2 is a consistency fix, not required to remove the lone surrogates — hunk 1 alone is sufficient. It does mean an emoji now counts as one character against the truncation budget rather than two, so emoji-containing snippets truncate slightly later than before. That seems closer to the intent of a visible-length budget, but happy to drop hunk 2 if you'd rather keep the diff minimal.
Behaviour for BMP-only content is byte-identical, which is why all pre-existing tests pass untouched.
✅ Verification
Added three tests to
test/utils/search.spec.ts:never splits an astral character, at any truncation boundary— the 1–40 sweep, asserting no lone surrogate at any lengthkeeps emoji before the match intactstill truncates a long prefix down to an ellipsis— guards against over-correcting into "never truncate"Verified the first two fail against unmodified
search.ts(expected [ 7, 8, 9, … ] to deeply equal []) and pass with the fix. The third passes both before and after by design.📝 Checklist