fix(search): honour fences and folded whitespace in Note highlighting

Two review findings, both real.

The intraword-underscore cleanup guarded code with a pattern that recognised
only the shortest delimiter forms — a bare ``` pair and a single-backtick span.
A ````-fenced block, a tilde fence, or a ``multi-backtick`` span ended the
region early and handed the rest of the author's code to the rewrite, turning
`a\_b` into `a_b` inside their code sample. Fenced blocks are now walked a line
at a time, tracking the opening delimiter exactly the way stripEmptyListItemLines
already does (three or more, closed only by a run at least as long), and the
inline branch matches a backtick RUN closed by one of equal length.

The note scanner claimed to be the same scan as the indexer's `findTextRanges`
but did not fold whitespace, which staging added since this branch was written.
The indexer folds every `\s` to a space, so a phrase matches across a soft line
break — which `remark-breaks` renders as a `<br>`, splitting the phrase over two
text nodes that a per-node scan could never see. The hit counted in the panel
and highlighted nowhere, the exact bug this branch exists to fix.

The plugin now scans runs of continuously-readable text rather than single
nodes, so a match spanning an inline boundary (a soft break, a bold word) is
wrapped as several marks sharing one ordinal. Runs end at any non-inline
element, so two paragraphs are never joined into a phrase the reader cannot see.

`foldSearchWhitespace` moved to `@sim/utils/string`: the canvas card renders
from a package, which cannot import from `apps/*`, and two copies of that rule
silently disagreeing is precisely what produced the second finding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: icecrasher321

apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/markdown-fidelity.ts

@@ -9,50 +9,70 @@ const FRONTMATTER_REGEX = /^---\r?\n(?:[\s\S]*?\r?\n)?---[ \t]*(?:\r?\n)*/
 const ESCAPED_CALLOUT_REGEX = /^(\s*>(?:\s*>)*\s*)\\\[!([A-Za-z]+)\\\]/gm
 
 /**
- * A code region \u2014 fenced block or inline span. Never rewritten by the cleanups below, and always
- * the FIRST branch of the alternations that use it so a candidate sitting inside code is consumed
- * as code and left verbatim.
+ * Alternates a code region (fenced block or inline span \u2014 never rewritten) with an inline link whose
+ * destination has no title and isn't angle-bracketed. The code branch is listed first so a link inside
+ * code is consumed as code and left untouched. The destination stops at `)` / whitespace, so a link
+ * carrying a title (`[x](url "t")`) never matches and is preserved verbatim.
  */
-const CODE_REGION_SOURCE = '(```[\\s\\S]*?```|~~~[\\s\\S]*?~~~|`[^`\\n]+`)'
-
-/**
- * Alternates a code region with an inline link whose destination has no title and isn't
- * angle-bracketed. The destination stops at `)` / whitespace, so a link carrying a title
- * (`[x](url "t")`) never matches and is preserved verbatim.
- */
-const CODE_OR_PLAIN_LINK_REGEX = new RegExp(
-  `${CODE_REGION_SOURCE}|\\[([^\\]]+)]\\(([^)\\s<>]+)\\)`,
-  'g'
-)
+const CODE_OR_PLAIN_LINK_REGEX =
+  /(```[\s\S]*?```|~~~[\s\S]*?~~~|`[^`\n]+`)|\[([^\]]+)]\(([^)\s<>]+)\)/g
+const HTTP_URL_REGEX = /^https?:\/\/\S+$/i
 
 /**
- * Alternates a code region with a single underscore that has a letter or digit on both sides.
+ * Alternates an inline code span with a single underscore that has a letter or digit on both sides.
  *
  * CommonMark's intraword rule means such an underscore can neither open nor close emphasis, so the
  * serializer's backslash before it carries no meaning \u2014 it just writes `SB\_ACTION\_ROUTER\_SECRET`
  * into the document. That is ugly in the file, and it silently breaks workflow search, which matches
  * against the stored markdown rather than the rendered text: searching `SB_ACTION` finds nothing in
  * a note whose stored form has a backslash the reader never sees.
  *
- * Code is excluded because the serializer emits it verbatim: a `\_` inside a fence is the author's
- * own backslash, not an escape this may drop.
+ * Code is excluded because the serializer emits it verbatim: a `\_` inside a span is the author's own
+ * backslash, not an escape this may drop. The span branch matches a backtick RUN and requires a run of
+ * the same length to close it, per CommonMark \u2014 a fixed single-backtick pattern would read ``` ``a`b`` ```
+ * as `` `a` `` plus loose text and rewrite the interior. Fenced blocks are handled a line at a time by
+ * {@link unescapeIntrawordUnderscores}, which is the only way to honour a fence of any length.
  *
- * Written with a capture group rather than a lookbehind: lookbehind only landed in Safari 16.4, and
- * an unsupported one throws when the pattern is constructed \u2014 taking the whole editor module with
- * it. The group is consumed and put back, and the lookahead is not, so runs like `A\_B\_C` still
- * match on every pair.
+ * The flanking character is a capture group rather than a lookbehind: lookbehind only landed in
+ * Safari 16.4, and an unsupported one throws when the pattern is constructed \u2014 taking the whole editor
+ * module with it. The group is consumed and put back, and the lookahead is not, so runs like `A\_B\_C`
+ * still match on every pair.
  */
-const CODE_OR_INTRAWORD_ESCAPED_UNDERSCORE = new RegExp(
-  `${CODE_REGION_SOURCE}|([\\p{L}\\p{N}])\\\\_(?=[\\p{L}\\p{N}])`,
-  'gu'
-)
-const HTTP_URL_REGEX = /^https?:\/\/\S+$/i
+const CODE_SPAN_OR_INTRAWORD_ESCAPED_UNDERSCORE =
+  /(`+)((?:[^`]|(?!\1)`)*?)\1(?!`)|([\p{L}\p{N}])\\_(?=[\p{L}\p{N}])/gu
 
-/** Drops the meaningless backslash before an intraword underscore, outside code. */
+/**
+ * Drops the meaningless backslash before an intraword underscore, outside code.
+ *
+ * Fenced blocks are skipped a line at a time, tracking the opening delimiter the same way
+ * {@link stripEmptyListItemLines} does: a fence is three OR MORE backticks or tildes and is closed
+ * only by a run of the same character at least as long, so a `````` ```` `````-fenced block wrapping
+ * ``` ``` ``` stays code throughout. Matching a fixed ``` pair instead would end the region early and
+ * hand the rest of the author's code to the rewrite.
+ */
 function unescapeIntrawordUnderscores(markdown: string): string {
-  return markdown.replace(CODE_OR_INTRAWORD_ESCAPED_UNDERSCORE, (match, code, flank) =>
-    code ? code : `${flank}_`
-  )
+  const lines = markdown.split('\n')
+  let fence: string | null = null
+
+  for (let i = 0; i < lines.length; i++) {
+    const delimiter = lines[i].match(FENCE_DELIMITER)?.[1]
+
+    if (fence) {
+      if (delimiter && delimiter[0] === fence[0] && delimiter.length >= fence.length) fence = null
+      continue
+    }
+    if (delimiter) {
+      fence = delimiter
+      continue
+    }
+
+    lines[i] = lines[i].replace(
+      CODE_SPAN_OR_INTRAWORD_ESCAPED_UNDERSCORE,
+      (match, ticks, _span, flank) => (ticks === undefined ? `${flank}_` : match)
+    )
+  }
+
+  return lines.join('\n')
 }
 
 /**

apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/round-trip.test.ts

@@ -237,6 +237,22 @@ describe('editor markdown round-trip', () => {
     expect(roundTrip('call `a\\_b` here')).toContain('a\\_b')
   })
 
+  /* A fence is three OR MORE delimiters, closed only by a run at least as long, and an inline span
+     opens and closes on backtick runs of equal length. Recognising just the shortest form ends the
+     code region early and hands the rest of the author's code to the rewrite. */
+  it('leaves code alone in a longer fence', () => {
+    expect(roundTrip('````\nx = a\\_b\n```\nstill code y = c\\_d\n````')).toContain('a\\_b')
+    expect(roundTrip('````\nx = a\\_b\n```\nstill code y = c\\_d\n````')).toContain('c\\_d')
+  })
+
+  it('leaves code alone in a tilde fence', () => {
+    expect(roundTrip('~~~~\nx = a\\_b\n~~~~')).toContain('a\\_b')
+  })
+
+  it('leaves code alone in a multi-backtick inline span', () => {
+    expect(roundTrip('call ``a\\_b`c`` here')).toContain('a\\_b')
+  })
+
   it('preserves an image url (does not drop the src)', () => {
     const out = roundTrip('![alt](https://example.com/i.png)')
     expect(out).toContain('![alt](https://example.com/i.png)')

apps/sim/lib/workflows/search-replace/indexer.ts

@@ -1,4 +1,5 @@
 import { isRecordLike } from '@sim/utils/object'
+import { foldSearchWhitespace } from '@sim/utils/string'
 import { DEFAULT_SUBBLOCK_TYPE } from '@sim/workflow-persistence/subblocks'
 import type { SubBlockType } from '@sim/workflow-types/blocks'
 import { isWorkflowBlockProtected } from '@sim/workflow-types/workflow'
@@ -10,7 +11,6 @@ import {
 } from '@/lib/workflows/search-replace/json-value-fields'
 import {
   buildBlockNamesByReferencePrefix,
-  foldSearchWhitespace,
   getResourceKindForSubBlock,
   matchesSearchText,
   parseInlineReferences,

apps/sim/lib/workflows/search-replace/resources/references.ts

@@ -1,3 +1,4 @@
+import { foldSearchWhitespace } from '@sim/utils/string'
 import {
   getWorkflowSearchSubBlockResourceKind,
   parseWorkflowSearchSubBlockResources,
@@ -140,18 +141,6 @@ export function parseStructuredResourceReferences(
   return parseWorkflowSearchSubBlockResources(value, subBlockConfig, selectorContext)
 }
 
-/**
- * Maps every Unicode whitespace character to a plain space, one-to-one.
- * Agent-authored block names and values routinely carry non-breaking or
- * narrow spaces that render identically to " " but never equal a typed
- * space, silently hiding matches. The replacement is length-preserving
- * (every `\s` character is a single UTF-16 unit), so indexes into the
- * folded string remain valid ranges into the original.
- */
-export function foldSearchWhitespace(value: string): string {
-  return value.replace(/\s/g, ' ')
-}
-
 export function matchesSearchText(
   candidate: string,
   query: string | undefined,

apps/sim/lib/workflows/search-replace/resources/resolvers.ts

@@ -1,4 +1,4 @@
-import { foldSearchWhitespace } from '@/lib/workflows/search-replace/resources/references'
+import { foldSearchWhitespace } from '@sim/utils/string'
 import type {
   WorkflowSearchMatch,
   WorkflowSearchMatchKind,

packages/utils/src/string.ts

@@ -162,3 +162,22 @@ export function formatQuotedNameList(names: string[], maxListed: number): string
   const overflow = names.length - maxListed
   return overflow > 0 ? `${listed} and ${overflow} more` : listed
 }
+
+/**
+ * Maps every Unicode whitespace character to a plain space, one-to-one.
+ *
+ * Agent-authored block names and values routinely carry non-breaking or narrow
+ * spaces that render identically to " " but never equal a typed space, silently
+ * hiding matches. The replacement is length-preserving (every `\s` character is
+ * a single UTF-16 unit), so indexes into the folded string remain valid ranges
+ * into the original.
+ *
+ * Lives here rather than beside the workflow search index because the Note card
+ * on the canvas has to fold identically to find the same occurrences, and it
+ * renders from `@sim/workflow-renderer` — a package, which cannot import from
+ * `apps/*`. Two copies of this rule silently disagreeing is precisely the bug
+ * that made a match count in the panel and highlight nowhere on the card.
+ */
+export function foldSearchWhitespace(value: string): string {
+  return value.replace(/\s/g, ' ')
+}

packages/workflow-renderer/src/note/note-search-highlight.test.tsx

@@ -13,7 +13,7 @@
  */
 
 import { act } from 'react'
-import type { Root } from 'hast'
+import type { Element, ElementContent, Root, RootContent } from 'hast'
 import { createRoot, type Root as ReactRoot } from 'react-dom/client'
 import { afterEach, beforeAll, describe, expect, it } from 'vitest'
 import {
@@ -204,6 +204,117 @@ describe('note search rehype plugin', () => {
   })
 })
 
+/*
+ * The indexer folds every `\s` to a space before matching, so a phrase can match
+ * across a soft line break — which `remark-breaks` renders as a `<br>` splitting
+ * the phrase over two text nodes. A per-node scan saw neither half, leaving the
+ * hit counted in the panel and highlighted nowhere on the card.
+ */
+describe('note search across inline boundaries', () => {
+  function markedTextsOf(tree: Root): string[] {
+    const texts: string[] = []
+    const walk = (node: Root | Element) => {
+      /* `Root['children']` and `Element['children']` are different unions, so iterating the
+         parameter directly widens each child to their intersection and drops narrowing. */
+      const children: Array<RootContent | ElementContent> = node.children
+      for (const child of children) {
+        if (child.type !== 'element') continue
+        if (child.tagName === 'mark') {
+          const [first] = child.children
+          texts.push(first?.type === 'text' ? first.value : '')
+          continue
+        }
+        walk(child)
+      }
+    }
+    walk(tree)
+    return texts
+  }
+
+  function paragraphWithBreak(before: string, after: string): Root {
+    return {
+      type: 'root',
+      children: [
+        {
+          type: 'element',
+          tagName: 'p',
+          properties: {},
+          children: [
+            { type: 'text', value: before },
+            { type: 'element', tagName: 'br', properties: {}, children: [] },
+            { type: 'text', value: after },
+          ],
+        },
+      ],
+    }
+  }
+
+  it('marks a phrase spanning a soft line break', () => {
+    const tree = paragraphWithBreak('the quick', 'brown fox')
+    noteSearchHighlightPlugin({ query: 'quick brown' })(tree)
+    expect(markedTextsOf(tree)).toEqual(['quick', 'brown'])
+  })
+
+  it('gives both halves of one hit the same ordinal', () => {
+    const tree = paragraphWithBreak('the quick', 'brown fox')
+    noteSearchHighlightPlugin({ query: 'quick brown' })(tree)
+
+    const ordinals: unknown[] = []
+    const walk = (node: Root | Element) => {
+      /* `Root['children']` and `Element['children']` are different unions, so iterating the
+         parameter directly widens each child to their intersection and drops narrowing. */
+      const children: Array<RootContent | ElementContent> = node.children
+      for (const child of children) {
+        if (child.type !== 'element') continue
+        if (child.tagName === 'mark') ordinals.push(child.properties.dataNoteSearchIndex)
+        else walk(child)
+      }
+    }
+    walk(tree)
+    expect(ordinals).toEqual(['0', '0'])
+  })
+
+  it('marks a phrase spanning a bold word', () => {
+    const tree: Root = {
+      type: 'root',
+      children: [
+        {
+          type: 'element',
+          tagName: 'p',
+          properties: {},
+          children: [
+            { type: 'text', value: 'a ' },
+            {
+              type: 'element',
+              tagName: 'strong',
+              properties: {},
+              children: [{ type: 'text', value: 'bold' }],
+            },
+            { type: 'text', value: ' word' },
+          ],
+        },
+      ],
+    }
+    noteSearchHighlightPlugin({ query: 'a bold word' })(tree)
+    expect(markedTextsOf(tree)).toEqual(['a ', 'bold', ' word'])
+  })
+
+  /* Two paragraphs are not one phrase on screen. Joining them would invent a hit
+     the reader cannot see — and one the indexer never counted, since the source
+     carries a blank line there, not a single space. */
+  it('does not join text across a block boundary', () => {
+    const tree = paragraphTree('the quick', 'brown fox')
+    noteSearchHighlightPlugin({ query: 'quick brown' })(tree)
+    expect(markedTextsOf(tree)).toEqual([])
+  })
+
+  it('folds a non-breaking space the way the indexer does', () => {
+    const tree = paragraphTree('a b')
+    noteSearchHighlightPlugin({ query: 'a b' })(tree)
+    expect(markedTextsOf(tree)).toEqual(['a b'])
+  })
+})
+
 describe('note search highlight rendering', () => {
   it('marks every occurrence in the read view', () => {
     const container = renderNote('first SB_SECRET line\n\nsecond SB_SECRET line', {