Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ Available themes: Aurora X, Dracula, GitHub Dark, GitHub Light, Monokai, Nord, O

## Languages

`apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsx`, `vue`, and `yaml`.
`apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`.

Each language is available from `@tanstack/highlight/languages/<name>`. The aggregate `@tanstack/highlight/languages` entry can tree-shake, while direct subpaths make isolation explicit.

Expand Down
3 changes: 2 additions & 1 deletion docs/guides/octane.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,11 @@ import { createHighlighter } from '@tanstack/highlight/core'
import { css } from '@tanstack/highlight/languages/css'
import { js } from '@tanstack/highlight/languages/js'
import { ts } from '@tanstack/highlight/languages/ts'
import { tsrx } from '@tanstack/highlight/languages/tsrx'
import { tsx } from '@tanstack/highlight/languages/tsx'

export const highlighter = createHighlighter({
languages: [css, js, ts, tsx],
languages: [css, js, ts, tsrx, tsx],
})
```

Expand Down
1 change: 1 addition & 0 deletions docs/language-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Every language is an isolated definition imported from `@tanstack/highlight/lang
| Svelte | `svelte` | - | Markup plus optional script/style and expression delegation |
| TOML | `toml` | - | Strings, comments, tables, properties |
| TypeScript | `ts` | `typescript`, `angular-ts` | JavaScript scanner plus TypeScript keywords/types |
| TSRX | `tsrx` | `octane` | TypeScript, contextual JSX, Octane component shorthand and template directives |
| TSX | `tsx` | - | TypeScript, contextual JSX, generic disambiguation |
| Vue | `vue` | - | Template markup plus optional script/style delegation |
| YAML | `yaml` | `yml` | Comment boundaries and block scalars |
Expand Down
3 changes: 2 additions & 1 deletion docs/reference/languages.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,11 @@ const highlighter = createHighlighter({
| `svelte` | `@tanstack/highlight/languages/svelte` | None |
| `toml` | `@tanstack/highlight/languages/toml` | None |
| `ts` | `@tanstack/highlight/languages/ts` | `typescript`, `angular-ts` |
| `tsrx` | `@tanstack/highlight/languages/tsrx` | `octane` |
| `tsx` | `@tanstack/highlight/languages/tsx` | None |
| `vue` | `@tanstack/highlight/languages/vue` | None |
| `yaml` | `@tanstack/highlight/languages/yaml` | `yml` |

`@tanstack/highlight/languages` re-exports `apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsx`, `vue`, and `yaml`. The barrel is convenient but individual subpaths make bundle intent explicit.
`@tanstack/highlight/languages` re-exports `apache`, `css`, `diff`, `dockerfile`, `ejs`, `env`, `html`, `http`, `js`, `json`, `jsx`, `markdown`, `mermaid`, `nginx`, `plaintext`, `python`, `scheme`, `shell`, `sql`, `svelte`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`. The barrel is convenient but individual subpaths make bundle intent explicit.

See the [language support matrix](../language-support) for the context-aware behavior and current scope of each registration.
2 changes: 2 additions & 0 deletions scripts/language-utils.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ export const supportedLanguages = [
'svelte',
'toml',
'ts',
'tsrx',
'tsx',
'vue',
'yaml',
Expand All @@ -46,6 +47,7 @@ const aliases = {
markdown: 'markdown',
md: 'markdown',
mjs: 'js',
octane: 'tsrx',
sh: 'shell',
shell: 'shell',
text: 'plaintext',
Expand Down
2 changes: 1 addition & 1 deletion scripts/test-package.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ if (typeof modularCore.createHighlighter !== 'function') {
throw new Error('The core entry does not export createHighlighter()')
}

if (!languages.tsx || !languages.shell) {
if (!languages.tsrx || !languages.tsx || !languages.shell) {
throw new Error('The languages entry does not export language definitions')
}

Expand Down
2 changes: 1 addition & 1 deletion skills/_artifacts/domain_map.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ skills:
- 'Choose direct language imports and intentional fallback behavior'
reference_candidates:
- topic: 'shipped language names, aliases, and isolated imports'
reason: 'The package exposes 25 independently importable language definitions with aliases.'
reason: 'The package exposes 26 independently importable language definitions with aliases.'
failure_modes:
- mistake: 'Importing the root entry in size-sensitive clients'
mechanism: 'The root entry constructs the all-language registry, so a bound helper retains every shipped language.'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Import only the definitions the application registers.
| Svelte | `svelte` | `@tanstack/highlight/languages/svelte` | - |
| TOML | `toml` | `@tanstack/highlight/languages/toml` | - |
| TypeScript | `ts` | `@tanstack/highlight/languages/ts` | `typescript`, `angular-ts` |
| TSRX | `tsrx` | `@tanstack/highlight/languages/tsrx` | `octane` |
| TSX | `tsx` | `@tanstack/highlight/languages/tsx` | - |
| Vue | `vue` | `@tanstack/highlight/languages/vue` | - |
| YAML | `yaml` | `@tanstack/highlight/languages/yaml` | `yml` |
Expand Down
3 changes: 3 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import { sql } from './languages/sql.js'
import { svelte } from './languages/svelte.js'
import { toml } from './languages/toml.js'
import { ts } from './languages/ts.js'
import { tsrx } from './languages/tsrx.js'
import { tsx } from './languages/tsx.js'
import { vue } from './languages/vue.js'
import { yaml } from './languages/yaml.js'
Expand Down Expand Up @@ -54,6 +55,7 @@ export type HighlightLanguage =
| 'svelte'
| 'toml'
| 'ts'
| 'tsrx'
| 'tsx'
| 'vue'
| 'yaml'
Expand Down Expand Up @@ -121,6 +123,7 @@ export const allLanguages = [
svelte,
toml,
ts,
tsrx,
tsx,
vue,
yaml,
Expand Down
49 changes: 39 additions & 10 deletions src/internal/script.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,21 @@ const semanticPatterns = [
},
] as const

export function collectScriptRanges(code: string, jsx = false) {
const lexical = collectScriptLexicalRanges(code, jsx)
const markup = jsx ? collectJsxRanges(code) : []
export function collectScriptRanges(
code: string,
jsx = false,
jsxAtLineStart = false,
) {
const lexical = collectScriptLexicalRanges(code, jsx, jsxAtLineStart)
const markup = jsx ? collectJsxRanges(code, jsxAtLineStart) : []
return collectPatternRanges(code, semanticPatterns, [...lexical, ...markup])
}

function collectScriptLexicalRanges(code: string, jsx: boolean) {
function collectScriptLexicalRanges(
code: string,
jsx: boolean,
jsxAtLineStart: boolean,
) {
const ranges: Array<TokenRange> = []
let index = 0

Expand Down Expand Up @@ -78,7 +86,7 @@ function collectScriptLexicalRanges(code: string, jsx: boolean) {
}

if (character === '`') {
index = collectTemplateRanges(code, index, ranges, jsx)
index = collectTemplateRanges(code, index, ranges, jsx, jsxAtLineStart)
continue
}

Expand Down Expand Up @@ -106,6 +114,7 @@ function collectTemplateRanges(
start: number,
ranges: Array<TokenRange>,
jsx: boolean,
jsxAtLineStart: boolean,
) {
let segmentStart = start
let index = start + 1
Expand All @@ -130,7 +139,12 @@ function collectTemplateRanges(
const end = findInterpolationEnd(code, index + 2)
const expressionEnd = end < 0 ? code.length : end
const expression = code.slice(index + 2, expressionEnd)
ranges.push(...offsetRanges(collectScriptRanges(expression, jsx), index + 2))
ranges.push(
...offsetRanges(
collectScriptRanges(expression, jsx, jsxAtLineStart),
index + 2,
),
)

if (end < 0) return code.length
ranges.push({ start: end, end: end + 1, className: 'operator' })
Expand Down Expand Up @@ -194,7 +208,7 @@ function skipTemplate(code: string, start: number) {
return code.length
}

function collectJsxRanges(code: string) {
function collectJsxRanges(code: string, jsxAtLineStart: boolean) {
const ranges: Array<TokenRange> = []
let index = 0
let jsxDepth = 0
Expand Down Expand Up @@ -229,7 +243,10 @@ function collectJsxRanges(code: string) {
index += 3
continue
}
if (code.startsWith('<>', index) && (jsxDepth > 0 || isJsxStart(code, index))) {
if (
code.startsWith('<>', index) &&
(jsxDepth > 0 || isJsxStart(code, index, jsxAtLineStart))
) {
jsxDepth++
index += 2
continue
Expand All @@ -239,7 +256,12 @@ function collectJsxRanges(code: string) {
const nameStart = index + (closing ? 2 : 1)
const nameMatch = /^[A-Za-z][\w:.-]*/.exec(code.slice(nameStart))
const inJsxText = jsxDepth > 0 && expressionDepth === 0
if (!nameMatch || (!closing && !inJsxText && !isJsxStart(code, index))) {
if (
!nameMatch ||
(!closing &&
!inJsxText &&
!isJsxStart(code, index, jsxAtLineStart))
) {
index++
continue
}
Expand Down Expand Up @@ -286,7 +308,14 @@ function collectJsxAttributes(
}
}

function isJsxStart(code: string, index: number) {
function isJsxStart(code: string, index: number, jsxAtLineStart: boolean) {
if (
jsxAtLineStart &&
/^\s*$/.test(code.slice(code.lastIndexOf('\n', index - 1) + 1, index))
) {
return true
}

const before = code.slice(0, index).trimEnd()
if (!before) return true
if (/\b(?:return|case|yield)$/.test(before)) return true
Expand Down
1 change: 1 addition & 0 deletions src/languages/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ export { sql } from './sql.js'
export { svelte } from './svelte.js'
export { toml } from './toml.js'
export { ts } from './ts.js'
export { tsrx } from './tsrx.js'
export { tsx } from './tsx.js'
export { vue } from './vue.js'
export { yaml } from './yaml.js'
49 changes: 49 additions & 0 deletions src/languages/tsrx.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import { defineLanguage, type TokenRange } from '../core.js'
import { collectScriptRanges } from '../internal/script.js'

const templateDirectives = new Set([
'@catch',
'@case',
'@default',
'@else',
'@empty',
'@for',
'@if',
'@pending',
'@switch',
'@try',
])

export const tsrx = defineLanguage({
name: 'tsrx',
aliases: ['octane'],
tokenize(code) {
const ranges: Array<TokenRange> = collectScriptRanges(code, true, true).map(
(range): TokenRange =>
range.className === 'function' &&
templateDirectives.has(code.slice(range.start, range.end))
? { ...range, className: 'keyword' }
: range,
)
const shorthand = /@\{/g
let match: RegExpExecArray | null

while ((match = shorthand.exec(code))) {
const range: TokenRange = {
start: match.index,
end: match.index + match[0].length,
className: 'keyword',
}

if (!ranges.some((candidate) => overlaps(candidate, range))) {
ranges.push(range)
}
}

return ranges
},
})

function overlaps(left: TokenRange, right: TokenRange) {
return left.start < right.end && right.start < left.end
}
21 changes: 21 additions & 0 deletions test/fixtures.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,27 @@ export type LanguageFixture = {
}

export const languageFixtures: Array<LanguageFixture> = [
{
lang: 'octane',
normalized: 'tsrx',
code: `import { useState } from 'octane'

export function Counter() @{
const [count, setCount] = useState(0)

<button onClick={() => setCount(count + 1)}>
{'Count: ' + count}
</button>
}`,
expectedClasses: [
'th-keyword',
'th-string',
'th-function',
'th-tag',
'th-attr',
'th-number',
],
},
{
lang: 'tsx',
normalized: 'tsx',
Expand Down
9 changes: 8 additions & 1 deletion test/generated/tanstack-doc-fixtures.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"perLanguageLimit": 20,
"maxCodeLength": 3000,
"count": 333,
"count": 334,
"fixtures": [
{
"file": "router/docs/router/how-to/deploy-to-production.md",
Expand Down Expand Up @@ -1914,6 +1914,13 @@
"lang": "ts",
"code": "import { chat } from \"@tanstack/ai\";\nimport { geminiText } from \"@tanstack/ai-gemini\";\n\nconst stream = chat({\n adapter: geminiText(\"gemini-2.5-pro\"),\n messages: [{ role: \"user\", content: \"Hello!\" }],\n});"
},
{
"file": "markdown/docs/core-concepts/rendering.md",
"line": 21,
"rawLang": "tsrx",
"lang": "tsrx",
"code": "import { Markdown } from '@tanstack/markdown/octane'\n\n<Markdown>{source}</Markdown>"
},
{
"file": "ai/docs/getting-started/devtools.md",
"line": 36,
Expand Down
9 changes: 9 additions & 0 deletions test/modular.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest'
import { createHighlighter } from '../src/core'
import { html } from '../src/languages/html'
import { js } from '../src/languages/js'
import { tsrx } from '../src/languages/tsrx'
import { tsx } from '../src/languages/tsx'

describe('language registration', () => {
Expand All @@ -24,6 +25,14 @@ describe('language registration', () => {
expect(markupOnly.highlight(code, { lang: 'html' }).html).not.toContain('th-keyword')
expect(withJavaScript.highlight(code, { lang: 'html' }).html).toContain('th-keyword')
})

it('registers the Octane alias with TSRX only', () => {
const highlighter = createHighlighter({ languages: [tsrx] })

expect(highlighter.listLanguages()).toEqual(['tsrx'])
expect(highlighter.normalizeLanguage('octane')).toBe('tsrx')
expect(highlighter.normalizeLanguage('tsx')).toBe('plaintext')
})
})

describe('decorations', () => {
Expand Down
Loading
Loading