Skip to content

Generate structured llms.txt with progressive disclosure#4774

Open
lennessyy wants to merge 3 commits into
mainfrom
reduce-llms-txt
Open

Generate structured llms.txt with progressive disclosure#4774
lennessyy wants to merge 3 commits into
mainfrom
reduce-llms-txt

Conversation

@lennessyy

@lennessyy lennessyy commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds llms.txt generation to the markdown-pages plugin, replacing docusaurus-plugin-llms for llms.txt (it still generates llms-full.txt)
  • Root llms.txt uses progressive disclosure: core primitives inline, everything else links to section-level llms.txt files
  • Core Primitives subsections auto-discovered from encyclopedia/ directory structure (titles from sidebar_label frontmatter)
  • SDK sections auto-discovered from develop/ subdirectories
  • Adds excludePaths config to omit deprecated content (tctl-v1) from llms.txt
  • Fixes undefined description bug in section llms.txt files

Known issues to address in follow-ups

  • Concepts section nearly empty: temporal.mdx lives under encyclopedia/ so it gets pulled into Core Primitives instead of Concepts
  • Duplicate encyclopedia section: Core Primitives inlines encyclopedia content, but Sections also links to encyclopedia/llms.txt
  • "Other" label in Core Primitives: encyclopedia pages not in a subdirectory (retry policies, interceptors, failures) get an unhelpful "Other" heading
  • Bottom "Other" section large: self-hosted-guide/* and evaluate/* could be their own sections
  • Cross-SDK develop pages uncategorized: worker performance, safe deployments, etc. fall into Other

Test plan

  • Build site and verify build/llms.txt structure
  • Verify section llms.txt files exist (e.g., build/cloud/llms.txt, build/develop/go/llms.txt)
  • Verify no > undefined in any section llms.txt
  • Add a new file under docs/encyclopedia/<subdir>/ and confirm it auto-appears in the right subsection

┆Attachments: EDU-6600 Generate structured llms.txt with progressive disclosure

- Core Primitives subsections now auto-discovered from encyclopedia/ directory structure
- Subsection titles pulled from sidebar_label frontmatter
- Add excludePaths option to filter pages from llms.txt (removes tctl-v1)
- Rewrite llms.txt intro to describe the file's purpose
- Add SDK section description explaining cross-SDK structure
@lennessyy lennessyy requested a review from a team as a code owner June 26, 2026 00:10
@vercel

vercel Bot commented Jun 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
temporal-documentation Ready Ready Preview, Comment Jun 26, 2026 12:13am

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

📖 Docs PR preview links

This PR does not change any pages in /docs. If you make updates, links to the modified pages will appear here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant