Skip to content

docs(site): keep llms.txt synchronized with live benchmark data - #281

Open
ramanbansal1 wants to merge 6 commits into
TIGER-AI-Lab:mainfrom
ramanbansal1:docs/264-docs-site
Open

docs(site): keep llms.txt synchronized with live benchmark data#281
ramanbansal1 wants to merge 6 commits into
TIGER-AI-Lab:mainfrom
ramanbansal1:docs/264-docs-site

Conversation

@ramanbansal1

Copy link
Copy Markdown
Contributor

What does this PR do?

Refactors llms.txt generation in src/clawbench/utils/update_llms.py to dynamically fetch live metadata, leaderboard data, canonical sitemap links, and CITATION.cff records. Adds unit and regression tests in tests/test_update_llms.py to ensure llms.txt stays strictly synchronized with live site data.

Resolves issues identified in #264:

  • Corpus & Platform Metadata: Updated quick facts to reflect maintained V1 (153 tasks / 144 live websites) and V2 (130 tasks) counts.
  • Dynamic Leaderboard Data: get_leaderboard_section() fetches live leaderboard data from https://claw-bench.com/api/leaderboard.json, generating V1 Hermes and V2 Hermes top 5 tables with pass rate, interception rate, and pass/total metrics.
  • Canonical Routes from Sitemap: fetch_sitemap_urls() parses https://claw-bench.com/sitemap.xml to replace old hash routes with canonical URLs (/leaderboard, /tasks, /traces, /difficulty, etc.).
  • Dataset Links: Added explicit canonical resource links for main task dataset (NAIL-Group/ClawBench), V1 execution traces (NAIL-Group/ClawBenchV1Trace), and V2 execution traces (TIGER-Lab/ClawBenchV2Trace).
  • Citation Metadata: parse_citation_bibtex() parses CITATION.cff to automatically format BibTeX citations without manual drift.
  • Regression Test: Added test_llms_txt_regression_check in tests/test_update_llms.py that fails if llms.txt diverges from live data/metadata generators.

Corpus

  • v2
  • v1
  • both
  • not applicable

Test plan

  • Executed uv run python -m clawbench.utils.update_llms to update llms.txt from live sources and verified generated markdown output.
  • Executed uv run pytest tests/test_update_llms.py (13 unit and regression test cases covering BibTeX parser, sitemap fetcher/fallback, leaderboard fetcher/fallback, quick facts, canonical resources, CLI dry-run, and workspace llms.txt regression check). All 13 tests passed.
  • Executed full entrypoint test suite uv run pytest tests/test_cli_entrypoints.py tests/test_update_llms.py (18 tests passed).

Related issues

Fixes #264

@Perry2004
Perry2004 self-requested a review August 11, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(site): keep llms.txt synchronized with live benchmark data

1 participant