Skip to content

Add dry-run subcommand for token-usage measurement#750

Open
jehna wants to merge 2 commits into
mainfrom
claude/vigilant-noether-awdmR
Open

Add dry-run subcommand for token-usage measurement#750
jehna wants to merge 2 commits into
mainfrom
claude/vigilant-noether-awdmR

Conversation

@jehna
Copy link
Copy Markdown
Owner

@jehna jehna commented May 28, 2026

Summary

  • New humanify dry-run <file> [--context-size N] [--tokenizer cl100k_base|o200k_base] subcommand. Walks every identifier the real pipeline would walk, builds the exact (system, user, schema) triple the LLM renamer would send, tokenises it via tiktoken-rs, and reports calls plus per-call and aggregate token totals. No LLM, no API key.
  • Extracts build_rename_prompt() out of LlmRenamer so the dry-run path tokenises the exact bytes the real path would have sent — no chance of drift between the two.
  • Used to produce the numbers in v3 uses ~3× more tokens than v2 for the same input #749 (token-usage comparison vs. v2).

Example

$ humanify dry-run fixtures/example.min.js
humanify dry-run
tokenizer       cl100k_base
context_size    500
calls           6
system tokens   240
user tokens     822
schema tokens   234
total tokens    1296
avg total/call  216
avg user/call   137

Test plan


Generated by Claude Code

claude added 2 commits May 28, 2026 06:20
Walks every identifier the real pipeline would walk, builds the same
(system, user, schema) triple the LLM renamer would send, tokenises with
cl100k_base or o200k_base via tiktoken-rs, and reports calls plus per-call
and aggregate token totals. No LLM, no API key.

Extracts build_rename_prompt() from LlmRenamer so the dry-run path
tokenises the exact bytes the real path would have sent.

https://claude.ai/code/session_01EYuAm8hmd8HR8CR8xngogJ
Draft markdown body for the comparison issue referenced in #724. Kept in
the tree so it's reviewable; will be posted as a GitHub issue separately
once approved.

https://claude.ai/code/session_01EYuAm8hmd8HR8CR8xngogJ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants