Skip to content

docs: document token-consumption and retrieval-volume metrics - #1

Open
tcconnally wants to merge 1 commit into
mainfrom
docs/metrics-token-and-retrieval-volume
Open

docs: document token-consumption and retrieval-volume metrics#1
tcconnally wants to merge 1 commit into
mainfrom
docs/metrics-token-and-retrieval-volume

Conversation

@tcconnally

Copy link
Copy Markdown

Adds a Metrics section to the README defining token consumption, retrieval volume, and cost-normalized recall as first-class reporting axes alongside accuracy.

Grounded in HiSkill (arXiv:2607.25853): compact, structured, relation-aware context achieved +17.33% success rate while reducing inference token consumption by 78.75% vs the strongest baseline — token efficiency is a decision-relevant benchmark control, not a nice-to-have.

Documentation-only change; no harness behavior modified.

Adds a Metrics section grounding cost-normalized recall in the HiSkill
finding (arXiv:2607.25853): compact structured context reached +17.33%
success at -78.75% inference tokens.
@tcconnally

Copy link
Copy Markdown
Author
/pr-memory-benchmarks-1

Round-2 additions for the metrics PR

The existing token-consumption/retrieval-volume proposal is the right landing point. Please consider adding these bounded controls before merge:

  • AgentMemBench as a memory-benchmark arm covering external knowledge, personal QA, and updates.
  • SkillEvolBench for frozen-deployment, context-shift, adversarial, and composition controls; skill-library growth is not capability by itself.
  • Judge perturbations from 2602.02219, 2603.11027, and 2506.22316: paired reversals, rubric/order canaries, and judge/config hashes.
  • Energy-to-Token Production: quality-conditioned tokens and energy fields, with unavailable/estimated values explicit.

The public report should keep prompt/body content out of rows and distinguish measured metrics from estimates.

@tcconnally

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant