From 2bbf7108d3eb3a57cc180b53e9641de0f3a71ef5 Mon Sep 17 00:00:00 2001 From: tcconnally Date: Mon, 3 Aug 2026 14:46:12 +0000 Subject: [PATCH] docs: document token-consumption and retrieval-volume metrics Adds a Metrics section grounding cost-normalized recall in the HiSkill finding (arXiv:2607.25853): compact structured context reached +17.33% success at -78.75% inference tokens. --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index 16d39b0..7803956 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,23 @@ Open-source evaluation suite to run benchmarks on memory-augmented LLM systems. | **LongMemEval** | 500 diverse questions, 6 types | 500 | Long-term memory across information extraction, temporal, and multi-session reasoning | | **BEAM** | 100 conversations per size bucket (100K–10M tokens) | 2,000+ | Real-world memory retrieval across 10 memory ability types | +## Metrics + +Memory quality is the headline, but cost is a first-class axis: a system that +reaches the same recall with a fraction of the retrieved context is +materially better for long-horizon agents. Alongside accuracy/recall, report +when the harness exposes it: + +- **Token consumption** — prompt, retrieved-context, and generated tokens per query (or per task where the harness aggregates). +- **Retrieval volume** — number of memory facts / passages retrieved per query, and the context budget used. +- **Cost-normalized recall** — recall per 1k tokens, so providers are compared on efficiency, not only quality. + +This mirrors the evidence from HiSkill (arXiv:2607.25853): compact, +structured, relation-aware context achieved +17.33% success rate while +*reducing* inference token consumption by 78.75% versus the strongest +baseline. Token-efficiency is a decision-relevant benchmark control, not a +nice-to-have. + ## Quick Start ```bash