diff --git a/README.md b/README.md index 16d39b0..7803956 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,23 @@ Open-source evaluation suite to run benchmarks on memory-augmented LLM systems. | **LongMemEval** | 500 diverse questions, 6 types | 500 | Long-term memory across information extraction, temporal, and multi-session reasoning | | **BEAM** | 100 conversations per size bucket (100K–10M tokens) | 2,000+ | Real-world memory retrieval across 10 memory ability types | +## Metrics + +Memory quality is the headline, but cost is a first-class axis: a system that +reaches the same recall with a fraction of the retrieved context is +materially better for long-horizon agents. Alongside accuracy/recall, report +when the harness exposes it: + +- **Token consumption** — prompt, retrieved-context, and generated tokens per query (or per task where the harness aggregates). +- **Retrieval volume** — number of memory facts / passages retrieved per query, and the context budget used. +- **Cost-normalized recall** — recall per 1k tokens, so providers are compared on efficiency, not only quality. + +This mirrors the evidence from HiSkill (arXiv:2607.25853): compact, +structured, relation-aware context achieved +17.33% success rate while +*reducing* inference token consumption by 78.75% versus the strongest +baseline. Token-efficiency is a decision-relevant benchmark control, not a +nice-to-have. + ## Quick Start ```bash