Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,23 @@ Open-source evaluation suite to run benchmarks on memory-augmented LLM systems.
| **LongMemEval** | 500 diverse questions, 6 types | 500 | Long-term memory across information extraction, temporal, and multi-session reasoning |
| **BEAM** | 100 conversations per size bucket (100K–10M tokens) | 2,000+ | Real-world memory retrieval across 10 memory ability types |

## Metrics

Memory quality is the headline, but cost is a first-class axis: a system that
reaches the same recall with a fraction of the retrieved context is
materially better for long-horizon agents. Alongside accuracy/recall, report
when the harness exposes it:

- **Token consumption** — prompt, retrieved-context, and generated tokens per query (or per task where the harness aggregates).
- **Retrieval volume** — number of memory facts / passages retrieved per query, and the context budget used.
- **Cost-normalized recall** — recall per 1k tokens, so providers are compared on efficiency, not only quality.

This mirrors the evidence from HiSkill (arXiv:2607.25853): compact,
structured, relation-aware context achieved +17.33% success rate while
*reducing* inference token consumption by 78.75% versus the strongest
baseline. Token-efficiency is a decision-relevant benchmark control, not a
nice-to-have.

## Quick Start

```bash
Expand Down