MemOps

Pipeline for generating and evaluating long-horizon "memory operations" benchmark data: personal background generation, evidence-conversation synthesis, distractor injection into unrelated conversations, and operation-metrics testing/evaluation against several memory/RAG baselines.

Pipeline

Step	Script	Purpose
1	1-generate_background.py	Generate personal background descriptions from info/topics.md
2	2-generate_evidence_conversation_w_verity.py	Generate + verify evidence conversations
3	3-generate_distractor_facts.py	Generate distractor facts for noisy longitudinal evaluation
4	4-inject_evidence_into_irrelevant_conversation.py	Inject evidence + distractors into sampled UltraChat conversations
5	5-test_operation_metrics.py	Run operation-metrics tests across RAG / long-context / no-context / mem0 baselines
5.5	5.5-evaluate_operation_metrics.py	LLM-judge evaluation of the step-5 outputs

Each step has a matching N-*.sh wrapper with sensible defaults (input/output dirs under generated_result/, model names, concurrency, etc.) — override any of it via environment variables, e.g. EVIDENCE_CONCURRENCY=8 ./2-generate_evidence_conversation_w_verity.sh.

Setup

pip install tqdm
# step 5 also needs, lazily, only if you use those baselines:
pip install openai rank-bm25

API key

Scripts resolve an LLM API key in this order:

1. Env vars: MEMTENSOR_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY / API_KEY
2. An api.md file at the repo root
3. A key.md file at the repo root (legacy name, still supported as a fallback)

api.md is the recommended file — create it locally (it's git-ignored) with either a bare key or an export KEY=... line:

sk-your-key-here

Some scripts (2, 3, 5) can rotate across multiple keys on a shared gateway — put one key per line in api.md to enable that.

LLM_BASE_URL controls the chat-completions endpoint and has no default — each script fails fast with a clear error if it's unset. Point it at any OpenAI-compatible endpoint, e.g. https://api.openai.com.

Note the default model names (claude-opus-4-6, gpt-4o, deepseek-v4-flash, glm-5.1, ...) span multiple model families and assume a gateway/router that can serve all of them behind one LLM_BASE_URL (e.g. OpenRouter, or your own proxy). If you're pointing straight at a single provider's API, override the per-step *_MODEL env vars (see each N-*.sh for the full list) to model names that provider actually serves.

Step 5's MemOS baseline additionally needs MEMOS_API_KEY (or a memos_api.md file); MEMOS_BASE_URL defaults to the public MemOS API and can be overridden.

UltraChat

Steps 2 and 4 sample "irrelevant" carrier conversations from UltraChat. Download it and point ULTRACHAT_PATH at it (defaults to ../UltraChat relative to this repo).

Not included: dense-rag / temp-lora baselines in step 5

5-test_operation_metrics.py can drive dense-rag and temp-lora baselines, but the dense_rag/ and temp_lora/ packages they import are internal components not published in this repo. Requesting those methods will fail with a clear RuntimeError pointing at the missing import; all other baselines (RAG, long-context, no-context, mem0, MemOS) are unaffected. bge-m3/ Qwen3.5-4B remain as the embedding/LoRA model defaults for anyone who wires their own implementation in.

License

MIT — see LICENSE.