--prefix-cache-file never persists: unwritable <path>.<model_id> path, and prefix cache populates 0 blocks

[kira-ariaki]

Summary

--prefix-cache-file <path> documents two behaviours: persist the prefix cache on shutdown, and warm it on startup. Neither works in practice, for two independent reasons:

1. The default path is unwritable. The file is <path>.<model_id>, but model_id contains a / (e.g. basecompute/Qwen3-0.6B), so the resulting path lands inside a subdirectory that BaseRT never creates. The save silently no-ops.
2. Even with the directory pre-created, the cache is always empty. The file is written but is a 20-byte header with a block count of 0, and reload reports warmed 0 blocks — because --prefix-cache never populates any block in the first place (see the last section).

Environment

• BaseRT 0.1.7, official macOS arm64 release bundle
• macOS 26.4.1 (build 25E253), M1 Max 64 GB
• Model: basecompute/Qwen3-0.6B, default-q4

Part 1 — <path>.<model_id> is unwritable when the model id contains /

mkdir -p ~/basert-pcache
basert-serve --model .../basecompute/Qwen3-0.6B/default-q4/model.base \
  --port 8896 --host 127.0.0.1 --max-context 40960 \
  --prefix-cache-file ~/basert-pcache/pc

Startup log:

Prefix cache: starting cold (could not load /Users/me/basert-pcache/pc.basecompute/Qwen3-0.6B:
  prefix_cache load: could not open file for read: /Users/me/basert-pcache/pc.basecompute/Qwen3-0.6B)

The intended file is pc.basecompute/Qwen3-0.6B — i.e. the / in the model id turned pc.basecompute into a directory component.

Sending traffic, then stopping the server with SIGTERM (clean exit confirmed), leaves the directory empty:

--- after shutdown:
/Users/me/basert-pcache        <- nothing else

There is no warning at shutdown that the save failed. Restarting reports starting cold again.

Pre-creating the missing directory makes the save fire:

mkdir -p ~/basert-pcache/pc.basecompute   # <- the only change

Prefix cache: saved to /Users/me/basert-pcache/pc.basecompute/Qwen3-0.6B

That confirms the root cause: the code does open(<path>.<model_id>) without creating parent directories, and every real model id contains a /.

Part 2 — the persisted cache is always empty

With the directory pre-created, the file is written but contains no blocks:

-rw-r--r--  1 me  staff  20 Jul 25 18:22 .../pc.basecompute/Qwen3-0.6B

00000000: 4250 5843 0100 1000 9dc9 c966 99e7 4f38  BPXC.......f..O8
00000010: 0000 0000                                ....

20 bytes: BPXC magic + version + hash + a trailing 0 count. Reload confirms it:

Prefix cache: warmed 0 blocks from /Users/me/basert-pcache/pc.basecompute/Qwen3-0.6B

This holds after substantial shared-prefix traffic, and with --continuous-batching enabled (which --help states the prefix cache is "effective with").

Part 3 — --prefix-cache appears not to populate any block at all

This is why the persisted file is empty. Server:

basert-serve --model .../Qwen3-0.6B/default-q4/model.base \
  --port 8896 --host 127.0.0.1 --max-context 40960 \
  --prefix-cache-file ~/basert-pcache3/pc --continuous-batching 4

Continuous batching: enabled for basecompute/Qwen3-0.6B (max 4 lanes, prefix cache ON)

Eight requests all sharing an identical ~4400-token prefix, differing only in a short suffix (one unrelated warmup request first, to absorb one-time init):

CONCURRENT (4 requests, shared prefix):
  per-request elapsed: [8.832, 6.619, 2.183, 4.398]  makespan 8.83s
SEQUENTIAL (4 more, same shared prefix):
  per-request elapsed: [2.118, 2.145, 2.107, 2.03]   total 8.4s

Per-request prefill throughput from the access log — every request re-prefills the full 4403 tokens at the same rate:

prompt=301  prefill=4318 t/s     <- warmup, unrelated prompt
prompt=4403 prefill=2040 t/s
prompt=4403 prefill=2004 t/s
prompt=4403 prefill=2000 t/s
prompt=4403 prefill=2006 t/s
prompt=4403 prefill=2101 t/s
prompt=4403 prefill=2074 t/s
prompt=4403 prefill=2112 t/s
prompt=4403 prefill=2112 t/s

If the shared prefix were being reused, requests 2..8 should skip most of the prefill. Instead all eight cost ~2.0-2.1 s and report full prompt_tokens=4403. /slots also reports cache_tokens=0 cache_position=0 throughout.

Expected

1. --prefix-cache-file should create parent directories for <path>.<model_id> (or sanitize the / in the model id), and warn loudly if the shutdown save fails instead of silently doing nothing.
2. --prefix-cache should populate blocks for repeated prompt prefixes so that the persisted file is non-empty and reload reports a non-zero block count.