From c4afbde98c059844587cbafe6bdeda4e1597a345 Mon Sep 17 00:00:00 2001 From: Michael Feth Date: Fri, 21 Aug 2026 19:14:36 -0400 Subject: [PATCH 1/2] Dump what the router thought of the experts that lost, not only the winners WASTE_DUMP_ROUTE records the top-K ids and their weights, which answers every question about the ranking the router actually made. It answers none about a ranking it did not. That gap is load-bearing. A residency prior promotes an expert the real router left just outside the top-K; a dynamic-k rule needs to know how far the k+1th sat behind the kth; a pruning criterion needs the scores of the experts it is proposing to delete. All three are questions about the losers, and a trace of the chosen cannot say which one, or by how much. So WASTE_DUMP_SCORES=path appends the whole vector: pos L v0 v1 .. v(E-1) `score[e] + bias[e]`, the quantity the selection loop ranks on -- not `w[j]`, which is the weight applied afterwards and takes `score[best]` without the bias. Getting that distinction wrong would make every offline re-ranking experiment quietly disagree with the engine. The reasoning is the one already written above WASTE_DUMP_ROUTE -- "is the signal there", and cheaper from a trace than from a build -- and so are the economics: one fprintf against a rebuild plus a sweep. It is the same shape of instrument, one field wider. Unset it costs a null check per layer and writes nothing. `make check` is 44 passed / 0 failed / 13 skipped, unchanged. Verified on Kimi-Linear: 3 tokens x 26 MoE layers wrote 78 lines of 258 fields (pos, layer, 256 experts). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01B5eVMiauR4Lkt8Dhc67MNb --- src/model.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/model.c b/src/model.c index 231247c..f1cd3e7 100644 --- a/src/model.c +++ b/src/model.c @@ -164,6 +164,7 @@ static int q8_off = 1; /* 1 = keep the trunk stored as int8 */ static int sdot_on = 0; /* 1 = also quantize activations (SDOT path) */ static int i8mm_on = 0; /* SMMLA batched matmul; costs activation int8 */ static const char *dump_route = NULL; /* WASTE_DUMP_ROUTE, see moe_layer */ +static const char *dump_scores = NULL; /* WASTE_DUMP_SCORES, see moe_layer */ /* Absolute position of the first token of the pass being routed. The dump * names each row by the token it belongs to rather than leaving a reader * to infer it from where the layer index wraps — which is a heuristic @@ -193,6 +194,7 @@ static void model_opts_init(void) /* Read once rather than per layer per token: moe_layer runs 92 * times a token and getenv is not free. */ dump_route = getenv("WASTE_DUMP_ROUTE"); + dump_scores = getenv("WASTE_DUMP_SCORES"); /* How many of the next layer's experts to fetch on the router's guess. * The layer boundary holds about six reads and the prediction's * precision falls off past there, so that is the default. 0 is off. */ @@ -2746,6 +2748,32 @@ static void moe_layer(waste_model *m, int L, const float *in, float *out, int *r for (int j = 0; j < K; j++) w[j] *= c->routed_scale; if (routed) for (int j = 0; j < K; j++) routed[j] = idx[j]; + /* WASTE_DUMP_SCORES=path appends one line per (token, layer): + * + * pos L v0 v1 .. v(E-1) + * + * the *selection* value for every expert, `score[e] + bias[e]` — the + * quantity the loop above ranks on, not the weight it later applies. + * + * WASTE_DUMP_ROUTE records the ids that won, and no question about a + * *different* ranking can be answered from winners alone. Anything that + * would reorder the top-K — a residency prior, a dynamic k, a pruning + * criterion — needs to know what the experts that lost were worth, and + * by how much they lost. A trace of the chosen cannot say. + * + * Same reasoning as the comment below, and the same economics: this is + * one fprintf against a rebuild plus a sweep. */ + if (dump_scores) { + FILE *sf = fopen(dump_scores, "a"); + if (sf) { + fprintf(sf, "%d %d", dump_pos0, L); + for (int e = 0; e < E; e++) + fprintf(sf, " %.6g", score[e] + (bias ? bias[e] : 0.0f)); + fputc('\n', sf); + fclose(sf); + } + } + /* WASTE_DUMP_ROUTE=path appends one line per (token, layer): * * L id0..idK-1 w0..wK-1 From 33b4d2e99f43ed9d5d46da69142456bf8feaed05 Mon Sep 17 00:00:00 2001 From: Michael Feth Date: Fri, 21 Aug 2026 19:15:34 -0400 Subject: [PATCH 2/2] A residency probe, and a router knob to measure it with MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stacked on the score dump, which is what made the offline half of this measurable. Both halves of the mechanism arXiv:2412.00099 describes, and neither is a default. `waste_ecache_resident_mask` answers "which of this layer's experts are here already" without claiming one, moving recency, or counting a hit. It takes the cache lock once for the whole layer rather than once per expert, because a single mutex covers the cache and the per-expert shape would take it E times per layer per token -- 83k times a token on K3. INFLIGHT reads as absent on purpose: a slot that has been asked for and has not arrived would send a router that trusted it straight into the read it was avoiding. `WASTE_CCR_LAMBDA` adds `lambda * range * resident[e]` to the selection value, ranking only. `w[j]` still takes the untouched `score[best]`, so the gating weights are what the model computed and only the choice of which K to run moves. Scaling by the layer's own logit range makes one lambda mean the same thing in a layer spanning 0.9 and one spanning 0.02. The paper averages that range over time; this uses the token in hand, because static mutable state would be wrong when waste.h promises several models open at once. Measured on Kimi-Linear with the lookahead ON, which was the question -- a prefetcher that makes about-to-be-needed experts resident and a prior that prefers resident experts could easily have been the same gain twice: lambda hit% GB/token KL(ref||ccr) 0 70.6 0.322 0 0.05 76.9 0.294 0.0166 0.10 82.5 0.249 0.0232 0.25 87.0 0.227 0.0525 lambda=0 reproduces the engine's measured baseline exactly, so the control holds and +11.9 points at 0.10 is on top of §35's prefetch rather than overlapping it. 22.7% fewer bytes per token at KL 0.0232 -- against README's own prices, top-8 is KL 0.037 and ships, top-4 is KL 0.118 and was refused. Unset it is inert: `make check` 44 passed / 0 failed / 13 skipped. It cannot become a default and §54 is why: "what routing buys is not which experts exist, nor how they are weighted on average -- it is the per-token exclusion", and this edits precisely that. Three things stand between here and a proposal, and none of them are done: the KL is one position on one prompt; it does *not* reproduce the greedy continuation, where top-8 clears both bars; and the §54 machine-wide effect is unchecked, so a hit-rate gain may yet be paid for in buckets that read no experts at all. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01B5eVMiauR4Lkt8Dhc67MNb --- src/ecache.c | 17 ++++++++++++++++ src/ecache.h | 16 +++++++++++++++ src/model.c | 56 ++++++++++++++++++++++++++++++++++++++++++---------- 3 files changed, 79 insertions(+), 10 deletions(-) diff --git a/src/ecache.c b/src/ecache.c index dd0cb5a..ff5d619 100644 --- a/src/ecache.c +++ b/src/ecache.c @@ -593,6 +593,23 @@ void waste_ecache_hint(waste_ecache *c, int layer, const int *ids, int n) ec_unlock(c); } +void waste_ecache_resident_mask(waste_ecache *c, int layer, int n, uint8_t *out) +{ + if (!out || n <= 0) return; + memset(out, 0, (size_t)n); + if (!c || c->n_slots <= 0) return; + + ec_lock(c); + for (int e = 0; e < n; e++) { + const int si = ec_lookup(c, ec_key(layer, e)); + /* READY only. An INFLIGHT slot has been asked for and has not + * arrived, so a router told it was resident would pick it and then + * wait on the same read it was trying to avoid. */ + if (si >= 0 && c->slot[si].state == EC_READY) out[e] = 1u; + } + ec_unlock(c); +} + const uint8_t *waste_ecache_get(waste_ecache *c, int layer, int expert, waste_fetch_fn fetch, void *user) { diff --git a/src/ecache.h b/src/ecache.h index 23e47b1..cd60bc2 100644 --- a/src/ecache.h +++ b/src/ecache.h @@ -167,6 +167,22 @@ void waste_ecache_io_stop(waste_ecache *c); * Reads for the first `depth` of them that are not resident start now, and * each waste_ecache_get releases one more into the pipe. Calling it with a * synchronous cache is a no-op, so call sites need no conditional. */ +/* Residency of a whole layer's experts, without touching any of them. + * + * `out[e]` becomes 1 when expert e of `layer` is in the cache and READY — + * usable with no disk read. Nothing is claimed, no recency moves, and + * nothing counts as a hit or a miss: this answers a question, it does not + * make a request. INFLIGHT deliberately reads as 0, so "resident" means + * arrived rather than asked for. + * + * One call per layer rather than one per expert because a single mutex + * covers the whole cache, and a per-expert probe would take it E times per + * layer per token — 83k times a token on K3. + * + * For a router that wants to prefer experts it already has (arXiv:2412.00099). + */ +void waste_ecache_resident_mask(waste_ecache *c, int layer, int n, uint8_t *out); + void waste_ecache_hint(waste_ecache *c, int layer, const int *ids, int n); /* Speculative fill for a layer that has not routed yet. Unlike a hint these diff --git a/src/model.c b/src/model.c index f1cd3e7..756d849 100644 --- a/src/model.c +++ b/src/model.c @@ -165,6 +165,7 @@ static int sdot_on = 0; /* 1 = also quantize activations (SDOT path) */ static int i8mm_on = 0; /* SMMLA batched matmul; costs activation int8 */ static const char *dump_route = NULL; /* WASTE_DUMP_ROUTE, see moe_layer */ static const char *dump_scores = NULL; /* WASTE_DUMP_SCORES, see moe_layer */ +static float ccr_lambda = 0.0f; /* WASTE_CCR_LAMBDA, see moe_layer */ /* Absolute position of the first token of the pass being routed. The dump * names each row by the token it belongs to rather than leaving a reader * to infer it from where the layer index wraps — which is a heuristic @@ -195,6 +196,9 @@ static void model_opts_init(void) * times a token and getenv is not free. */ dump_route = getenv("WASTE_DUMP_ROUTE"); dump_scores = getenv("WASTE_DUMP_SCORES"); + { const char *s = getenv("WASTE_CCR_LAMBDA"); + ccr_lambda = s ? (float)atof(s) : 0.0f; + if (!(ccr_lambda > 0.0f)) ccr_lambda = 0.0f; } /* also catches NaN */ /* How many of the next layer's experts to fetch on the router's guess. * The layer boundary holds about six reads and the prediction's * precision falls off past there, so that is the default. 0 is off. */ @@ -2725,6 +2729,37 @@ static void moe_layer(waste_model *m, int L, const float *in, float *out, int *r float *score = sc + E; for (int e = 0; e < E; e++) score[e] = 1.0f / (1.0f + expf(-sc[e])); + /* Cache-conditional routing (arXiv:2412.00099), off unless WASTE_CCR_LAMBDA + * asks for it. Experts already in the cache get a bonus **for ranking + * only** — `w[j]` below still takes the untouched `score[best]`, so the + * gating weights are exactly what the model computed and only the choice + * of which K to run moves. + * + * The bonus is scaled by this layer's own logit range so one lambda means + * the same thing in a layer whose scores span 0.9 and one whose scores + * span 0.02. The paper uses a running mean of that range; this uses the + * range of the token in hand, which needs no per-layer state — and static + * mutable state would be wrong here anyway, since waste.h promises several + * models can be open in one process. + * + * This changes what the model outputs. LEARNED §54 is the reason it can + * never become a default: "what routing buys is not which experts exist, + * nor how they are weighted on average — it is the per-token exclusion", + * and this edits precisely that. It is an instrument for measuring the + * hit-rate/KL trade, and it stays one until a KL curve says otherwise. */ + uint8_t resident[1024]; + float ccr_bump = 0.0f; + if (ccr_lambda > 0.0f && E <= (int)sizeof resident) { + waste_ecache_resident_mask(&m->cache, L, E, resident); + float lo = 1e30f, hi = -1e30f; + for (int e = 0; e < E; e++) { + const float v = score[e] + (bias ? bias[e] : 0.0f); + if (v < lo) lo = v; + if (v > hi) hi = v; + } + ccr_bump = ccr_lambda * (hi - lo); + } + int idx[64]; float w[64]; for (int j = 0; j < K; j++) { @@ -2734,11 +2769,12 @@ static void moe_layer(waste_model *m, int L, const float *in, float *out, int *r int taken = 0; for (int p = 0; p < j; p++) if (idx[p] == e) { taken = 1; break; } if (taken) continue; - const float v = score[e] + (bias ? bias[e] : 0.0f); + float v = score[e] + (bias ? bias[e] : 0.0f); + if (ccr_bump > 0.0f && resident[e]) v += ccr_bump; if (v > bv) { bv = v; best = e; } } idx[j] = best; - w[j] = score[best]; + w[j] = score[best]; /* unmodified: ranking moved, gating did not */ } if (c->renorm && K > 1) { float s = 0; @@ -2755,14 +2791,14 @@ static void moe_layer(waste_model *m, int L, const float *in, float *out, int *r * the *selection* value for every expert, `score[e] + bias[e]` — the * quantity the loop above ranks on, not the weight it later applies. * - * WASTE_DUMP_ROUTE records the ids that won, and no question about a - * *different* ranking can be answered from winners alone. Anything that - * would reorder the top-K — a residency prior, a dynamic k, a pruning - * criterion — needs to know what the experts that lost were worth, and - * by how much they lost. A trace of the chosen cannot say. - * - * Same reasoning as the comment below, and the same economics: this is - * one fprintf against a rebuild plus a sweep. */ + * WASTE_DUMP_ROUTE records the ids that won. Any question about a + * *different* ranking needs the ones that lost too: a residency prior + * (arXiv:2412.00099) promotes an expert the real router placed outside + * the top-K, and a trace of winners cannot say which one or by how much. + * So this dumps the whole vector, and the question "would a cache-aware + * ranking hit more often, and how far does the distribution move" is + * answerable offline — which is the same reasoning, and the same + * economics, as the comment below. */ if (dump_scores) { FILE *sf = fopen(dump_scores, "a"); if (sf) {