Summary
Qwen3.5/3.6 hybrid models (Gated DeltaNet) currently keep a single recurrent state, so baseRT_sequence_create is rejected and every request is served strictly one at a time. Measured on an M1 Max, four concurrent requests take exactly 4× the wall time of one, with aggregate throughput flat — the second, third and fourth requests gain nothing from concurrency.
Both vLLM and llama.cpp allocate N independent recurrent states for hybrid/recurrent architectures, so this looks like an implementation gap rather than an architectural limit.
Environment
- BaseRT 0.1.7, official macOS arm64 release bundle
- macOS 26.4.1 (build 25E253), M1 Max 64 GB
- Model:
Qwen/Qwen3.6-35B-A3B, default-q4
Current behaviour
baseRT_sequence_create is not supported for Qwen3.5/3.6 hybrid models
(Gated-DeltaNet recurrent state is single-sequence only)
Because sequences cannot be created, --continuous-batching has no lanes to fill for these models and requests serialize.
Measurement
512 generated tokens per request, identical prompt shape, same process:
N=1 makespan 5.85 / 5.51 / 5.37 s aggregate 87.5 / 93.0 / 95.3 tok/s
N=4 makespan 22.05 / 22.49 s aggregate 92.9 / 91.1 tok/s
Per-request completion times in the N=4 run:
request 0 finished at 5.5 s
request 1 finished at 11.0 s
request 2 finished at 16.8 s
request 3 finished at 22.5 s
A perfect staircase in units of the single-request latency — the definition of full serialization. Aggregate throughput at N=4 (~92 tok/s) is identical to N=1 (~92 tok/s), so concurrency currently buys nothing on hybrid models.
Prior art
- vLLM —
MambaCacheManager maintains per-sequence recurrent state, and the hybrid allocator manages attention KV blocks and recurrent (SSM/GDN) state side by side for Qwen3-Next-style hybrids.
- llama.cpp —
llama_memory_recurrent allocates one recurrent state slot per sequence, sized by n_seq_max, alongside the regular KV cache.
Both ship this for production hybrid serving today.
Why this looks tractable
Recurrent state is constant-size — it does not grow with sequence length, unlike a KV cache. For Qwen3.6-35B-A3B the per-sequence GDN state is small relative to the 20 GB of weights, so supporting even 4-8 lanes should cost little memory.
From the shipped binary, baseRT::GDNStateCache::{create,destroy} already exist as independently constructed/destroyed objects, which suggests the allocation site is well isolated. GDNStateCache::create computes its size from n_layers, k_heads, head_dim and conv_k only — a single state, with no slot/sequence dimension. Adding an outer slot dimension there (and a slot index in the GDN kernels: gdn_in_proj, gdn_conv_silu, gdn_recurrence, gdn_mp_factor_f16, gdn_mp_scan_f16) appears to be the shape of the change.
Scope note
This request is specifically about N independent recurrent states so that concurrent requests stop serializing.
It is explicitly not a request for prefix sharing on GDN layers — a recurrent state is a compressed summary of all preceding tokens and cannot be truncated or forked at an arbitrary position, so prefix reuse across different requests is a genuine architectural limitation for those layers. Independent per-sequence state does not require it.
Impact
On a 64 GB M1 Max, Qwen3.6-35B-A3B is otherwise an excellent fit (~92 tok/s single stream, 262 K context verified working). But because every request serializes, the box cannot serve more than one user at a time, which currently rules the model out for any multi-user or agentic workload with parallel tool calls.
Summary
Qwen3.5/3.6 hybrid models (Gated DeltaNet) currently keep a single recurrent state, so
baseRT_sequence_createis rejected and every request is served strictly one at a time. Measured on an M1 Max, four concurrent requests take exactly 4× the wall time of one, with aggregate throughput flat — the second, third and fourth requests gain nothing from concurrency.Both vLLM and llama.cpp allocate N independent recurrent states for hybrid/recurrent architectures, so this looks like an implementation gap rather than an architectural limit.
Environment
Qwen/Qwen3.6-35B-A3B,default-q4Current behaviour
Because sequences cannot be created,
--continuous-batchinghas no lanes to fill for these models and requests serialize.Measurement
512 generated tokens per request, identical prompt shape, same process:
Per-request completion times in the N=4 run:
A perfect staircase in units of the single-request latency — the definition of full serialization. Aggregate throughput at N=4 (~92 tok/s) is identical to N=1 (~92 tok/s), so concurrency currently buys nothing on hybrid models.
Prior art
MambaCacheManagermaintains per-sequence recurrent state, and the hybrid allocator manages attention KV blocks and recurrent (SSM/GDN) state side by side for Qwen3-Next-style hybrids.llama_memory_recurrentallocates one recurrent state slot per sequence, sized byn_seq_max, alongside the regular KV cache.Both ship this for production hybrid serving today.
Why this looks tractable
Recurrent state is constant-size — it does not grow with sequence length, unlike a KV cache. For Qwen3.6-35B-A3B the per-sequence GDN state is small relative to the 20 GB of weights, so supporting even 4-8 lanes should cost little memory.
From the shipped binary,
baseRT::GDNStateCache::{create,destroy}already exist as independently constructed/destroyed objects, which suggests the allocation site is well isolated.GDNStateCache::createcomputes its size fromn_layers,k_heads,head_dimandconv_konly — a single state, with no slot/sequence dimension. Adding an outer slot dimension there (and a slot index in the GDN kernels:gdn_in_proj,gdn_conv_silu,gdn_recurrence,gdn_mp_factor_f16,gdn_mp_scan_f16) appears to be the shape of the change.Scope note
This request is specifically about N independent recurrent states so that concurrent requests stop serializing.
It is explicitly not a request for prefix sharing on GDN layers — a recurrent state is a compressed summary of all preceding tokens and cannot be truncated or forked at an arbitrary position, so prefix reuse across different requests is a genuine architectural limitation for those layers. Independent per-sequence state does not require it.
Impact
On a 64 GB M1 Max,
Qwen3.6-35B-A3Bis otherwise an excellent fit (~92 tok/s single stream, 262 K context verified working). But because every request serializes, the box cannot serve more than one user at a time, which currently rules the model out for any multi-user or agentic workload with parallel tool calls.