Skip to content

Hybrid (Qwen3.5/3.6 GDN) models serialize all requests: support N independent recurrent states #31

Description

@kira-ariaki

Summary

Qwen3.5/3.6 hybrid models (Gated DeltaNet) currently keep a single recurrent state, so baseRT_sequence_create is rejected and every request is served strictly one at a time. Measured on an M1 Max, four concurrent requests take exactly 4× the wall time of one, with aggregate throughput flat — the second, third and fourth requests gain nothing from concurrency.

Both vLLM and llama.cpp allocate N independent recurrent states for hybrid/recurrent architectures, so this looks like an implementation gap rather than an architectural limit.

Environment

  • BaseRT 0.1.7, official macOS arm64 release bundle
  • macOS 26.4.1 (build 25E253), M1 Max 64 GB
  • Model: Qwen/Qwen3.6-35B-A3B, default-q4

Current behaviour

baseRT_sequence_create is not supported for Qwen3.5/3.6 hybrid models
(Gated-DeltaNet recurrent state is single-sequence only)

Because sequences cannot be created, --continuous-batching has no lanes to fill for these models and requests serialize.

Measurement

512 generated tokens per request, identical prompt shape, same process:

N=1   makespan 5.85 / 5.51 / 5.37 s     aggregate 87.5 / 93.0 / 95.3 tok/s
N=4   makespan 22.05 / 22.49 s          aggregate 92.9 / 91.1 tok/s

Per-request completion times in the N=4 run:

request 0 finished at  5.5 s
request 1 finished at 11.0 s
request 2 finished at 16.8 s
request 3 finished at 22.5 s

A perfect staircase in units of the single-request latency — the definition of full serialization. Aggregate throughput at N=4 (~92 tok/s) is identical to N=1 (~92 tok/s), so concurrency currently buys nothing on hybrid models.

Prior art

  • vLLMMambaCacheManager maintains per-sequence recurrent state, and the hybrid allocator manages attention KV blocks and recurrent (SSM/GDN) state side by side for Qwen3-Next-style hybrids.
  • llama.cppllama_memory_recurrent allocates one recurrent state slot per sequence, sized by n_seq_max, alongside the regular KV cache.

Both ship this for production hybrid serving today.

Why this looks tractable

Recurrent state is constant-size — it does not grow with sequence length, unlike a KV cache. For Qwen3.6-35B-A3B the per-sequence GDN state is small relative to the 20 GB of weights, so supporting even 4-8 lanes should cost little memory.

From the shipped binary, baseRT::GDNStateCache::{create,destroy} already exist as independently constructed/destroyed objects, which suggests the allocation site is well isolated. GDNStateCache::create computes its size from n_layers, k_heads, head_dim and conv_k only — a single state, with no slot/sequence dimension. Adding an outer slot dimension there (and a slot index in the GDN kernels: gdn_in_proj, gdn_conv_silu, gdn_recurrence, gdn_mp_factor_f16, gdn_mp_scan_f16) appears to be the shape of the change.

Scope note

This request is specifically about N independent recurrent states so that concurrent requests stop serializing.

It is explicitly not a request for prefix sharing on GDN layers — a recurrent state is a compressed summary of all preceding tokens and cannot be truncated or forked at an arbitrary position, so prefix reuse across different requests is a genuine architectural limitation for those layers. Independent per-sequence state does not require it.

Impact

On a 64 GB M1 Max, Qwen3.6-35B-A3B is otherwise an excellent fit (~92 tok/s single stream, 262 K context verified working). But because every request serializes, the box cannot serve more than one user at a time, which currently rules the model out for any multi-user or agentic workload with parallel tool calls.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions