30 tok/s for 20B MoE on 8 GB VRAM. Flat throughput to 32K context. Native MXFP4 + GGUF Q4_K/Q5_K/Q6_K via ggml CUDA kernels — zero dequant. Expert offloading for models that don't fit in GPU memory.
Run a 397B MoE model on a MacBook with C/Metal inference, 4.4+ tok/s, and tool calling
Add a description, image, and links to the expert-offloading topic page so that developers can more easily learn about it.
To associate your repository with the expert-offloading topic, visit your repo's landing page and select "manage topics."