“"currently the best option especially combined with AMD Epyc NPS0" — but also "requires at least 16GB VRAM CUDA GPU" — ubergarm, comparing inference engines at https://github.com/ubergarm/r1-ktransformers-guide/issues/11”
You know that feeling when the model you want to run is 671 billion parameters and your GPU has 24GB? The usual answer is to shrink the model until it fits, and watch quality go with it, or to stream expert weights over PCIe and watch a 32 GB/s link become the bottleneck. The research notes the older approach's real cost: Fiddler-style offloading fired over 7,000 CUDA kernel launches per decoded token, burning 73% of GPU execution time on launch overhead alone, while PyTorch's oneDNN AMX path hit only 7% of the CPU's theoretical peak. So you end up with idle silicon on both sides of the PCIe bus and 4.68 tokens/s to show for it.
Think of a mixture-of-experts model as a hospital: every patient sees reception, but then gets routed to two or three specialists out of a couple hundred on staff. Reception is small and busy, so KTransformers puts it on the GPU — attention plus the shared experts. The specialists are numerous but each one is rarely called, so they stay in CPU DRAM and do their work there, with kernels hand-written for Intel AMX during prefill and AVX-512 during decode. The clever part is Expert Deferral: instead of making the GPU wait for every specialist to finish, a subset of experts is deferred one layer forward and computed while the GPU handles the next layer's attention, then merged back in. The paper reports this pushes CPU and GPU utilization from 74%/28% to 100%/37% on DeepSeek-V3, a 33% decode throughput gain, at an average accuracy cost of 0.5% versus 13.3% if you had simply skipped those experts.
If you run large MoE models on your own hardware — a lab, a regulated industry where data cannot leave the building, or a team pricing an on-prem box against an H100 cluster — this is the technique to understand. It pays off best if you have an AMX-capable Xeon or an EPYC 9000, hundreds of GB of DDR5, and at least a 16GB CUDA GPU. Skip it if you are serving high concurrency: the paper explicitly evaluates batch size 1 and scopes itself to 'low concurrency (e.g., single or few requests per batch)', and it will not help you on Apple Silicon, ARM, or a laptop.
Yes, if you are sizing on-prem inference hardware — the technique is peer-reviewed at SOSP '25, ships monthly releases, and is running commercially inside Approaching.AI's token platform, which is about as strong a production signal as an academic kernel library gets. But treat the marketing framing skeptically: the project site says 'Deploy 100B+ parameter models locally with just a single RTX 5090', while engineers who ran it describe a floor of 16GB VRAM, 600GB–1TB of DDR5, and specific Intel or AMD generations. Also note that the original integrated framework has been moved to archive/ and serving is now delegated to SGLang, so integrate against kt-kernel and SGLang, not the pre-v0.4.2 ...
Deep-dive insight, Easy and Pro modes, plus action playbooks — the full breakdown is one tap away.