““I must say, this is incredible. I’d never thought it’d be possible to run these massive models on consumer-grade compute.” — Hacker News commenter”
You know that feeling when a model fits the task you want to test but cannot fit in your RAM or VRAM? Colibri targets that gap for MoE models, where each token only needs a routed subset of experts. Instead of requiring every expert to stay in accelerator memory, it reads the selected experts from a storage hierarchy. You trade simpler memory planning for local storage requirements, hardware-sensitive speed, and backend validation work.
Think of the model as a library: Colibri keeps the main reference desk open in RAM or VRAM and fetches only the books the next request needs from NVMe. For each token, it runs the dense trunk, routes to a subset of MoE experts, then loads those experts through an LRU cache with pinned entries and lookahead. You set `COLI_MODEL` to a supported converted model and call `coli chat`, `coli web`, or `coli serve`. The C CPU engine runs without engine dependencies; optional CUDA, HIP, Metal, and Vulkan paths can handle supported accelerator configurations.
If you already run local models and have fast storage but limited RAM or VRAM, Colibri gives you a concrete MoE runtime to evaluate. It also suits systems engineers who want to study cache placement, expert routing, and reproducible performance evidence. Skip it if you need a low-risk serving platform across untested GPU configurations: open issues report output correctness problems on specific Vulkan and GB10 paths.
Treat Colibri as a beta-stage, specialist runtime worth testing on an exact model and hardware match. v1.11.0 adds a ninth model family, closes five bugs, and expands CI coverage, while the repository still reports 131 open issues and two open GPU-path correctness reports. Its value lies in its storage-tiered MoE design, not in a blanket production-readiness claim.
Deep-dive insight, Easy and Pro modes, plus action playbooks — the full breakdown is one tap away.