JustVugg/colibri

TL;DR: Colibrì is a pure-C, zero-dependency inference engine that runs 744B–2.8T-parameter open-weight MoE models on ordinary machines by treating NVMe, RAM, and VRAM as one placement hierarchy and streaming routed experts from disk on demand — "a JIT, but for weights." The honest numbers are modest (~1.8 tok/s on a 128 GB CPU desktop for GLM's 744B, up to ~6.8 tok/s on 6× RTX 5090), but the engineering discipline is unusual: placement changes speed, never semantics, everything is A/B'd end-to-end, and negative results are published in the README.

The core idea

A 744B MoE activates only ~40B parameters per token, and only ~11 GB of those (the routed experts) change token to token. Colibrì's bet: the dense part (~17B params — attention, shared experts, embeddings) lives resident in RAM at int4 (~9.9 GB), while the 19,456 routed experts (~19 MB each, ~370 GB total) stay on disk and are staged when the router asks for them. Measured routing heat drives a per-layer LRU plus a learned pinned hot-store persisted in .coli_usage, so the engine literally gets faster on your workload. Routing is claimed to be 71.6% predictable one layer ahead, so a lookahead thread prefetches while the current layer computes.

I/O is treated as part of the engine, not an externality: an expert's three matrices are stored adjacent and read in one pread, batched positions read each unique expert once, O_DIRECT is a tunable (+34% decode measured on one box, drive-dependent), and a second SSD holding a full or partial mirror gets deterministic hash-based read striping weighted by measured bandwidth. There's also a local cluster mode where Macs act as disk-backed expert workers over persistent TCP.

What's there

Nine model families run today, each as one C file over shared headers: GLM-5.2/5.3 (744B), GLM-5.3-Flash (321B, vision), Inkling (975B), Kimi K3 (2.8T, streamed straight from native MXFP4 shards, no conversion), DeepSeek V4/V4.1 Flash, Qwen3.8-Flash-Next, Qwen3.6, and OLMoE. Same coli chat / coli serve / coli web front end for all; prebuilt binaries for Linux/macOS/Windows; pre-converted int4 containers on Hugging Face (GLM-5.2 is 372 GB). Correctness is validated against a transformers oracle token-by-token; MLA KV state is stored 57× compressed (576 floats/token vs 32,768) and persists across restarts. There's a web dashboard with a measured "expert atlas" (13,260 characterized GLM-5.2 experts clustered by routing affinity — Python, SQL, poetry, Chinese — with live routing visualized per turn), and a "Brio mode" that scores a closed set of answer options by reading their probabilities instead of generating, with entropy as a calibrated uncertainty knob.

What to make of the claims

The README is refreshingly anti-hype for a 37k-star repo. It reports its own failures: MTP speculation measured a 32% loss at 85% expert hit rate; DeepSeek V4's drafters ship disabled because rejected-suffix replay through recurrent state cost more than drafts saved (one 14-token answer took 495 s); learned pins can overfit a prompt. An open-hypotheses table lists what still lacks controlled A/Bs. That's the right epistemics — but temper expectations accordingly: on realistic hardware this is 1–2 tok/s for the flagship model, dominated by disk bandwidth, with a 372 GB–1.6 TB download first. Quality at int4-gs64 has ablations but the older per-row int4 mirrors caused real degradation (think-mode loops), so container choice matters. This is a research platform for personal/agent use, not a serving stack.

For this reader, the interesting bits beyond accessibility: routing predictability (71.6% one layer ahead) as an exploitable structure, the expert atlas as a lightweight interpretability artifact that emerged from a systems need, and Brio mode as a cheap classification-via-LLM primitive (2.4–5.7× faster than generation, with usable entropy). The "one .c file per architecture over shared headers" design also makes it one of the more hackable frontier-model runtimes around.

Try it

Apache-2.0. Needs only gcc/clang with OpenMP (Python just for the launcher and converter). Start with Qwen3.6 (~20 GB, 24 GB RAM) before committing to a 372 GB download:

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh                      # builds + self-tests
make qwen36                     # or: make glm (needs 372 GB model + 16 GB RAM)
COLI_MODEL=/nvme/qwen36_i4_gs64 ./coli chat
./coli web --model /nvme/qwen36_i4_gs64   # dashboard + expert atlas

Model containers: GLM-5.2 int4-g64 (use gs64, int8 MTP head — the README is explicit that other variants break). Docs and benchmarks live in docs/.