sqliteai/warp

TL;DR: WARP is a dependency-free C inference engine that runs the full 2.78T-parameter Kimi K3 on a 64 GB MacBook Pro at ~0.6 tok/s by keeping the model trunk in RAM and streaming the sparsely-activated MoE experts straight off NVMe, with the leftover RAM as a bounded expert cache. It is not a distill or a prune — 982 GB of weights on disk, logits verified against PyTorch to ~4e-6 — and the more practical result inside is GLM-5.3-Flash (313B) at 3.9 tok/s, where a 16 GB laptop runs at 90% of the speed of a 64 GB one. This treats NVMe bandwidth, not RAM, as the binding constraint on local MoE inference, and it's a pattern others will copy.

How it works

MoE models are ~4% active per token (K3: 2.78T total, ~110B active). WARP's design follows from that: the shared trunk (attention, embeddings, routers — 27.28 GB for K3) stays resident; each selected expert is laid out in the container so it costs exactly one aligned disk read; reads overlap compute; and a lookahead router predicts next-layer experts and prefetches them early, while the real router still decides — so lookahead changes timing, never output. Experts are 3-bit residual VQ, sensitive shared weights stay at 4–8 bits. K3's linear attention plus compressed latent KV cache keeps context cheap (0.21 GB at 4K vs 11.25 GB uncompressed), so the hard floor to open K3 is 29.19 GB.

The measurement discipline is unusually good. One finding worth remembering: growing the expert cache past the auto-chosen budget raises the hit rate (36→41%) while throughput collapses eightfold — the engine is inside its budget but the OS isn't, so cache hits become page faults. More memory is not always faster. There's also a quality knob: dropping K3 from top-16 to top-8 experts/token gives 1.49× speed at KL 0.037 and identical greedy continuations; top-4 breaks.

What's actually there

Code (C11, builds with make, no BLAS/Python/CUDA at inference time), CLI, OpenAI-compatible HTTP server with streaming/tools/images, embeddable libwaste.a, converters for K3, GLM-5.3-Flash, Kimi-Linear, and the DeepSeek-V3 family (V3/R1/K2), a BitTorrent magnet for the pre-converted 982 GB K3 container, and extensive docs including a deliberately-kept log of negative results (docs/LEARNED.md). Correctness is checked against PyTorch oracles layer-by-layer, with a thoughtful treatment of the fact that top-K routing makes logit distance a bad oracle for MoE. Vision works for both K3 and GLM (GLM's is cheap: 40 prompt positions for a small image; K3's is expensive at ~2.8 s per image position). Apache-2.0. Notably, the maintainers state the code is LLM-written with human-driven design — and the stated endgame is K3 improving WARP locally.

Caveats

• K3 is a demonstration, not a daily driver: 0.45–0.62 tok/s, ~17 GB read per cold token, 4.6 TB read per 200 tokens. Sustained NVMe read at that volume is real SSD wear; the README doesn't address endurance.
• Prefill runs at decode speed — a 2,000-token prompt takes minutes before the first output token. This is the biggest practical gap for chat/agent use.
• CPU-only today; Metal/CUDA are listed as unexplored. Format and API not frozen; project is moving fast.
• The container must be on internal NVMe (12.78 vs 0.94 GB/s on a tested USB enclosure). Multi-drive expert striping ships but is honestly labeled as unmeasured.
• The README is refreshingly anti-overclaim (ranges, failure modes, red tests explained), which makes its numbers more credible than most.

The real headline is GLM-5.3-Flash: a 313B multimodal model with a separate reasoning channel, running usably (3.9 tok/s) on a 16 GB machine with 112 GB of NVMe. That reframes what "consumer hardware" means for frontier-adjacent MoE models.

Try it

Start with Kimi-Linear (19 GB container, 1.32 GB RAM, ~17 tok/s) to test the engine, or GLM for a real model:

git clone https://github.com/sqliteai/warp && cd waste
make && make check   # model-free synthetic test suite

# GLM-5.3-Flash: 306 GiB staging + 112 GB internal NVMe
tools/fetch_weights.sh --repo zai-org/GLM-5.3-Flash --dest /staging/glm53
uv run --with torch python tools/convert.py \
  --src /staging/glm53 --out ~/models/glm53.waste --jobs 3
./waste chat ~/models/glm53.waste

Repo: https://github.com/sqliteai/warp — see docs/LEARNED.md for the negative-results log, which is worth reading on its own.