drumih/turbo-fieldfare

TL;DR: TurboFieldfare is a from-scratch Swift + Metal inference runtime that runs Gemma 4 26B-A4B (26B total, ~3.88B active params/token) in roughly 2 GB of RAM on any Apple Silicon Mac, including base 8 GB machines. The trick is treating the MoE structure as a paging opportunity: the shared expert, attention weights, router, and FP16 KV cache stay resident (~1.35 GB core), while routed experts are streamed on demand from SSD with a small LFU cache. It's model-specific, not a llama.cpp/MLX wrapper, and comes with measured numbers: 5.1–6.3 tok/s on an 8 GB M2 Air, 31–35 tok/s on an M5 Pro.

Why this is interesting

Most local-inference stacks assume weights fit in unified memory (or spill to mmap and let the page cache flail). TurboFieldfare instead exploits a property specific to fine-grained MoE models: per token, only top-8 of the routed experts per layer are actually needed, and expert reuse across tokens is high enough that a 16-slot per-layer LFU cache plus bounded parallel pread from SSD keeps the GPU fed. The result is a 14.3 GB on-disk model running in a ~2 GB working set — a ~7x memory reduction that makes a 26B-class model usable on the cheapest Macs Apple sells. The technique should generalize to other MoE checkpoints, though this codebase deliberately doesn't.

How it works

Each decode step walks the 30 transformer layers (25 sliding-window with bounded circular KV storage, 5 full-attention with linear KV). Metal computes attention and the router from resident 4-bit weights; the CPU plans against the layer's 16-slot LFU expert cache and issues bounded parallel reads for misses directly into Metal-visible buffers, overlapped with the shared-expert compute. Prefill runs in chunks of up to 128 tokens so one fetched expert amortizes across many rows.

The installer is also memory-bounded: it streams byte ranges from the pinned Hugging Face revision and repacks them directly into the .gturbo layout without ever materializing the full checkpoint, then hash-verifies the result. Weights are MLX affine 4-bit (group 64), 8-bit router. So note: the "26B in 2 GB" headline rides on aggressive quantization plus SSD paging — there are no quality benchmarks in the README, only throughput.

What's actually there

A lot, for a solo project: the runtime library with custom Metal kernels (quantized GEMV, attention, MoE, RoPE, sampling, fusions), a native SwiftUI Mac app with a sibling decode-service process, a CLI, a loopback OpenAI-compatible server (streaming + function tools, client-side execution), the streaming installer/verifier, and an optional 1.1 GB vision tower pack for image input (M2+; it releases routed experts while the tower runs to stay in budget). There's a curated record of 103 audited experiments — including failed ideas — which is unusually honest engineering documentation. Apache-2.0 for the code; weights download separately under Google's terms.

Caveats

• Bleeding-edge platform lock: requires macOS 26, Metal 4, Swift 6.2/Xcode 26, arm64 only. No older macOS support.
• Single model: pinned Gemma 4 26B-A4B IT checkpoint only. The design is portable in principle; the code is not.
• 5–6 tok/s on an 8 GB M2 is usable, not fast, and the README correctly flags variance from prompt length and page-cache state. Sustained SSD read traffic per token is worth watching (wear, thermal, latency spikes) — the README doesn't quantify it.
• No quality evals on the 4-bit quantized model. Context in the memory figure is 4K KV; longer contexts cost more.
• Single-process discipline required: run only one model-owning executable at a time.
• Audio/video out of scope; tools only via the server, with client-side execution.

Try it

git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac   # Download (~15 GB), Load Model, prompt

# or CLI against an installed model:
swift run -c release TurboFieldfareCLI \
  --model scratch/gemma4.gturbo \
  --chat-prompt "Explain chunked prefill in one paragraph."

Worth a read even if you never run it: the "experiments that shaped TurboFieldfare" doc is a rare public record of what worked and what reversed under validation in SSD-streamed MoE inference — likely the most transferable artifact here.