Edge0/Edge0-35B-A3B-preview

TL;DR: Edge0-35B-A3B is a 4-bit, 256-expert MoE (Qwen3.6-35B-A3B base, K=4 active) that decodes at ~15 tok/s in under 3 GiB of active RAM on a Mac mini, by leaving the full checkpoint on SSD and streaming experts on demand. The interesting part isn't the checkpoint — it's the system around it: a learned prerouter that predicts next-step expert routing so SSD fetches overlap compute, plus distillation-trained "Recover-LoRA" adapters that claw back most of the int4 quality loss. That combination turns storage bandwidth into effective model capacity, which is a genuinely different point in the edge-inference design space than "quantize harder."

How it works

Three pieces, all shipping in this repo (base + lora_edge0_35b.safetensors + prerouter_edge0_35b.safetensors, auto-loaded by the edge0 framework):

SSD expert offload. The 4-bit checkpoint never gets loaded into RAM. Expert weights are fetched per-token as the router selects them, so peak memory is bounded by the active set (attention + shared params + 4-of-256 experts per layer × 40 layers), not the 35B total. No sharding, no upfront weight download into memory.

Prerouter. Naive offload stalls every layer on storage latency. Edge0 trains a small head that predicts expert routing one step ahead, so the next token's expert loads are issued while the current forward pass runs. They claim up to +59% decode throughput from this alone, with the gain growing with storage latency, model size, and routed width K — which makes sense: the more you'd otherwise stall, the more prediction buys you.

Recover-LoRA. Rather than merging quantization-aware finetuning into the weights, the int4 base is frozen and LoRA adapters are distilled from the FP16 teacher. Adapters stay unmerged, so one read-only base can serve many adapter sets — a nice property for batch serving multiple personas/tasks off one quantized artifact without re-quantizing.

Evidence

Self-reported, run by the authors with OpenCompass under identical settings for both models. Average gap of the full pipeline (int4 + adapters) vs the fp16 base: 3.9 points.

Speed on a Mac mini M4 Pro (24 GB): 14.9–17.7 tok/s decode, 113/140 tok/s prefill (cold/warm), 2.9 GiB peak active memory on short contexts. Note the gaps are largest on the reasoning-heavy benchmarks (AIME −6.1, HumanEval −4.2) — quantization losses tend to bite hardest exactly where long chains of tokens compound errors.

Caveats

• Everything is self-reported, and the model is inseparable from the edge0 framework — the checkpoint plus adapters only make sense loaded via edge0. MLX backend only, so Apple Silicon only for now; other backends are "roadmap."
• SSD implications are unaddressed. Reads-only, so wear shouldn't be a concern, but sustained random-read bandwidth on phone-class flash (vs Mac NVMe) is the real question for the "phone-class memory" framing — nothing here runs on a phone yet.
• The base is listed as "Qwen3.6-35B-A3B," which isn't a config I recognize from Qwen's public releases; the architecture tag qwen3_5_moe suggests a derivative. Worth checking provenance.
• 3 GiB is short-context peak; KV cache grows on top of it. Agentic/tool use is explicitly flagged as weak in this preview.
• There's an 8B-A1B sibling reported to run in ~1 GiB if you want the smaller end.

The prerouter idea is the piece most likely to outlive this particular release — speculative expert prefetch has floated around (LLM-in-a-flash, Mixtral offloading tricks), but shipping a trained prerouting head co-packaged with the checkpoint, with gains that scale with storage latency and K, is a clean productization of it. If it ports off MLX, it's a general recipe for running big sparse models on any machine with fast flash and little RAM.

Try it

pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'
huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview
export EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview
edge0 chat --name edge0-35b --prompt "Introduce yourself"
# or: edge0 serve --name edge0-35b --port 8085   (OpenAI-compatible API)

Apache 2.0. Needs Apple Silicon with fast internal storage; the full checkpoint download is large even though RAM stays under 3 GiB.