memovai/mimimodel

TL;DR: MimiModel is a ~2,000-line, dependency-free C99 engine that runs Cactus Compute's Needle 2 — a 45M-parameter, 2-bit quantized tool-calling LLM — entirely on a $5 ESP32-S3, with the 13.7 MB of weights memory-mapped in flash and never expanded into RAM. It matches the official (closed, ARM-only) engine's accuracy on a 961-case function-calling eval (69.6% vs 69.2%), and the engineering write-up — a Hadamard-identity kernel that multiplies straight off packed 2-bit bytes, a KV prefix cache worth 8.2× end-to-end, grammar-constrained decoding — is unusually rigorous and honest.

What it is and why it matters

The official Needle engine can't run here: its compute core ships as prebuilt binaries and its open kernels target ARM NEON. This project reimplements the open .cact model spec from scratch for the Xtensa LX7, llama2.c-style, and validates the whole chain per-position, per-logit against the official JAX decode (max diff 3e-4). The result is structured tool calling — "turn on pin 5" → [{"name":"gpio_on","arguments":{"pin":5}}] — with no Linux, no Python, no network, on a microcontroller with 512 KB of internal SRAM.

The interesting part for a systems-minded reader is how 13.7 MB of weights get used from a chip with 8 MB PSRAM. The answer sets a template for extreme-edge inference.

The three tricks that make it work

1. Multiply without dequantizing. Cactus Quants stores weights as 2-bit indices into a shared unit-sphere codebook, plus one fp16 norm per 128-element group, reconstructed through a Walsh–Hadamard rotation. Naively you'd expand 13.7 MB of weights every token — impossible. Because H is symmetric and orthogonal, (unit·H)·x ≡ unit·(H·x): transform the activation once per 128-group with a fast WHT (896 adds), then the matvec is a codebook-weighted dot product read in place from flash via esp_partition_mmap. Model "load" takes 48 ms.

2. KV prefix cache. In a tool-calling agent the tools block is byte-identical every call (288 of ~300 prompt tokens). Its KV rows stay live across requests; only the query is prefilled. Historically: 241 s cold → 29 s warm (8.2×). An attention sink protects the first 160 prompt tokens beside a 256-token recent window, keeping the int8 KV cache at 416 rows.

3. Grammar-constrained decoding. Every candidate token's bytes are validated against one continuous grammar compiled from active tool schemas — no teacher-forced fragments. This was the single biggest accuracy fix: the old segmented approach scored 48.8%; the paired ablations recovering the 21-point gap (reasoning-token cutoff, prefix preservation, continuous grammar) are documented.

Evidence

On google/mobile-actions (961 cases, ordered strict exact match): 69.6% vs the official engine's 69.2% on identical inputs — though error distributions differ (name accuracy 90.8% vs 98.1%; MimiModel wins more argument rows, under-calls more). On device, the fixed one-tool workload:

Larger schemas take minutes (a 333-token input: 413 s, matching host output byte-for-byte). The optimization log is a good read on its own — including negative results like a hand-written int16 PIE assembly path that was numerically correct and 3× slower because 2-bit unpacking dominates.

What's there and what to expect

Everything is in the repo: engine (needle.c), host CLI with persistent serial daemon, numpy reference cross-checked against JAX, ESP-IDF project, benchmark harnesses with raw artifacts. MIT license; weights are Apache-2.0, downloaded separately from Hugging Face. You need an ESP32-S3 with 16 MB flash + 8 MB PSRAM and ESP-IDF v5.5+.

Caveats the README itself flags: English-only, it will call a tool when you say hello (no decline behavior; the confidence head exists in the weights but isn't wired up), boolean arguments are ~coin-flip (splitting gpio_write(pin,state) into gpio_on/gpio_off fixed it), and the CLI emits tool-call JSON but does not execute anything. The device benchmark rows are parity checks, not a population accuracy estimate — the README is unusually careful about not overclaiming, down to keeping Cactus's non-comparable published number separate.

Try it

pip install -e .                       # host CLI
cd needle-esp32s3
idf.py set-target esp32s3
idf.py -DNEEDLE_FAST_MATH=ON build
idf.py -p /dev/ttyUSB0 flash
./scripts/flash_weights.sh /dev/ttyUSB0   # 13.7 MB weights → flash partition
cd ..
mimimodel tools import examples/tools/demo.json --profile demo --activate
mimimodel run "Turn on the flashlight."
# [{"name":"turn_on_flashlight","arguments":{}}]

Flash the app before or with the weights — a SPIFFS partition overlapping the weight region will silently auto-format it. Repo: https://github.com/memovai/mimimodel