slvDev/esp32-ai

TL;DR: A 28.9M-parameter LLM generating text at 9.88 tok/s entirely on an ESP32-S3 (512KB SRAM), with no network. The trick is Gemma 3n's Per-Layer Embeddings repurposed for a microcontroller's memory hierarchy: 25M of the 28.9M parameters sit in a flash lookup table that is read rather than computed on, with only ~450 bytes pulled per token. The model itself is a TinyStories toy, but the memory-tiering writeup and the full reproduction pipeline (training, ablations, quantization, firmware) are the real deliverable.

What it is

An MIT-licensed demonstration of running a language model on a chip with three orders of magnitude less fast memory than a phone. The insight is not new inference kernels but a parameter-placement argument: the size ceiling for on-device models is usually set by the embedding tables, and if most of your parameters live in an embedding table, they don't need to be in RAM at all — an embedding lookup only touches a few rows per token. Google made this observation for phones with Gemma 3n's Per-Layer Embeddings; this project shows it survives the drop to a microcontroller, where "slow memory" means SPI flash.

How it fits

The ESP32-S3 has three memory tiers, and the model is split by access frequency per token, not by layer:

• SRAM (512KB, fast): activations and norm weights — touched many times per token.
• PSRAM (8MB, medium): the dense transformer core and output head — scanned once per position.
• Flash (16MB, slow): the 25M-parameter per-layer embedding table — about 6 rows (~450 bytes) read per token.

The consequence: model capacity is decoupled from RAM. At 4-bit quantization the whole thing is 14.9MB, and end-to-end speed is 9.88 tok/s (94.9 ms/token of compute — so roughly a millisecond per token of non-compute overhead, presumably flash reads and display).

What's actually in the repo

• Two deployable models: tinystories (story generation) and barista (espresso Q&A) — the board holds one at a time.
• Firmware for each in firmware/, reusable architecture code in src/ (repo is Python + C, Karpathy's llama2.c is the acknowledged reference for the C path).
• The full training/ablation/quantization pipeline in research/tinystories/ that reproduces the published numbers, and a RESULTS.md with on-chip measurements and ablations.
• Fetch/deploy scripts with SHA-256 pinning on model downloads.

That's a notably complete release for a hobby-scale project, and the README is refreshingly honest about limits: the model writes short coherent stories and nothing else. It won't follow instructions or know facts, and the author explicitly says the memory trick doesn't change that — the reasoning-capable part of the model is still tiny.

How to read it

The genuinely useful content is the memory-layout methodology, not the model. PLE was designed at Google for phone-class devices; this shows the access-frequency-based tiering degrades gracefully down to SPI-flash latencies, because embedding lookups are so sparse (~450B/token against a 25MB table). If you care about edge inference, the open question — flagged in the triage note too — is whether this transfers to models that do something useful: a capable model needs a bigger dense core, and the dense core is exactly the part that can't live in flash. PSRAM (8MB here) becomes the binding constraint, not flash. The RESULTS.md ablations are where to look for how the parameter budget was split between table and core.

Also worth noting: 4.2k stars in about two weeks says there's real appetite for serious extreme-edge LLM engineering writeups, and the reproducible pipeline makes this a decent starting scaffold if you want to try PLE-style tiering on beefier MCUs (or on phones/SBCs with slow storage).

Trying it

You need an ESP32-S3 board (512KB SRAM / 8MB PSRAM / 16MB flash) plus optionally a small display. Two commands, one that touches the network and one that touches the board:

scripts/fetch_model.sh tinystories   # download, verify (SHA-256 pinned), install into artifacts/
scripts/deploy.sh tinystories        # generate headers, run gates, compile, flash

Swap tinystories for barista for the Q&A model. Boot output and firmware details are in firmware/esp32_tinystories/README.md; method and measurements in RESULTS.md.