sbryngelson/ANEForge

TL;DR: ANEForge is a Python framework that compiles tensor graphs directly into Apple Neural Engine programs and dispatches them through the same private aned stack CoreML uses internally — bypassing CoreML entirely. That unlocks things Apple never exposed: training (forward, backward, Adam) on the ANE, the engine's fused-attention hardware layer, LLM decode with resident KV-cache, and cross-compilation for 28 M1–M5 targets. It's built on serious reverse engineering (documented in a companion arXiv writeup), works from an ordinary user process with no entitlements or SIP-disabling, and comes with a startlingly broad set of reproducible demos.

Why this is interesting

The ANE is a substantial chunk of silicon in every Apple device, and until now the only door to it was CoreML: inference-only, and the scheduler decides whether your model even lands on the engine or silently falls back to CPU/GPU. MLX and PyTorch-MPS never touch it — they're GPU frameworks. ANEForge is, as far as I know, the first public system to compile and dispatch ANE programs directly, and the first evidence anyone outside Apple has trained a network on this hardware (a CNN from scratch to 71% on CIFAR-10, on a chip shipped as inference-only).

The energy numbers are the practical hook: 8–16× less whole-package energy than the M-series GPU at fp16, with the ANE also faster on conv/encoder workloads (ResNet-18: 0.33 ms vs 2.03 ms on MPS).

How it works

You build a lazy tensor graph in Python from 58 fused operators (conv, matmul/einsum, norms, softmax, attention, geometry) plus 19 "bridge" ops the public toolchain never emits — including af.sdpa, which drives the engine's native fused-attention layer that Apple's own compiler decomposes. af.compile lowers the graph into one ANE program with int8/int4-LUT/fp16 weight compression, and a small e5rt shim (compiled from source on your Mac, linking Apple frameworks) dispatches it. Dispatch floor is ~70 µs; KV-cache and optimizer state stay resident on the engine across calls via buffer aliasing. Programs past the ~2 GB single-program ceiling get auto-segmented, which is how 8B–27B LLMs fit.

What's actually there and the evidence

This is unusually well-substantiated for a reverse-engineering project:

• MLPerf: the reference ResNet-50 runs pure-ANE and passes the upstream MLCommons submission_checker (v5.1, all three edge scenarios VALID) at reference accuracy (fp16 76.44% = fp32).
• Fidelity: ResNet-18, ViT-B/16, MiniLM at cosine 1.0000 vs reference; Whisper-base.en (both towers on ANE) matches HF's greedy transcript; GPT-2 medium matches HF fp32 greedy 16/16 at ~140 tok/s.
• LLMs: Qwen3-0.6B decodes at ~75 tok/s, 8B at ~7.5, exact speculative decoding gives 2.28× (the README's nice observation: verify(K) ≈ verify(1) because ANE decode is latency-bound), MoE from GGUF, and a 27B DeltaNet+attention hybrid — all pure-ANE.
• Training: CIFAR-10 CNN to 71%, a neural-cellular-automaton trained with gradient checkpointing on-engine, ONNX transfer-learning example.
• Demos of spectral Navier–Stokes and Gray–Scott reaction-diffusion running entirely on the engine, a correctness corpus (tests/run_corpus.py), per-op × per-device coverage tables, docs on readthedocs, two arXiv papers, and a community roofline benchmark feeding an HF leaderboard.

Caveats

The whole thing rests on private, undocumented framework symbols; any macOS update could break it, and Apple could plausibly close the door. Verified only on M1 Max and M5 Pro; the roofline campaign exists precisely because fp16 numeric cliffs differ across silicon. Big MoE decode is weight-bandwidth-bound (~2 tok/s at 30B, int8), and the GPU still edges the ANE on ViT-B/16 latency. Stable Diffusion runs per-component with 1.5–4.4% relative error — fine but not the cosine-1.0 story of the encoders. MIT-licensed, but read the NOTICE: nothing here is an API contract from Apple. Treat it as excellent research infrastructure, not production.

Try it

Apple Silicon Mac, macOS 14+, Xcode CLT, Python 3.10+:

pip install "aneforge[models]"
python -m aneforge.build   # ahead-of-time compile of the dispatch shim

import aneforge as af
txt = af.load_gpt2("gpt2").generate_text("The Neural Engine is", 20)
asr = af.load_whisper("openai/whisper-base.en").transcribe(audio)

Or clone the repo and run python examples/demo.py to watch a transformer train from scratch on the engine. Docs at aneforge.readthedocs.io; the reverse-engineering writeup (worth reading on its own) is at ane-guide.readthedocs.io.