suanmiao/fly-self-driving

TL;DR: Someone took the measured wiring diagram of a male fruit fly — 165,122 neurons, 25.6M synapses from the Janelia MaleCNS connectome — froze the adjacency, learned only per-synapse gains and per-neuron leaks, and trained it with imitation + DAgger to steer a simulated car from a 64×32 windscreen. It completes 58/60 held-out streets with traffic, collision-free, beating a shuffled-wiring control (16/20) and a small MLP (18/20). It's a real, if single-seed, closed-loop test of whether biological topology matters for visual control — and the biggest lift came from a one-line change: never resetting the recurrent state.

What it actually is

This is a follow-on to Mark Unthank's flyhard (connectome-as-recurrent-network recipe), extending it from a steering-wheel pilot to closed-loop driving on procedurally generated two-lane streets with parked cars, oncoming traffic, and a slow car to overtake. Every traced neuron is a signed-rate unit; every measured synapse is an edge; the adjacency never changes. The learnable parameters are exactly one gain per synapse plus one leak per neuron (25.7M total), all confined to edges the fly actually has.

The interfaces are deliberately dumb: a random, frozen projection from pixels onto the 4,114 optic-lobe sensory neurons, and a random, frozen readout from the 708 ventral-cord motor neurons to a steering angle. Nothing engineered can carry the signal alone — the linear baseline (1,153 params, pixels→steering) tops out at 12/20 with traffic.

Training and evidence

Recipe: a pure-pursuit expert (reads true state) drives 60 training streets with steering noise; the connectome is fit to its commands via truncated BPTT over 16-decision windows (1,200 updates), then three DAgger rounds (3×500 updates) where the brain drives and the expert relabels. ~35 minutes on one H100; the whole project cost under $25 of GPU time.

Closed-loop eval on three independent sets of 20 unseen streets:

Two things stand out. First, behavior cloning alone gets the connectome only to 12/20 — DAgger does serious work here, as it should for a recurrent policy with compounding errors. Second, the shuffled control (same degree of freedom count, random rewiring, same recipe) lands at 16/20. That gap — roughly 19.3 vs 16 — is the entire claim that topology matters, and it rests on one training seed per condition. It's suggestive, not conclusive. The persistent-state ablation (3/20 → 17/20 on the earlier task) is actually the more dramatic and interesting number: it argues that the connectome's usefulness shows up only when you treat it as a dynamical system, not a feedforward feature extractor.

Also worth noting: the random pixel-to-optic-lobe map means this says nothing about fly vision — retinotopy, motion detection circuits, none of it is used as the fly uses it. What's tested is whether the wiring graph is a good recurrent substrate under gain modulation, which connects loosely to the reservoir-computing and lottery-ticket framings of connectome networks.

What's in the repo, and what isn't

Included: full training code for both tasks (flat-road pilot and street), the DAgger trainer, all baselines, drop-in flyhard patches (Apple-silicon MPS kernel, a 2.5× faster CUDA backward that caches the transposed graph), a Blender replay renderer, recorded episodes, and docs with controls and caveats. Not included: the ~1 GB connectome (fetched by flyhard's verified scripts), ~600 MB checkpoints, and the videos (on the site). MIT license. Honest caveats in the README: no braking/speed control on the street task yet, single seed per condition, and it was built "by agents in a Kylon workspace" — worth keeping in mind when reading docs/method.md.

Try it

Needs a sibling flyhard checkout; runs on one H100 (~35 min) or Apple silicon (one GPU job at a time):

git clone https://github.com/MarkUnthank/flyhard ../flyhard
cd ../flyhard && pip install -e . && python scripts/acquire_connectome.py && python scripts/prepare_graph.py && cd -

git clone https://github.com/suanmiao/fly-self-driving && cd fly-self-driving
pip install -r requirements.txt
cp flyhard-patches/connectome*.py ../flyhard/src/flyhard/

cd tasks/street
python train_street.py --model connectome --graph-mode measured \
  --window 16 --batch 4 --steps 1200 --lr 0.04 --out runs/street-measured

Videos, failures, and the interactive explainer are at fly-self-driving.kylon.app. If you want to poke at the core claim, rerun --graph-mode shuffled with a few seeds and see whether the gap holds.