murphyk/neuronbench

TL;DR: NeuronBench is a small, sharp benchmark for mechanism discovery via experiment design: an agent gets a black-box Hodgkin–Huxley neuron with one hidden extra membrane current, a hard budget of current-clamp protocols and channel blockers, and is scored on how well it forecasts interventions it never ran. The clever bit: five of six worlds have mechanisms that are deliberately invisible to every textbook probe and described only as "an unidentified membrane current," so an LLM can't pattern-match its way to the answer — you must actually simulate candidate mechanisms and design the discriminating experiment. From murphyk (yes, that one), MIT-licensed, with a paper attached.

What it actually is

Each world is a plain Na⁺/K⁺/leak spiker plus one hidden current — Iₕ sag, T-type rebound burst, slow Na⁺ fatigue, D-type K⁺ delay, low-threshold rebound block, or (as a recallable control) the M-current. The agent is never told the candidate set, or even that there are exactly two hypotheses. It gets an opaque prior, a reference plain-cell model, a 9-protocol design pool (4 textbook steps + 5 non-standard conditioning/paired-pulse protocols, exactly one of which reveals each world's mechanism), and four blockers (TTX/TEA/Cd/XE991) as do(g=0) interventions.

Two regimes:

• Deterministic — noiseless traces, each protocol usable once; a budget of b buys b distinct experiments.
• Stochastic — Fox–Lu channel noise with finite channel count N, observations are noisy (σ = 2 mV), sub-sampled (~4 ms) voltage traces. The likelihood p(trace | mechanism) is intractable and must be estimated by simulation. Budget becomes (protocol, repeats): repeating averages noise down ~1/√r, so where to spend repeats is part of the design problem. As N → ∞ you recover the deterministic case.

The scored task is counterfactual forecasting on held-out interventions: predicted spike counts (spike_forecast_mse, the headline, comparable across LLM and model-based agents) and predicted voltage traces (feature_forecast_mse, scored against internal features — spike counts plus sub-threshold shape like V-min, sag, run-down — which agents never see). An LLM that can't emit a trajectory submits empty traces and eats the penalty rather than being excused.

Why it's interesting

This is a clean, executable instantiation of Bayesian optimal experiment design meets agentic AI, in a domain (single-cell electrophysiology) where the ground-truth generative model is fully known. The anti-memorization design is the standout: because the novel mechanisms fire identically to a plain cell under standard steps and blockers, the only path to a good forecast is simulating candidate mechanisms against the free forward model (world.simulate costs no budget) and finding the one non-textbook conditioning protocol that discriminates. That directly tests whether an agent can do model-based hypothesis-driven science rather than retrieve textbook signatures — a live question for LLM-scientist claims.

The design-boundary discipline is also unusually good: the package ships no likelihoods, particle filters, SMC, or VOI machinery, explicitly to avoid leaking "the right way to score." The only bundled agent is a pure in-context LLM baseline (injectable client, OpenRouter example).

What's actually there

• Python package: 6-world registry, deterministic + Fox–Lu stochastic simulators, protocol/blocker/budget API, evaluator, LLM baseline agent.
• A browser-playable demo (app/patch.html), a self-contained port claimed to be validated against the Python model "to the spike."
• A companion paper (arXiv 2608.09696) introducing an LLM-assisted Bayesian experiment-design agent.
• No numbers in the README — no baseline scores, no LLM-vs-model-based comparison. Evidence of difficulty lives in the paper, not here.

Caveats

Six hand-designed worlds is small; the shared 9-protocol pool means a solver could in principle overfit the pool structure (one revealing protocol per world) rather than do genuine inference — the selection note flags exactly this. Truth being "plain cell + exactly one extra current" is a strong structural prior even if unstated. The repo is days old (42 stars), pip install neuronbench is marked "once released," so install from checkout. And note the framing quirk: the README says agents aren't told there are two hypotheses, yet the worlds are literally two-hypothesis problems — how much the open-endedness matters in practice depends on the solver.

Try it

git clone https://github.com/murphyk/neuronbench && cd neuronbench
pip install -e ".[llm]"
export OPENROUTER_API_KEY=...
python examples/run_baseline.py --world ca_rebound --stochastic --budget 8

Or just open app/patch.html in a browser and play the deterministic version yourself — it's a good way to feel why the textbook probes fail.