MINERVA: How Small Can a Manipulation Policy Be and Still Solve LIBERO?

TL;DR: A 0.54-million-parameter policy — no language model, no pretrained vision, task selection via a 40-entry lookup table — scores 95.1% on the standard LIBERO manipulation benchmark, within 2.4 points of the 4.1B-parameter π0.5 while running 1,400× faster on a laptop CPU. The paper's real contribution isn't the tiny model; it's the measurement. It shows that standard LIBERO demands about half a million parameters of task-indexed visuomotor memorization and nothing more, that a single training seed moves results by ±1 point (swallowing most reported ablation gains), that flow matching buys nothing over one-pass L1 regression on this benchmark, and — via a clever permutation probe — that "instruction following" on LIBERO is causally just task selection from memory.

The question nobody had bothered to measure

LIBERO is the default benchmark for vision-language-action (VLA) models: four suites of 10 manipulation tasks each (Spatial, Object, Goal, Long-horizon), with demonstrations for imitation learning. The leaderboard is dominated by models that bolt action heads onto pretrained vision-language backbones — OpenVLA at 7B parameters, π0 and π0.5 at 3–4B — all reporting success in the 95–98% band.

Here's the awkward detail: standard LIBERO trains and evaluates on the same 40 tasks in the same scenes with the same instructions. Prior work (LIBERO-PRO) had already noted that perturbing objects or layouts collapses these models toward 0%, suggesting the benchmark is largely a memorization test where the language instruction merely disambiguates which of 40 memorized behaviors to run. But nobody had asked the follow-up question: if that's what the benchmark measures, how much model capacity does it actually demand?

This is not just a gotcha. For deployment, it's the question. Once a robot's task set is fixed, generalization capacity you paid for but don't use is pure cost — VRAM, latency, power. If you knew the minimal sufficient capacity for your task set, you could build (or distill) a policy at that floor and run it on an embedded CPU. The authors call this the empirical capacity floor: the smallest model in a family that still hits target success under a fixed protocol. Their move is to take the memorization reading of LIBERO completely seriously, build a policy family containing nothing the benchmark doesn't exercise, and shrink it until it breaks.

The instrument: strip everything the benchmark can't detect

If the instruction's only functional role is to pick one of 40 known tasks, you don't need a language encoder — you need a 40-way index. So MINERVA replaces the entire language pathway with a learned embedding table of 40 entries and spends the whole parameter budget on the two things standard LIBERO does exercise: visual state estimation and closed-loop action generation.

Concretely: both camera views (downsampled to 128²) pass through one shared four-stage depthwise-separable CNN, FiLM-modulated by the task embedding, ending in spatial-softmax keypoint extraction. Proprioception goes through a two-layer MLP. Everything is concatenated over two observation steps into a conditioning vector. The action head generates a 16-step action chunk — trained either with conditional flow matching or as plain L1 regression (learned queries in, chunk out, one forward pass). Smaller models are distilled from a larger teacher's velocity field. At inference the policy replans every control step and averages overlapping chunk predictions (ACT-style temporal ensembling).

Then they scale the total budget from 9.7M down to 0.09M and watch where it breaks.

The scaling curve: saturation at 1M, collapse below 0.25M

The curve has a clean shape: from 1M to 10M parameters, average success moves only 0.7 points — capacity stops being the binding constraint around one million parameters. Below 0.54M it breaks sharply (88.6% at 0.24M, 68.3% at 0.09M), and the collapse is structured: at 0.24M the model still solves 95.2% of Spatial but only 73.0% of Long. Short-horizon pick-and-place survives in models far too small to chain subgoals. If anything on standard LIBERO genuinely prices capacity, it's the long-horizon suite.

The shipped 0.54M model (with the regression head) scores 95.75%; the 0.99M model reaches 96.75%, within 0.75 points of π0.5's reported 97.5%. On the three short-horizon suites, the 1M model matches or exceeds π0.5; the entire remaining gap is long-horizon (92.8 vs. 96.0).

The instrument calibrates itself: the ±1-point seed band

Because these models train in ~2 hours on one GPU, the authors could afford something the VLA literature mostly can't: retraining key ablations under three seeds, each evaluated over 2,000 rollouts. The result is sobering. The baseline 1M configuration spans 94.60–96.75% across three training seeds — a ±1-point band on the 4-suite average, with the long suite alone spanning 87.6–92.8%. That band is comparable to many single-run ablation deltas reported on this benchmark. Read that back onto the leaderboard: single-run gains of a point or less on LIBERO are indistinguishable from seed noise.

Only two design choices survive the band:

Chunk length (−3.16 points at chunk 8, −2.20 at chunk 32), and the failure mode is asymmetric in an interpretable way. Short chunks break the Goal suite (86–89% vs. ~97%): goal tasks pose different targets in one scene, so the policy must commit to a branch, and short chunks let it dither between modes. Long chunks break the Long suite (80–82%): multi-stage tasks need reactions at subgoal boundaries, and long open-loop chunks can't provide them. Chunk 16 sacrifices neither.

Vision allocation (−1.41 ± seed-replicated): starving the encoder from 0.49M to 0.13M at constant total budget costs points that no amount of action-head capacity recovers, with the damage concentrated on Long (−4.6). A six-point sweep redistributing a fixed ~1M budget is one-sided — any split giving vision 50–80% lands in a flat 96.3–96.8 band. The design rule for tiny policies: shrink the action head, not the eyes.

And one widely assumed choice does not survive: flow matching is indistinguishable from direct L1 regression (+0.34 for regression, well inside the band, n=3). Amusingly, the single-seed comparison had regression 1.70 points worse — the sign flipped under seed averaging, a small object lesson in why the band matters. Since regression needs one forward pass instead of ten Euler steps, it's up to 3.8× faster (2.2 ms vs. 8.2 ms per chunk on GPU), so that's what they ship. This is benchmark-specific — LIBERO demonstrations may simply not be multimodal enough to need a generative head — but it's a clean data point against the reflexive "diffusion/flow head by default" convention.

The inference-side findings echo it: replan every step, average overlapping chunks. Temporal ensembling (96.3%) beats BID-style candidate selection (95.0%) and plain chunk execution (94.1%); RTC-style soft inpainting hurts (91.8%). Mode-seeking sampling (initial-noise temperature 0.85) adds up to 2.3 points, but only on sub-1M models where the velocity field is noisiest.

The permutation probe: making memorization causal

The headline "task IDs replace language at no cost" is correlational. The probe makes it causal. Take the trained 1M checkpoint, freeze all weights, and rewrite only the task-ID mapping — feed the policy the wrong index for each task and see what it does.

Rotating each task's ID within its suite collapses success from 96.75% to 6.5%. Forcing every task to its suite's task-0 ID gives 16.2% — and Spatial, Goal, and Long land almost exactly at the 1-in-10 rate you'd predict if the policy simply executes whichever memorized task the ID names, regardless of what the scene shows. Only the Object suite retains a 22–36% residue, because its scenes are visually distinctive enough to partially override the conditioning.

So the conditioning channel is doing task selection, not task understanding — the policy is essentially a jukebox with 40 records, and the ID is the button. What standard LIBERO's "instruction following" measures is whether the button works. The triage note asks the right follow-up: do billion-parameter VLAs pass an analogous probe (paraphrased or swapped instructions)? This paper doesn't run it on them, but LIBERO-PRO's near-0% under instruction perturbation strongly suggests they wouldn't either.

What the 95% headline omits: LIBERO-Plus

The authors then price the memorization explicitly, evaluating on LIBERO-Plus (10,030 perturbed variants across 7 factors):

Three things to notice. Language rephrasing costs essentially nothing (variants are visually identical to base scenes) — consistent with the probe. Going from 1M to 5M buys ~10 points, almost entirely on semantic/geometric factors (new objects: 62 → 96) — so robustness is partly capacity-limited, just not below 1M. And photometric robustness is near zero at every tested scale (lighting ≤ 6.5%): a from-scratch CNN with no augmentation and no pretrained perception has memorized appearances, and no parameter count in this range fixes it. This is the honest counterpart to the headline — the paper is explicitly not claiming tiny policies generalize. It's quantifying that standard LIBERO never asked them to.

Deployment as a side effect

The 0.54M regression model produces an action chunk in ~5 ms on eight laptop CPU threads (0.03 GB memory), vs. 1.0 s for SmolVLA (450M) and 12.8 s for π0.5 on the same hardware — 113× and 1,400× respectively. That makes the success-maximizing protocol (replan every step) affordable at ~100 Hz on an embedded CPU, where for the VLA baselines it means ~1 Hz and 0.08 Hz. The task-ID interface is arguably a feature for system integration: a planner that already knows which skill to run selects it by index, no language model in the control loop. Code and checkpoints are open-sourced.

What to make of it

If this holds, three things change. First, LIBERO scores above ~97% should stop being read as evidence about VLA capability; the benchmark is satisfiable by half a megaparameter of task-indexed memorization, so headroom claims there measure something else (or nothing — remember the seed band). Perturbation-based protocols like LIBERO-Plus/PRO become the meaningful evaluations. Second, the "capacity floor" framing is a genuinely useful research direction: measure the minimal sufficient capacity of a fixed task set, then build or distill to it. Their own distillation result is intriguing — the distilled 4.89M model beats its own 7.12M teacher. Third, the seed-band and flow-vs-regression findings are methodological warnings with reach beyond this paper: much of the imitation-learning ablation literature on this benchmark is likely reporting noise, and generative action heads may be unnecessary for unimodal demonstration data.

Reasons for skepticism. The headline models, the LIBERO-90 run (94.6% over 89 tasks at 0.995M — the recipe survives 2.25× the task count), and most sweep points are single runs inside the very ±1-point band the paper itself establishes; only five key ablations get three seeds. Everything rests on one simulator and one evaluation seed. The π0.5 comparison uses that model's reported number at 10 episodes/task vs. MINERVA's 50 — a reference point, not a protocol-matched comparison. The "capacity floor" is a floor of this architecture family under this protocol, not an information-theoretic bound; a different family might go lower or need more. No real-robot validation yet, though the latency numbers make that a plausible next step. And the photometric fragility is probably fixable with mundane augmentation, which would slightly muddy the clean "memorization" reading.

Where to spend your time in the paper: Section V-C (the seed-band calibration and what survives it) and V-G (the permutation probe) are the intellectually load-bearing parts; V-D's budget-allocation sweep ("the eyes, not the head") is the most practically transferable design rule; Table IV is the necessary corrective to the headline.