Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

TL;DR: Take a big frozen behavior-cloned robot policy, bolt on a ~1B-parameter Q-function, and let the Q-function do two jobs: pick the best of N sampled action chunks at inference, and absorb every deployment rollout — successes and failures — into critic-only fine-tuning. The policy's weights are never touched. This exploits a simple asymmetry: BC can only learn from successes, but a value function can learn from anything. Five iterations of this loop on a real bimanual robot take cup-stacking from 40% to 90% and wallet-card-insertion from 25% to 80%, with no new human demonstrations, while filtered SFT on the same rollout budget stalls at 55% and 30%. If the recipe generalizes, it's the cheapest credible path yet to VLA policies that get better on the job.

The demo ceiling

Modern robot manipulation runs on behavior cloning: collect teleoperated demonstrations, train a large vision-language-action (VLA) model to imitate them. It works remarkably well, but the recipe has a structural flaw baked into its objective. Imitation learning maximizes the likelihood of demonstrated actions — so the only data it can consume is data you'd want to imitate. When a deployed policy fumbles a cup, that failure trajectory is worthless to BC. Worse than worthless: training on it would teach the policy to fumble. The only way to improve is to pay humans for more demos.

The field's dominant response is RL fine-tuning of the policy itself. But doing on-policy RL with sparse rewards on a multi-billion-parameter model is expensive, unstable, and — most insidiously — risks corrupting the BC prior that made the policy competent in the first place. The other classical option, value-based planning (train a critic, use it to re-rank actions at inference), had never been demonstrated at VLA scale, where action spaces are high-dimensional flow-matching chunk generators and naive sampled search is intractable.

The asymmetry that makes it work

The paper's core observation is almost embarrassingly simple once stated: the actor and the critic have different data diets.

A BC policy answers "what would the expert do here?" — a question only success data can answer. A Q-function answers "how good is this action in this state?" — a question any trajectory answers. A failed rollout is a perfectly valid training example for a critic: it says these actions in these states led to zero reward. The two components can therefore be decoupled: keep the expensive, hard-to-train, easy-to-break BC policy frozen forever, and route all deployment experience — the signal BC must throw away — into a much smaller off-policy Q-function that's cheap and safe to update.

The Q-function then earns its keep at inference time. Instead of executing one BC sample, draw N candidate action chunks from the frozen policy, score each with Q_\phi, and execute a softmax-weighted average:

where \mathbf{o}_t is the camera observation, \boldsymbol{\ell} the language instruction, \mathbf{a}^{(n)} the n-th candidate action chunk, and \lambda a temperature. No auxiliary actor, no gradient steps on the policy, no iterative search — one batched forward pass through the critic.

The mechanism, concretely

Three design choices make this actually work; each addresses a known failure mode.

1. The critic scores action chunks, not single actions, with categorical outputs. Modern VLAs emit chunks \mathbf{a}_{t:t+H} of H{=}32 actions per planning step. The Q-function treats the whole chunk as one super-action ("Q-chunking"), which shrinks the effective bootstrapping horizon by a factor of H — critical under sparse terminal rewards, where long-horizon TD error compounds. On top of that, instead of scalar Q-regression (notoriously unstable with sparse, bimodal returns), the head outputs B{=}101 logits over a bin grid on [0,1] (HL-Gauss regression); the scalar Q is the expectation over softmaxed bins. The Bellman target uses the next chunk from the buffer as the bootstrap action rather than querying the planner — cheaper, and for demonstration data this is exactly what \pi^{\text{BC}} would produce, so the target is an unbiased estimate of Q^{\pi^{\text{BC}}}. A failed rollout is simply an all-zero-reward trajectory whose terminal chunk doesn't bootstrap, pinning its target to zero. The critic has its own DinoV2 vision and T5 language encoders, fully parameter-disjoint from the policy — a firewall, so critic updates can never corrupt the policy.

2. Candidates must live on the BC manifold. This is the paper's most instructive negative result. They first tried classic MPPI: perturb the BC's chunk with Gaussian noise, score, iterate. It hurt — 89.5% on LIBERO-10 vs. 90.0% for the unguided policy. Why? Jagged noise produces chunks the critic has never seen during training, and out-of-support inputs get spuriously high Q-values — the standard Q-overestimation trap. Smoothing the noise temporally recovered a +3pp gain, but the elegant fix is to skip perturbation entirely: flow-matching heads are natively multi-modal, so N raw draws (using a truncated 3-step denoising pass, which is both faster and more diverse than the full 10-step pass) already cover distinct plausible action modes, every one inside the critic's training support. Same 93.0% success as smoothed MPPI, 1.7× faster, three fewer hyperparameters. The Q-function is only ever asked to rank options the BC already considers plausible — it never has to extrapolate.

3. The loop. Each iteration: run the planner for M episodes per task (100 in sim, 20 on hardware), append everything to the replay buffer, take S{=}200 gradient steps on the critic only (minibatches drawn half from the original demos, half from online rollouts), update the EMA target network, repeat. Exploration comes for free from the multi-modality of the proposals: any behavior the BC head produces with non-negligible probability can be selected, amplified in the buffer, and rewarded by the critic.

A practical point that makes this deployable: the critic's encoders run once per planning step (~25 ms) regardless of N; only the ~500M decoder scales with candidates (~2–3 ms each). A full planning step with N{=}32 takes 400 ms on bimanual RoboTwin — 1.6× faster than a single 10-step BC inference, comfortably inside the 960 ms replan budget. Value-guided planning here is not a latency tax; it's roughly free.

The evidence

Simulation first. Across four LIBERO suites and 47 RoboTwin bimanual tasks, offline Q-weighted selection alone buys a modest +1.3pp on average — the critic, trained only on the same successful demos as the policy, hasn't seen failures yet and can't discriminate much. Ten iterations of self-improvement is where the value shows up: every benchmark rises, mean success 92.1% → 97.6%. On suites already at ceiling (LIBERO-Object at 100%), the loop instead shortens successful episodes (139 → 120 steps) — the critic learns to prefer faster paths to reward, a nice sanity check that it's optimizing return, not just re-ranking noise.

The baseline comparison on LIBERO-10 (Figure 2a of the paper) is the sharpest evidence that the specific recipe matters. Under an identical online rollout budget: Best-of-N (same loop, argmax instead of weighted average) plateaus at 95%; filtered SFT (re-imitate only successful rollouts) plateaus at 93.5%; IBRL collapses; DSRL oscillates between 69% and 91%; DAWR hovers below the frozen BC. Q-Planning reaches 99% and does so stably. The filtered-SFT gap is the interpretable one: it's the direct measurement of what the failure signal is worth, since SFT sees the same successes and can only discard the failures.

The real-robot results are the headline, and rightly so. Two contact-rich bimanual tasks on 6-DoF YAM arms, 100 base demos each, BC frozen, no teleoperation during the loop — human involvement limited to scene resets and a per-episode success label:

Stack-cups: 40% (raw BC) → 65% (Q-selection alone) → 90% after five iterations. Insert-wallet: 25% → 40% → 80%. SFT on successful rollouts, under the same budget, actually degrades — stalling at 55% and 30%. On hardware, where BC baselines are far from ceiling, the gains are enormous and the failure signal is demonstrably the ingredient doing the work.

What changes, and what to doubt

If this holds up, the practical implication is significant: self-improvement cost scales with the critic's size, not the policy's. As VLA backbones head toward 10B+ parameters, "freeze the policy, train a ~1B critic on deployment data" is a dramatically more tractable proposition than PPO on the full model — and it's plug-and-play with any released checkpoint, since the critic's encoders are disjoint from the policy's. The framing also isn't imitation-specific: any proposal distribution producing diverse plausible chunks (world-model rollouts, offline-RL policies) could stand in for the BC head. This feels like a recipe with adoption potential, in the same way test-time best-of-N with a reward model became standard practice in LLMs — here it's best-of-N with a learnable reward model that keeps improving.

Reasons for skepticism, plainly:

• Hard exploration ceiling. The planner can only select what the BC head can produce with non-negligible probability. Tasks where the base policy generates zero successful chunks are unreachable — this is selection and amplification, not skill discovery. Insert-wallet at a 25% baseline is respectable, but "BC succeeds sometimes" is a real precondition.
• Reward supervision isn't free. The loop needs a per-episode success label — the environment bit in sim, a human label on hardware. That's far cheaper than teleoperation, but it's not autonomy; scaling to open-ended deployment needs learned or language-conditioned success detectors.
• Real-robot evaluation is thin. Two tasks, 20 episodes per iteration, and success rates are measured on the collection episodes themselves rather than a held-out evaluation. Confidence intervals at n{=}20 are wide; the 25 → 80 trend is convincing, the exact numbers less so.
• Simulation suites are near ceiling, so most sim gains are small in absolute terms (93 → 99 on LIBERO-10); the burden of proof falls almost entirely on the two real tasks. One BC backbone (FastWAM) was tested.
• "Small" critic is relative. A ~1B-parameter Q-function trained on 4× H200s is small next to a VLA, not small in absolute terms.
• There's concurrent BC-to-Q work (cited as [43]) sharing the frozen-BC + off-policy-critic ingredients; the differentiators here are the actor-free planner and the VLA-scale, real-hardware demonstration.

Where to spend your reading time

Section 4.2 and Table 2 (why vanilla MPPI fails and on-manifold sampling fixes it) is the most transferable insight — a clean, mechanistic story about Q-overestimation and proposal support that applies well beyond robotics. Section 3.2's training details (Q-chunking, HL-Gauss, the one-step-shifted bootstrap justified in Appendix C) are the answer to "why doesn't the critic blow up on sparse rewards," and Appendix E's latency profile is worth a skim if you care about deploying anything like this in a real control loop.