Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience

TL;DR: SCORE improves real-world robot policies using only simulated RL — no new real-world data, no fine-tuning of the base policy, no distillation — by constraining the RL agent to only produce actions that a real-data flow-matching policy could already produce. The mechanism is elegantly simple: freeze the flow policy and let RL pick the input noise. Across eight dexterous multi-fingered manipulation tasks on real hardware, this lifts average success from 37.8% to 89.9% (best baseline: 59.5%) while making the policy 36.8% faster. The interesting part is the framing: the constraint is on the support of the base policy, not its distribution, and the paper makes a crisp argument for why that distinction is the whole ballgame.

The problem: simulation is a liar, but a useful one

Imitation-learned robot policies are rarely good enough. Even at 90% success they're often slow, imprecise, and fall apart under perturbations. The natural fix is RL — let the robot try, fail, and improve — but real-world RL on a 23-DoF Franka-plus-LEAP-hand system is expensive, slow, and occasionally destroys hardware.

Simulation offers millions of cheap parallel rollouts, privileged state, and safe failures. But simulators lie about exactly the things that matter for dexterous manipulation: contact dynamics, friction, deformation, low-level control. Unconstrained RL in simulation is an optimizer with a jailbroken world model — it will find and exploit every discrepancy. The paper documents this vividly: an unconstrained flow-RL baseline (FPO) learns to slam its hand into the table to trap a soccer ball, exploiting simulator-specific contact physics. High reward in sim, unsafe and useless on hardware. Their FPO baseline actually drops real-world success from 37.8% to 17.8% while doing great in simulation.

The standard fix is regularization: penalize the improved policy for drifting from the base policy, typically with a BC loss (forward-KL), a discriminator (Jensen-Shannon), or a bounded residual (Wasserstein-1). But this creates a tradeoff with no good setting of the knob. Tight constraint: you're stuck reproducing the base policy's slow, imprecise behaviors, including its failure modes. Loose constraint: you drift into simulator-exploiting territory. The paper's Figure 6 sweeps the BC coefficient for the RialTo baseline across four orders of magnitude and finds exactly this: every setting either collapses, reward-hacks, or under-improves, with the best middle ground (60% real success on Cube Pinch) still far below SCORE's 100%.

The key idea: constrain the support, not the distribution

Here's the reframe. A distributional constraint says: your action distribution must stay close to the base policy's distribution. A support constraint says: you may only take actions the base policy could have taken — but you can reweight them however you like.

The difference matters because imitation-learned policies are multimodal. Your demonstration data contains fast trajectories and slow ones, precise grasps and sloppy ones, direct paths and roundabout ones. A distributional constraint forces you to preserve this whole mixture, suboptimal modes included — you can only nudge it by \epsilon. A support constraint lets RL put all its probability mass on the fastest, most precise mode, and zero mass on everything else, while still guaranteeing that every action executed is one the real-world data demonstrated to be physically realizable.

Distributional (KL ≤ ε) both modes preserved: slow behavior survives

Support (SCORE) all mass on the best mode, zero on the rest — still in support

The paper backs this with a clean formal argument (Appendix E, worth reading). Distributional constraints provably cannot recover the optimum even when it's inside the base policy's support: with a Gaussian base policy and reward concentrated at a^* > \epsilon, Wasserstein-1 duality shows no \epsilon-constrained policy can put its mass at a^*, even though a^* has nonzero base density. Meanwhile, under two assumptions — (1) simulation preserves the ranking of real-world-realizable policies, and (2) any policy absolutely continuous with respect to a realizable policy is itself realizable — support-constrained sim RL provably recovers the best real-world policy within the base policy's support.

Those assumptions are doing real work, and it's worth being explicit about what they buy. The claim is not that simulation is accurate. It's the weaker claim that for behaviors already demonstrated in the real world, sim performance orderings transfer. Unrealizable behaviors (table-slamming) can score arbitrarily high in sim — but the support constraint fences them off entirely, so the ranking assumption only ever needs to hold on the safe set.

The mechanism: RL over the noise of a frozen flow policy

How do you actually implement a support constraint? Prior offline-RL approaches use discriminators or conservative Q-modifications — fiddly, unstable machinery. SCORE's implementation is much cleaner, and it falls out of the structure of generative policies.

The base policy is a conditional flow-matching model: given observation o (point cloud, proprioception, action history), it maps a Gaussian noise sample z \sim \mathcal{N}(0, I) through a learned velocity field to an action chunk a = \pi_{\text{base}}(o, z). Different noise samples land in different modes of the demonstrated behavior. So the set of all actions the base policy can produce for observation o is exactly \mathcal{A}_{\text{base}}(o) = \{\pi_{\text{base}}(o, z) : z \in \mathcal{Z}\}.

Now the trick: freeze the flow model, and train a small RL policy that chooses z. The steering policy \pi_{\text{steer}}(z \mid o) can only navigate within the manifold of behaviors the base policy already covers — the support constraint is architectural, not a soft penalty with a tuning knob. This is "flow steering," adapted from methods (DSRL, RFS) originally built for real-world online RL, here deployed entirely in simulation.

The full method (SCORE, using RFS) adds one refinement: alongside z, the steering policy outputs a small residual a_r, so the executed action is a = \pi_{\text{base}}(o, z) + a_r. This is a soft support constraint — a \delta-neighborhood around the support — which supplies the millimeter-scale precision dexterous manipulation demands without permitting large drift. The pure-latent variant (SCORE-DSRL) is a hard constraint and does nearly as well (79.2% vs 89.9%).

The training setup is deliberately boring, which is a feature. PPO with sparse rewards — the reward is the evaluation success condition, one set of hyperparameters across all eight tasks, no dense shaping. Environments are built from real-world scans (iPhone-scanned objects into IsaacLab), with domain randomization over mass, friction, scale, and poses. One clever engineering choice makes the parallelism work: instead of rendering depth images (which caps you at ~512 environments/GPU), they sample points from meshes and transform them with ground-truth poses, preserving the point-cloud observation structure at 4096 environments/GPU. An asymmetric actor-critic gives the critic privileged simulator state (object poses) while the actor sees only deployment observations — so the trained actor deploys directly, zero distillation.

The evidence

Eight contact-rich tasks on a Franka FR3 + LEAP hand: grasping, pouring, pushing, lightbulb screwing, thin-card picking, plate placement. Base policies trained on 90–170 teleoperated demos per task.

The pattern in this chart is the paper's argument in miniature. Residual-RL — a Wasserstein-style distributional constraint — does fine when the base policy already picked a decent strategy and just needs local polish (Credit Card: 10%→80%). But on Ball Pour, where the base policy's dominant mode is wrong, bounded corrections can't escape it: 11%→0%. SCORE reaches 89% on the same task by selecting a different mode from the prior. Speed tells the same story: Residual-RL is 19% faster than base on average, SCORE 36.8% faster — steering to the fast mode beats perturbing the slow one.

Also telling: simulation success and real success decorrelate exactly as the theory predicts. FPO hits 61.8% average in sim and 17.8% in reality. SCORE's sim numbers (76.2%) are lower than its real numbers (89.9%) because the sim uses broader randomization — the constrained policy doesn't need to win in sim, it needs to transfer.

The coverage ablations are, to me, the most thought-provoking part of the paper. Since SCORE can only amplify what's in the support, the value of demonstration data changes character:

Retry data — deliberate misses followed by recoveries — does nothing for the imitation policy, but it puts recovery behavior into the support, which SCORE then amplifies into 100% reliability. Suboptimal play data on an uncovered region of the workspace: same effect. A multi-task prior over three grasping tasks underperforms single-task priors as a BC policy (behaviors interfere), but after steering it beats single-task SCORE on out-of-distribution resets (95% vs 55%) by borrowing coverage across tasks. The paper's closing suggestion follows naturally and is a genuinely interesting research direction: pretraining datasets should be designed for support breadth, not zero-shot success. That inverts the standard imitation-learning intuition that demo quality is paramount, and it rhymes with why people believe messy internet-scale data helps LLM post-training.

What to be skeptical about

The support constraint is also the ceiling. SCORE cannot create behaviors; it can only reweight and locally refine them. The cross-object experiments show both edges: steering a bottle-grasp prior to pinch a carrot works (22%→67%) because pinches exist in the prior; steering a cup-grasp prior fails because they don't. With distractor cubes added, SCORE recovers only 56%, and only on half the workspace — if the base policy's perception can't distinguish the situation, there's nothing to steer. This directly answers the selection note's question: on tasks where the base policy is weak because its support lacks the needed behavior, SCORE fails by construction. Its wins on low-performing bases (Ball Pour at 11%, Credit Card at 10%) worked because the right behaviors were latent in the data, just rarely expressed.

Real trial counts are small. Per-task real evaluations range from 8 (Lightbulb) to 24 trials, so individual task numbers carry wide error bars; the aggregate over ~120 trials per method is more trustworthy. Some real reset distributions are also narrower than sim (fixed card pose, fixed target cup).

Fixed, scanned scenes. Every task requires scanning the deployment environment into IsaacLab and hand-configuring articulations. This is a per-task real-to-sim pipeline, not a general recipe yet — though far cheaper than real-world RL.

Baseline fairness looks solid but note the asymmetry. They swept 150 FPO configurations and tuned RialTo's BC coefficient across four orders of magnitude, which is more baseline diligence than most papers show. Still, FPO and RialTo were designed for somewhat different settings, and the strongest fair comparison — Residual-RL — is essentially SCORE with the noise-steering branch ablated, which it loses to convincingly.

The ranking assumption is unverifiable in general. The whole guarantee rests on sim preserving the ordering of realizable policies. Domain randomization presumably helps, but nothing enforces it; tasks with dynamics the simulator fundamentally mishandles (they note the squishy soccer ball, where SCORE tops out at 60%) will violate it.

Why this matters

The concept isn't new in isolation — support constraints come from offline RL (BEAR and successors), and flow steering comes from DSRL/RFS. The contribution is recognizing that these pieces compose into a solution for a different problem: off-domain policy improvement, where interaction is abundant but only in the wrong dynamics. The support constraint neutralizes exactly the failure mode (simulator exploitation) that has made real-to-sim-to-real pipelines dependent on reward engineering, distillation, and deploy-tune-redeploy loops. That SCORE runs with sparse rewards and one hyperparameter set across eight tasks, and deploys the sim-trained actor directly, is what makes it plausible as a paradigm rather than a result.

If it holds up, the practical implication is significant: every lab with an imitation-learned policy and a scannable workspace gets a nearly-free +50-point improvement loop, and the data-collection playbook shifts toward deliberately gathering retries, play, and multi-task coverage — data that looks useless for BC but is fuel for steering. The natural next experiment, which the authors flag, is applying this on top of large multi-task priors (VLA-style policies), whose broad support is exactly what the method wants.

Where to spend your time in the paper: Section 5.2 (the coverage ablations — the most conceptually novel findings), then Appendix E (the formal argument, short and readable, especially Proposition E.2's Wasserstein counterexample), then Appendix C's qualitative failure modes if you want visceral intuition for what "simulator exploitation" looks like in practice.