Bicycle Acrobatics with Reinforcement Learning

TL;DR: A 23.5 kg bicycle robot — two wheels in tandem, five actuators, statically unstable by construction — learns front flips, kip-ups from lying upside down, wheelies balanced on one wheel, and jumps onto 1 m tables, all trained with PPO in simulation and deployed on hardware. The two ideas worth stealing are (1) a deliberate matching of reward density to stunt difficulty — five different RL formulations, from sparse waypoint rewards up to dense motion imitation, each used only where it's needed — and (2) an orchestrator that chains these specialist policies using state-space conditions rather than timers or button presses, which is what makes 20+ consecutive kip→jump→flip→kip-down loops possible on real hardware. This is from the Robotics and AI Institute (the Boston Dynamics-descended lab), on their custom UMV platform.

Why a bicycle is the hard mode of legged robotics

There's a standing trade-off in mobile robots: legs handle obstacles but burn energy and move slowly; wheels are fast and efficient but stop dead at a curb. The UMV (Ultra Mobility Vehicle) is a bet that a bicycle with an articulated body can get both — and this paper is about the control problem that bet creates.

Three things make a bicycle strictly harder to control than a quadruped or humanoid:

No support polygon. A quadruped with locked joints stands; a bicycle with locked joints falls over. Its contact with the world is a line, so it is statically unstable at all times, including while "standing still."

Non-minimum phase dynamics. To turn right at speed you must first steer momentarily left (counter-steering). The instantaneous response to control is opposite to the eventual one — the classic signature of a system where naive feedback fights you.

Almost no degrees of freedom to spend. A humanoid arrests a fall with an ankle torque or an arm swing; a quadruped steps out to make a new base. UMV has five actuated joints total: rear wheel drive, steering, and three joints (head, left/right fins) that move a heavy "head" mass relative to the bike frame through a spatial linkage. To not fall, it must drive along a curve. There is no stepping out.

Prior RL work got bicycles to balance and ride, and one 2014-era paper (Tan et al.) learned stunts in a simplified simulator that never touched hardware. Nothing like a physical bicycle robot doing a front flip existed.

Idea 1: a spectrum of reward densities, one per stunt

The methodological core is not one algorithm but a taxonomy. The authors order five RL formulations by how much structure the reward supplies, and assign each stunt to the sparsest formulation that can learn it:

entropy coefficient and training iterations both decrease left to right: sparse tasks need exploration (30K iters); imitation needs adherence (15K iters)

The interesting endpoints:

Jumping is never rewarded directly. The jump policy \pi_j is trained with waypoint following: reach a 2D goal x_{\text{goal}}, given a robot-centric heightmap. The trick is the training terrain — the robot starts inside a ring of increasing height (a terrain curriculum), so the only way to reach the goal is to figure out jumping. Training with boxes instead of rings, or a fixed ring height, fails to converge. Because the same policy is also trained on flat ground, one network handles driving, approaching, jumping up, traversing, and jumping down.

Flips can't be reward-shaped, so they're imitated — but there's no MoCap for bicycles. No human demonstration dataset exists for this morphology, and retargeting from humanoids is meaningless. They use Iterative Motion Imitation (IMI), which tolerates references that are initially kinematically or dynamically infeasible, so a rough hand-designed trajectory suffices. For the lateral jump they pull a neat bootstrap: train a waypoint-following lateral jump first, record its trajectories, then re-train via imitation of those trajectories — because imitation trains in half the iterations, enabling faster sim-to-real cycles.

Kip-ups sit in between. The kip-up (springing from lying upside-down to standing on wheels) is specified only as a terminal SE3 pose; the policy finds the explosive strategy itself: thrust the base to over 1 m height, retract to gain momentum, swing the wheels underneath, absorb the landing. This task needed a higher-fidelity triangular-mesh collision model of the head, since the maneuver starts with the head grinding against the floor.

There's also a clean empirical regularity buried in the training tables: the PPO entropy coefficient and iteration count both scale inversely with reward density (0.006 and 30K iterations for waypoint following, 0.0025 and 15K for imitation). Sparse tasks need exploration to discover the behavior; dense tasks need to stop exploring and track. Obvious in hindsight, useful as a recipe.

Idea 2: the orchestrator is funnel composition, not a script

The second question — how do you chain a kip-up into driving into a jump into a flip on real hardware — is where most "trick" papers stop at one trick. The authors build an orchestrator that is formally a finite state machine, but designed around sequential composition (Burridge et al.'s funnel idea from the late 90s): each policy has a region of state space it can handle (its funnel), and a transition edge fires only when the current policy has driven the robot into the entry region of the next one.

The key property this buys: each policy's job includes reaching the next policy's basin. If a landing is sloppy, the waypoint policy absorbs the variation before the flip condition can fire. That's why the composed repertoires don't accumulate error across a 5-loop, 4-stunt run — every transition is a re-funneling. The authors are candid that the transition conditions (waypoint radii, yaw tolerances, 0.8 s settle times) are hand-designed heuristics, and they flag learned, continuous transitions as future work. But note what the orchestrator is not doing: it isn't stabilizing anything or blending actions. It's a switch. All the dynamics — recovering from a hard 1 m landing, balancing on one wheel — live inside the learned policies. The composed success rates back this up: the full kip-jump-flip repertoire runs at 0.97, essentially the product of its parts, over 20+ consecutive hardware trials.

The evidence

Everything reported is on physical robots, with per-stunt success rates and repetition counts:

The physical intensity is worth registering because it explains why sim-to-real here is nontrivial: the kip-up peaks at ≈4600 W mechanical power and ≈130 A bus current (all momentum must come from a standstill, upside down); the flip hits ≈19 rad/s angular velocity (a full rotation in a fraction of a second on a 23.5 kg machine); jump landings from 1 m drive high regeneration power as the actuators back-drive to absorb impact; the kip-down logs ≈22 m/s² — a controlled backwards fall, because with five DOFs there is no way to fall gently.

Two emergent behaviors are the paper's best evidence that the policies learned strategies rather than memorized trajectories. First, a "snake-like" climb: instead of a ballistic leap from far out, the robot approaches close, pitches up right at the lip, and pitches down right before touchdown — an energy-minimizing solution the torque/impact penalties selected for. Second, when trained with noise and drift injected into the heightmap, the policy consistently developed a brief wheelie at the table's far edge before descending — a hedging posture against the table edge not being where perception says it is. And the single-table jump policy, deployed zero-shot on two tables, works (0.67); the two-table policy generalizes to three (0.50). Ablations in the supplement sharpen the envelope: the jump policy, trained only up to 100 cm obstacles, holds 100% success through 100 cm in simulation and only collapses past 120–160 cm.

Regarding the flips specifically — the triage question: standalone flip-and-drive orchestration is reported at 1.00 success, and flips inside the long repertoire at 0.97 across 20+ trials, which is genuinely strong. But the flip-onto-a-table ablation shows how narrow the imitation envelope is: 100% up to 50 cm table height, ≈10% at 70 cm, and a ±5 cm sweet spot on table position, with 0% success at −20 cm offset (front wheel eats the table face). Imitation gives you the flip; it doesn't give you a flip that reasons about the world.

What to make of it, and what to be skeptical about

If the result holds up, the headline is that the agility/efficiency trade-off is softer than assumed: a five-actuator, single-track vehicle can do parkour-class maneuvers that we associated with 12–28-DOF legged machines, using entirely standard RL machinery (PPO, Isaac Lab, MLPs, asymmetric actor-critic, constraints-as-terminations, domain randomization). Nothing algorithmically exotic — the contribution is the decomposition: right reward density per skill, funnel-composed switching, and serious sim-to-real engineering (drop-test-calibrated compliant wheel contact models; dynamically computed torque limits from motor electrical constraints rather than fixed clamps; testing deployment code in MuJoCo, a different simulator than training, to catch simulator overfitting).

The honest caveats:

• No onboard perception. Localization is MoCap-fused; the "heightmap" the jump policy consumes is sampled from a map built offline from known table positions. Generalization to "unseen multi-table configurations" means unseen geometry, not unseen sensing. Moving this outdoors with onboard exteroception is a real open problem the paper doesn't touch.
• The orchestrator is hand-tuned per repertoire, and several transitions (kip-up, kip-down, flip) are user-triggered buttons; the autonomy is in execution and sequencing, not in deciding what trick to do.
• Precise placement is the weak axis. Anything requiring accurate wheel positioning across multiple tables drops to 0.50–0.67. The authors' own diagnosis is sound: they need to split a high-level online motion generator from a general low-level tracking policy, rather than baking each behavior into its own network.
• Success-rate denominators for some Table 1 entries are unstated ("–"), and this is a single lab's single platform with substantial engineering headcount behind it — reproduction cost is high.

Where to spend your reading time: the Methods sections on the five training formulations and the Orchestrator (with Fig. 8), plus supplement S3 (ablations) if you want the failure envelopes — the reward tables (S3–S7) are also unusually complete and would let you actually rebuild these policies. And watch the videos on the project page; a bicycle kipping up from its back and riding off is one of those results that reads differently in motion.