pollen-robotics/microduck_rl

TL;DR: Pollen Robotics (the Reachy people) have open-sourced the complete RL training stack for Microduck, an 800 g / 25 cm biped — not just environments, but the full sim2real recipe: a physics-accurate voltage-level servo model, gear-backlash simulation with encoder-side-of-play observations, domain randomization, a shared observation contract that lets a runtime hot-swap walk/recover/trick policies mid-episode, and a validated ONNX publishing pipeline to Hugging Face Hub. It's rare to see a deployable legged-robot stack released end-to-end like this, and the actuator-fidelity thesis is the interesting technical claim.

What it is

RL training environments for Microduck built on mjlab (MuJoCo Warp + rsl_rl, PPO), training at 50 Hz across 4096 parallel envs on a CUDA GPU, with ~1–2 h to a usable gait. The task registry goes well beyond walking: fall recovery fused into the walk policy, stand-up from face-down/face-up/sitting, sit↔stand, ground-picking with the mouth, ball kicks (with a ball-blind actor), a forward roll over the head ("roulade"), and a whole family of passive roller-skating tasks — velocity tracking on wheels, swizzle skating, crouching while gliding, sliding down slopes, spinning, and getting up onto the wheels.

The sim2real recipe is the point

The README's core claim: at this scale — tiny Dynamixel XL330 servos driving an 800 g biped — actuator fidelity is most of the sim2real gap. So instead of ideal PD actuators, every task trains against the BAM (Rhoban) M6 actuator model: voltage control law, back-EMF, Coulomb/Stribeck/load-dependent friction, with per-env DR on battery voltage, voltage sag under load, command delay, and friction magnitude.

The backlash modeling is unusually careful. Every main task has a -Backlash twin with ±1° of gear play in series with each of the 14 servos, implemented as an unactuated hinge. Crucially, because the real encoder sits on the output side of the play, both the firmware PD emulation and the joint_pos/joint_vel observations read through the backlash (qpos[servo] + qpos[backlash]). Observation/action dims are unchanged, so export and runtime are untouched. This is the kind of detail that usually lives in someone's head and dies with the project; here it's in tasks/backlash.py, and AGENTS.md distills the reward-design lessons (explicitly aimed at AI coding agents working in the repo, which is itself a nice touch).

The hot-swap architecture

All policies share a fixed 61-dim actor observation: 48 proprioception + commands [twist(3), head_pose(4), body_pose(6)], with unused command slots zero-padded rather than dropped. That single contract is what lets the onboard daemon swap any policy in at any moment — walking, then a recovery policy after a fall, then a trick. The exporter bakes the observation normalizer into the ONNX graph, so a policy is one self-contained [1,61] → [1,14] file.

There's also a community-distribution story: uv run publish validates the graph shape, runs it on plausible inputs (refusing NaNs or constant outputs), fills a manifest from git/wandb provenance, and uploads to HF Hub so anyone with a Microduck can robotctl policy add it without a firmware release.

What's there, what's not

There: all environments, MJCF models (Onshape-exported), the actuator/backlash code, CPU-only regression tests locking joint mappings and reward signs, HF Jobs support for GPU-less training, an infer_policy.py keyboard-driven sim rehearsal of the exact multi-policy switching, and a demo video of the real robot. Apache-2.0 code; 3D models are CC BY-SA-NC (no commercial hardware clones).

Not there: quantitative sim2real numbers — no tracking-error tables, success rates, or ablations showing how much the BAM/backlash modeling actually buys versus plain DR. Evidence is "it demonstrably works on video." And the payoff obviously requires the ~$300-class Microduck hardware; without it you get a very good worked example. That said, the recipe itself — voltage-level actuator modeling, encoder-through-backlash observations, the fixed-contract hot-swap pattern — is the transferable artifact, and it's directly relevant to anyone doing sim2real on cheap-servo platforms (Duckling/BDX derivatives, ToddlerBot-style robots).

Try it

git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl

# needs a CUDA GPU; ~1-2 h to a usable gait
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096
# or, no GPU: append --hf-jobs to train on Hugging Face Jobs

uv run scripts/export.py Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <run>
uv run scripts/infer_policy.py --walking output.onnx   # CPU MuJoCo, keyboard-driven

Read AGENTS.md first — the distilled reward-design playbook is arguably the most valuable file in the repo.