Issue 29 · Pick 09 Robotics ✓ read
Towards Human-level Dexterous Teleoperation
TL;DR: Dexterous teleoperation systems today map your hand's joint angles onto a robot hand and hope the object comes along for the ride — which is why teleoperated robot hands can grasp but essentially cannot do in-hand manipulation. TeleDexter reframes teleoperation as hand–object co-tracking: the operator specifies where the fingertips and the object should be, and a learned RL controller figures out the contact physics needed to get both there. The critical training trick is to abandon frame-by-frame imitation of human demonstrations in favor of reaching a sequence of subgoals, giving the policy freedom to invent its own contact strategies in between. The result: 75% average success on seven real-world tasks including in-hand reorientation, finger gaiting, screwdriver tightening, and lightbulb replacement, where every baseline scores at or near zero — and the collected demos train autonomous diffusion policies for tasks no existing teleop system could even demonstrate.
Why teleoperating a dexterous hand is not like teleoperating a gripper
For a parallel-jaw gripper, teleoperation is nearly solved: track the wrist, open/close the gripper, done. For a 22-DoF five-finger hand attempting the things human hands actually do — spinning a screwdriver with finger gaiting, flipping a hammer from face-down to claw-down without setting it down — the dominant approach fails in an instructive way.
That approach is kinematic retargeting: track the operator's hand (vision or glove), solve an optimization mapping human finger poses onto the robot's different kinematics, and command those joint angles. This gives a low-latency, intuitive interface. But notice what's absent from the entire pipeline: the object. The robot is a joint-angle puppet. It knows nothing about contact forces, friction, or inertia. When the human's fingers roll a cylinder, the robot's fingers make superficially similar motions on a slightly different hand with different friction, and the object slips out. In the paper's evaluation, the retargeting baselines (DexRT, GeoRT) almost never get past the first stage of any task that requires in-hand reorientation — 0–6.7% success across the board.
A more recent alternative (DexGen) trains a generative "action prior" in simulation that converts coarse teleop commands into contact-rich finger actions. But it trains on synthetically generated grasp transitions of dubious physical feasibility, and its generative model drifts during closed-loop rollouts — it also scores 0% on every task here.
The gap this exposes: the operator can tell the robot what the hand should do, but nobody is closing the loop on what the object should do.
The key idea: track the object too, and let RL own the contacts
TeleDexter's reframing is simple to state. The teleop goal at each timestep is a pair g_t = (\hat{\boldsymbol{p}}^{\mathrm{tip}}_t, \hat{T}^o_t): target fingertip positions \hat{\boldsymbol{p}}^{\mathrm{tip}}_t \in \mathbb{R}^{N_f \times 3} and a target object pose \hat{T}^o_t \in SE(3), both captured live from the operator's hand and the real object they're holding (via motion capture) and expressed in the wrist frame. A low-level policy \pi_\theta(\boldsymbol{o}_t, g_t), trained entirely in simulation with RL, outputs joint position targets that physically drive both the hand and the manipulated object toward the goal. The arm tracks the wrist separately via IK.
The division of labor is the point: the goal specifies what the hand–object system should achieve; the learned controller decides how — which fingers push, when to break and remake contact, how to gait. The operator's slippery human-hand strategy never needs to be feasible on the robot; the policy just needs to reach the same hand–object configurations by any physically valid means.
How the controller is trained: subgoals, not frames
Prior work has tried guiding RL with human hand–object reference motions, but almost always with dense frame-wise tracking: reward the policy for matching the reference at every timestep. The authors argue this is precisely why those methods top out at grasping and wrist motion — copying every frame leaves no slack to discover contact-switching strategies. If the human's exact finger trajectory is infeasible on the robot, the policy fails; it can never say "I'll release the index finger and take a different path to the same end state."
TeleDexter's consecutive subgoal co-tracking loosens the leash. Each reference trajectory is chopped into subgoals sampled at varying intervals (growing from ~0.67s to ~1.33s apart over training). A subgoal counts as reached when all fingertip errors are under 3 cm, object position error under 1 cm, and object rotation error under 10°, held for several frames. Between subgoals, the policy is unconstrained — free to gait fingers, break contact, and find its own route.
The reward is a hybrid: a sparse subgoal bonus that fires on reaching each waypoint (scaled by the subgoal gap so bigger jumps pay more), plus a small dense tracking term (\alpha_{\mathrm{dense}}=0.1) that shapes early exploration, minus a per-step time cost. The ablation (Table 4, sim) is stark: under sparse evaluation, the subgoal-trained policy chains 178–187 consecutive subgoals on hammer and screwdriver, while dense-tracking-trained policies stall at ~2.7. Even on dense evaluation — which favors the dense baseline — the subgoal policy survives ~3× longer episodes. This is the paper's cleanest evidence for its central design claim.
Two more ingredients matter:
Geometry-aware retargeting of the reference data. The human demos (~50 minutes per object of unscripted mocap interaction: in-hand translation, rotation, free-play with gaiting and tool motions) are retargeted to the robot in two stages — standard vector retargeting, then an object-mesh-aware refinement with SDF-based surface attraction, interpenetration penalties, and self-collision terms. This produces subgoals that are at least geometrically feasible: fingers on the object surface, not inside it. The same human data retargets to both hands, which is how one dataset serves two embodiments.
Random action masking for sim-to-real. With probability 0.15, a random subset of 3 joints has its commands frozen at the previous value for 1–10 steps. The policy must succeed despite partially stale actuation — a proxy for the motor lag, backlash, and compliance mismatch of real hardware. The authors call it their single most impactful sim-to-real intervention, and the real-world ablation backs it up: on ScrewdriverUse, success drops from 73.3% to 0% without it.
Training is single-stage RL (SAPG, a multi-block PPO variant whose parallel exploration blocks matter for covering distinct contact modes — vanilla PPO underperforms substantially) in Isaac Gym: ~62,000 parallel environments, ~10^{10} steps, about a day on 4×RTX 5090 per object per hand. A curriculum anneals gravity, tolerances, and subgoal spacing.
The evidence
Seven real tasks on a Franka FR3 + SharpaWave (5-finger, 22-DoF) hand, 15 trials each, staged scoring: three reorientation tasks (cylinder, cuboid, Stanford bunny) and four long-horizon tool-use tasks — hammering nails in and pulling them out with the claw (7 stages), sweeping with a brush in two orientations (7 stages), tightening a screw with sustained finger gaiting (5 stages), and screwing a lightbulb in until it lights, then unscrewing it (6 stages).
The stage-wise breakdown (Appendix A.3, worth reading) shows where the gap opens: every baseline collapses at the first stage requiring in-hand reorientation or grasp transition, not at grasping. On BulbReplace, TeleDexter completes both screw-in and unscrew rotation stages in 15/15 trials, failing only twice at final placement. On ScrewdriverUse, 13/15 trials sustain continuous finger gaiting through tightening — a contact mode no baseline executes even once (DexRT's single completed trial being the lone exception across all tool tasks). The narrow gap between success rate (75.2%) and task progress (87.1%) says failures happen late, not early.
The same pipeline transfers to LeapHand (4-finger, 16-DoF) from the same human reference data — 60–73% on reorientation — and a stress test streams arbitrary randomly-sampled hand–object goals at the controller: 41 consecutive repositions on a cylinder (97.6% per-target success), 12 on a cuboid, well beyond memorized trajectories.
Finally, the autonomy payoff: 50 teleoperated demos per task train Conv-UNet Diffusion Policies (frozen DINOv2 vision) that run without a human — 73% on hammer-and-drive, 47% on bulb install, 40% on brush sweep. Modest numbers, but for tasks that no baseline teleop system could demonstrate at all, so this data literally couldn't be collected before.
What to be skeptical of
The baselines are weak by construction, and one is a re-implementation. The retargeting baselines were never designed for in-hand manipulation; showing they fail at it confirms known limitations rather than beating a strong contender. More importantly, DexGen has no public code — the authors re-implemented it and substituted their own co-tracking controller for an underspecified intermediate stage. A 0%-everywhere result for a re-implemented baseline warrants caution. The fairest read is "no existing system does this," not "TeleDexter beats strong rivals."
The comparison isn't fully apples-to-apples on the easy part. TeleDexter itself uses kinematic retargeting for the pre-grasp approach and switches to the learned controller only after stable contact — a manual phase switch by the operator. Reasonable engineering, but it means the co-tracking controller never handles contact acquisition.
Per-object controllers and a mocap fortress. Each controller is object-specific: a new object means new mocap collection (~50 min), retargeting, and a day of RL. Deployment requires a NOKOV optical mocap system with marker gloves and marker-tagged objects. This is a lab instrument, not a product; the authors acknowledge both limitations. Objects are also wrapped in friction tape.
Operator effects are unreported. Success rates presumably reflect trained operators; there's no operator study, latency analysis, or learnability assessment.
Known failure modes are informative. The failure taxonomy (Appendix A.4) is refreshingly honest: impulsive tool–environment forces (hammer strikes) are out-of-distribution because training is free-space-only; fingers jam on concave geometry (the bunny); and without tactile sensing the policy can't tell a pressing finger from a sliding one, causing regrasp stalls.
Why it matters anyway
The framing shift — teleoperation as object tracking with learned contact execution, trained via subgoals rather than imitation — feels like the right abstraction, and the ablations tie the headline result to the specific design choices rather than just RL-plus-compute. If the mocap dependency falls to vision-based tracking and the per-object controllers become object-conditioned, this is a data-collection engine for exactly the manipulation regime — finger gaiting, regrasping, functional tool use — where current robot learning datasets are essentially empty. The demonstrated pipeline from human motion → sim RL → real teleop → BC autonomy, all for skills previously undemonstrable, is the concrete contribution.
Where to spend your time: Section 3.1 for the subgoal formulation and hybrid reward, Table 4 for the subgoal-vs-dense ablation that carries the paper's core claim, and Appendix A.3–A.4 for the stage-wise breakdowns and failure analysis, which are more informative than the headline table.