From Human Hands to Robot Hands: Learning Dexterous Manipulation from Motion Capture

The Data Problem

Learning-based robot manipulation improves where training data exists, and for contact-rich dexterous tasks that data barely exists. Teleoperation produces it slowly: one operator, one robot, one hour at a time, through an interface that filters out the fast, force-driven adjustments the skill depends on. Human hands perform these tasks constantly, and motion capture records them at joint resolution. Three gaps separate such a recording from robot training data: it carries no forces, it violates physics when replayed, and the human hand differs from any robot hand in geometry, joint limits, and actuation.

This post presents a pipeline that closes those three gaps. It takes a human motion-capture recording as input and produces a deployable robot policy as output, with no teleoperated robot data at any point.

Our R2S2R Pipeline

The pipeline converts a human motion-capture recording into a closed-loop dexterous manipulation policy in four stages.

Human motion capture records hand and object kinematics at 120 Hz on a desktop-scale system with five binocular infrared cameras. The operator wears an optical-inertial glove, so capture works at a desk rather than in a marker-based mocap room. Recorded motion is expressed through the MANO hand model (Romero et al., 2017), a parametric model of hand pose and shape, with per-subject shape estimated from a short calibration sequence. The result is frame-wise hand pose and object motion, the kinematic input to everything that follows.

Physics refinement re-executes the captured motion with a residual-RL policy on a simulated actuated MANO hand, recovering per-frame contact and force annotations. A single policy handles all tasks jointly, so processing a new demonstration requires only a forward pass. Perturbation-based synthesis expands each demonstration into hundreds of variants under varied initial conditions.

Contact-anchored retargeting transfers the demonstrated contact structure to a target robot hand through morphology-agnostic numerical optimization. The optimization preserves where fingers touch the object rather than copying joint angles. Changing the target hand changes the optimization geometry but not the mathematical formulation; an embodiment profile is the only hand-specific input.

Residual policy learning restores dynamic feasibility on the robot embodiment via the same residual mechanism, producing a policy that deploys to real hardware with zero real-robot training data. The policy outputs small corrections to the retargeted reference rather than full joint-position commands from scratch, making sim-to-real transfer substantially easier.

Where the Gap Is: Mocap to Simulation

Motion capture records the state of a human hand and an object at high precision, but only the kinematics, with no guarantee of physical consistency. The real-to-sim gap comes mainly from two sources: the conversion from raw mocap to the MANO model introduces tracking error, and kinematics alone carries no contact forces. Replayed as-is in simulation, the recorded trajectory shows fingers hovering just above the object surface without real contact, or passing straight through it: visually close, physically broken.

Residual RL: Learned Data Repair

To close this gap, we treat the captured MANO pose sequence as a kinematic reference and train a residual-RL policy to track it with a simulated actuated MANO hand. At each control step the policy outputs a small correction added to the current reference frame, keeping the commanded pose within a bounded envelope around the recorded motion. The policy maximizes a reward combining object-trajectory tracking (penalizing deviation from the recorded object motion) with contact-achievement terms that encourage designated fingers to reach the object surface.

For simulation, the parametric MANO mesh is given an articulated, actuated skeleton (22 finger joints plus a 6-DoF wrist) that the residual policy drives. Because the mocap reference is already millimeter-accurate, the corrections stay small and training converges quickly.

One policy across all tasks. A single policy is trained jointly across all training trajectories (our own mocap recordings plus the DexCanvas dataset (Xu et al., 2025)), conditioned on object geometry, the expected contact pattern, and current and future reference object motion. Because the policy only repairs kinematic references into physically consistent trajectories, these observations are always available at both training and inference time. Given any new demonstration from the training distribution, physics refinement produces a physically consistent version with contact annotations in a single forward pass.

Contact Annotation

With physically consistent hand-object interaction in place, the simulator yields what no passive recording can: each physics-refined trajectory stores per-frame contact labels (which finger regions are in contact), object-frame contact locations (where on the object surface each contact occurs), and reconstructed force vectors. Obtaining even partial contact data on a real hand would require dense tactile sensors. These annotations drive the retargeting step below, and serve as a supervisory signal in the downstream experiments.

Data Synthesis

The trained policy doubles as a data augmentation engine. We perturb initial hand and object poses, randomize friction and mass, and add random external force perturbations. Each variant preserves the task's contact structure while exploring neighboring dynamically feasible trajectories. One human demonstration expands into hundreds of physically consistent variants, with no additional human effort, only GPU hours.

Retargeting to Robot Hands

Retargeting human hand motion to a robot hand is an established problem (DexPilot, Handa et al. 2020; AnyTeleop, Qin et al. 2023; SPIDER, Pan et al. 2025). What changes here is the starting point: the per-frame contact annotations from the previous stage. Because we know which finger region touches the object, where, and in what order, we can retarget what the fingers do to the object (where they touch it, and in what order) rather than how the joints are configured. A human hand and a robot hand differ in kinematic structure, joint ranges, and link proportions, so identical joint angles produce different contact locations; contact is the right thing to preserve.

Contact-anchored Optimization

For each frame of a physics-refined trajectory, we optimize the robot hand's configuration (wrist SE(3) and the hand's active finger controls) so that designated fingertip surface points approach the recorded MANO contact locations on the object. Frames are segmented by their contact pattern (the set of finger regions simultaneously in contact), and different objectives apply depending on how many fingers are active.

Multi-contact frames are where the real work happens. The optimization minimizes a weighted sum of terms, each serving a distinct purpose:

• Contact-anchor term (primary objective): each contact label from the physics-refined trajectory names an anatomical finger region (e.g., "index fingertip" or "thumb pad"). On the robot hand, this region maps to a set of mesh surface points. For each contact, we select the surface point whose direction from the link center best matches the corresponding MANO contact direction, with distance as a tiebreaker. The optimization then adjusts the hand configuration so this point approaches the recorded object-surface position, penalizing distance beyond a 1 cm tolerance.

• Object penetration: penalizes any robot finger geometry that passes through the object mesh, the same artifact we saw in raw mocap playback, now prevented on the robot side.

• Self-penetration: penalizes collisions between the robot's own finger links. Different morphologies have different collision geometries; this term keeps the solution physically realizable.

• Hand-object distance: penalizes fingers drifting too far from the object. Without this, the optimizer can trivially satisfy the no-penetration constraint by pulling the hand away entirely.

• Joint-limit violations: penalizes configurations that exceed the robot's URDF joint limits. A retargeted pose that looks correct but violates joint limits cannot be executed.

• Ground penetration: prevents the hand or object from passing through the ground plane, which matters for tabletop tasks where the object rests on a surface.

• Temporal smoothness (three terms): regularizes wrist translation, finger controls, and wrist rotation across consecutive frames. Without smoothness, the optimizer can find per-frame solutions that are individually valid but produce jerky, unexecutable motion.

Sparse-contact frames (zero or one active contact) use only feasibility terms (penetration, hand-object distance, and joint limits), with the wrist pose frozen to the MANO-derived initialization and only finger controls optimized. The intuition: when few or no fingers are in contact, the hand is in transit, and the priority is keeping the motion smooth and collision-free rather than hitting specific contact targets.

Initialization. Each frame is initialized from the corresponding MANO configuration. The 22 MANO finger-joint values are mapped to the robot hand's active finger controls through an approximate kinematic correspondence, clipped to the robot's joint limits. Wrist orientation is taken from the MANO wrist relative to the object; wrist translation is initialized as the mean offset between robot and MANO fingertip positions.

Morphology Generality

The optimization operates on forward-kinematic surface points, embodiment geometry, joint limits, and an active control vector. Changing the target hand changes the optimization dimension and geometric queries but not the mathematical loss. An embodiment profile (kinematic model, control-to-joint coupling, joint limits, and candidate surface points per contact region) is the only hand-specific input. The same human demonstration then maps to every supported hand; the human hand is the common ground.

From Retargeted Motion to a Robot Policy

Retargeted trajectories are not yet physically consistent on the target hand: the retargeted trajectory places the robot's fingers at the correct contact locations in the correct order, but as a kinematic solution it ignores actuator torque limits, tendon coupling, and object dynamics under the robot's contact geometry. This is the same kind of gap as the one between raw mocap and the simulated MANO hand, now in a new body, and we close it the same way as there: a second residual-RL policy learns corrections to the retargeted reference, using the same residual formulation as physics refinement. Corrections are applied to wrist translation and the hand's active finger controls; wrist orientation follows the retargeted reference unchanged. The deployment target is the DexHand 021, DexRobot's tendon-driven five-finger hand. The policy's 12 active finger controls are routed through the hand's deterministic tendon coupling to drive 20 physical finger joints.

After the final demonstrated contact, finger correction decay is set to zero (no fade), so accumulated corrections persist and established grasps are maintained. As with physics refinement, a single policy is trained jointly across all retargeted trajectories using PPO.

Experiments

All simulation experiments evaluate closed-loop rollouts under perturbation. Unless a section states otherwise: a rollout succeeds when every manipulated object stays within 3 cm of its recorded reference trajectory throughout execution, with an additional 30° final-orientation bound for orientation-constrained tasks. Single-hand evaluation covers 14 objects (85 object–gesture combinations, 20 attempts per trajectory under 2 cm pose perturbation); bimanual evaluation covers 25 tasks (250 trajectories, 50 attempts per trajectory, object size perturbed by ±10%).

Physics Refinement

Raw mocap playback succeeds on only 7.3% of single-hand trajectories across 14 objects and 16.7% on bimanual tasks (25 tasks, 250 trajectories). Physics refinement brings this to 67.9% on single-hand and 63.5% on bimanual, with one shared policy and no per-task tuning.

Contact supervision matters quantitatively. Removing explicit contact conditioning drops success from 67.9% to 59.7% on single-hand tasks (−8.2 pp) and from 63.5% to 55.2% on bimanual (−8.3 pp). The benefit is largest on tasks where success hinges on sustained fingertip contact (Stand, Balance, and Place tasks); the one inversion, Lay, is the smallest cell in the table at 30 attempts.

The same shared policy generalizes to unseen objects: tested on six YCB objects absent from training (~1,300 rollouts each), it achieves 72.4% with zero fine-tuning.

Retargeting Quality

100 MANO trajectories retargeted to DexHand 021, compared against four baselines: Position and Vector retargeting from AnyTeleop, DexPilot, and SPIDER. Our method is the only one that simultaneously achieves high contact fidelity (14.7 mm), low joint-limit violations (3.2%), and the lowest temporal jitter (0.53 rad/s). Vector retargeting achieves the best raw fidelity (10.6 mm) but at 29.9% joint-limit violations and 3.46 rad/s jitter: trajectories that look right on paper but violate the robot's physical constraints.

Cross-embodiment Transfer

200 single-arm trajectories retargeted to four robot hands (DexHand 021, DexHand 021 Pro, Shadow Hand, and Allegro) with identical training configuration and no per-hand tuning. Each trajectory tested 50 times under ±2 cm perturbation (10,000 episodes per hand).

All four hands gain substantially from residual correction: average +26.6 pp. Allegro, the most morphologically distant from MANO, gains the most (+34.9 pp). Switching hands requires only a new embodiment profile and retraining the robot-side residual policy, with no changes to the formulation.

Contact as a Downstream Supervisory Signal

A diffusion policy trained with per-frame contact forces appended (80-dim) versus kinematics only (64-dim), with the same architecture, data, and hyperparameters. Four objects × four contact patterns, tested 50 times each under a 0.5 N load and a strict 3 cm / 30° criterion.

Contact conditioning raises the average success rate from 34.7% to 46.5% (+11.8 pp). The largest single-object gain is +19.2 pp. Contact force tells the policy not just where the fingers are, but whether the grasp is actually working.

Real-world Deployment

Four bimanual tasks, all requiring sustained, coordinated multi-finger contact: stacking paper cups, passing and standing a bottle, pouring water, and in-hand bottle reorientation.

Results over 10+ trials each with no manual intervention: stacking cups 7/10, passing and standing a bottle 5/10, pouring water 6/10, in-hand reorientation 3/10. Overall above 50%, with policy weights frozen and zero real-robot training data at any stage. The remaining failures come primarily from the tendon-driven hand's sim-to-real dynamics gap and real-world perception noise.

Paired Data for In-Context Learning

The immediate product of this pipeline is paired training data. One human demonstration enters; hundreds of physically consistent robot trajectories leave, each preserving the task's contact structure but differing in initial object pose, hand pose, friction, and mass. Every trajectory carries per-frame contact labels and reconstructed forces.

This pairing, one demonstration with many executions of the same intent under varied conditions, is the input that In-Context Learning (ICL) for manipulation needs. A policy trained on such pairs can condition on a demonstration and adapt its execution to the state in front of it, instead of memorizing one trajectory. Because retargeting treats the embodiment as a profile rather than a rebuild, the same demonstration supplies paired data for each supported robot hand. And because the deployed policies run autonomously on real hardware, they collect real interaction data as they work, adding ground-truth contact experience that simulation cannot supply.

Extending to a new hand still means writing an embodiment profile and retraining that hand's residual policy; we have done this for four hands, and retargeting assumes the target hand can physically reach the required contacts. The sim-to-real gap in tendon-driven dynamics is visible in the real-world results above, and in-hand reorientation, at 3/10, is where it costs most. The next step is to train a demonstration-conditioned policy on this paired data and measure how far in-context execution generalizes beyond the training task set.

@misc{dexgem2026real2sim2real,
  title  = {From Human Hands to Robot Hands: Learning Dexterous Manipulation from Motion Capture},
  author = {Sun, JieQ and Yang, Zihao and Liu, Chengyuan and Zhou, Yu and Lv, Runze and Zhang, Jianxi and Lu, Irvine},
  year   = {2026},
  url    = {https://dex-gem.ai/blog/real2sim2real},
  note   = {Blog post}
}