Hands That Know: Introducing DexGEM Lab
TL;DR
Contact-rich dexterous manipulation is the unsolved core of robot manipulation, and the obstacle is data. The action signal of skilled human hands was never recorded at scale; without it, the improvement loop that drove language AI cannot begin for physical dexterity. DexGEM builds this first rung: capturing human dexterous skill through ego video and motion capture, completing the force channel through simulation, and using the human hand as common ground for transfer to any robot morphology. We call the knowledge this work demands feel.
What a hand does
When we motion-capture a human hand rotating a small object in-hand, the recording shows twenty-plus degrees of freedom moving at once. All five fingertips stay in continuous motion, and the work comes in bursts: fast rolls separated by slower phases where the fingers reposition. Contact patches migrate across the finger surfaces as the object rolls.
Dexterous skill has two channels: kinematics and force. Cameras see the kinematics partially, because fingers occlude each other and the object hides the contact surfaces. Motion capture records the full kinematic chain. But forces (grip pressures, friction, the normal forces that keep the object from launching) leave no visual trace. What cameras see, what motion capture records, what simulation lets a policy experience: two channels, three instruments, and no single instrument is sufficient alone.
The unsolved core
Machines learned to see, read, and reason. None of them has a feel for how things work in your hands.
Robot manipulation has made real progress on pick-and-place, bin-picking, tabletop rearrangement, all tasks where a gripper opens, closes, and moves. Contact changes the structure of the problem, not just its difficulty. Gripper-level tasks are nearly open-loop: the bottleneck is perception and planning, which is exactly why vision-language model progress transfers directly to them. Contact-rich tasks move the bottleneck to closed-loop, high-frequency, force-mediated control, where vision is no longer the binding constraint. In-hand reorientation, insertion under force, cable routing, tool use, deformable manipulation, all of it a continuous stream of micro-adjustments. The work that factories, kitchens, operating rooms, and repair shops run on sits in this regime.
We call this feel: what skilled hands learn through doing — knowledge that lives in the act, not in the description. Specifically: the knowledge that contact-rich work demands. How hard to squeeze before a connector seats. When to release a rolling object so it lands in the target orientation. The stiffness change that means a snap-fit is closing. Force, slip, timing, micro-adjustment. These are the variables that decide whether the object stays in-hand or hits the floor.
Scaling the data
Our core bet is that scalable dexterous intelligence requires a human-centered data pipeline.
The language-model field climbed toward self-improving systems in stages. First rung: pretrain on the byproduct of human activity, the internet corpus, text that humans already produced. Second: give the model closed-loop execution, specifically tool use and agentic behavior, bootstrapped from synthetic instruction data. Third, now arriving: agents that improve through their own execution. Each rung requires the one below it.
In manipulation, the first rung is missing. How humans manipulate, the detailed action signal of skilled hands at work, was never recorded at scale. Without that corpus, closed-loop self-improvement for physical dexterity cannot start.
A natural question: the internet already contains vast amounts of hand video, including large egocentric corpora. Why isn't that enough? Third-person video lacks the action channel at usable resolution, and even egocentric recordings carry no joint-resolution kinematics and no force signal. The corpus that language AI got for free has no physical counterpart.
Teleoperation and human-robot collaboration do close the loop. For gripper-level and quasi-static tasks, teleop works and scales reasonably; the field's visible manipulation results are built on it. The failure is specific: the interface attenuates exactly the high-frequency, force-mediated channel that in-hand skill runs on, and its per-hour economics fail at scale. The signal problem is mechanical.
The clip above shows the multi-fingered hand working a bottle task in two recording sessions. On the right, the autonomous policy produces a fluid, continuous reorientation. On the left, the teleop operator hesitates on approach. Three mechanisms likely account for what you see. Force feedback is absent, so the operator guesses grip force and the fingers close unevenly. Latency between intention and actuation turns each correction into an oscillation. And the operator's own hand has different proportions and friction surfaces than the one being controlled, so twenty years of finger-curling intuition does not transfer. The bottle is knocked over, and you can watch it happen in the final seconds.
Web text scaled because it was already digital, self-labeling (the next token is the label), and a byproduct of activity humans do anyway: near-zero marginal collection cost. Teleop fails all three tests: collection is the entire cost, and the action signal passes through an instrument that degrades it. Human skilled work is the one physical-domain source that can approach the byproduct property: people do dexterous work regardless, and the label (what the hand did, at joint resolution) is in the signal itself. Instrumented capture is not free today; gloves drift and mocap setups have cost. What it buys is exactly what teleop cannot offer: resolution, naturalness, and durability of the signal.
This knowledge lives in hands — so we learn it from hands, not from instruments that borrow them. Ego video sees approach angle and pre-contact posture. Motion capture records the full kinematic chain at the speed the skill actually happens. The precedent is autonomous driving, which scaled on data from humans driving cars directly, with no instrument between the driver and the machine. Together these channels constitute the first rung: a corpus of human dexterous skill recorded at joint resolution.
Recording yields kinematics. The force channel (grip pressure, friction, the normal forces that keep a rolling object stable) is missing from any passive recording. A policy training in simulation on tasks derived from recorded human motion experiences contact physics consistent with those kinematics. Simulation is where the invisible half of the signal becomes computable. It does not recover the true forces of the recorded episode; it supplies physically consistent contact experience. The residual gap between simulated and real contact is open. We own the question, and it is the subject of a later post.
Morphology-specific robot data dies with its hardware generation. The human hand has a demonstration supply no robot fleet can match, and a stable identity over time: data collected in 2026 can still train the 2030 model. The human hand is the common ground: skill learned on the simulated human hand transfers to each specific robot morphology as a focused adaptation problem. That transfer is our working bet, and the first open problem below.
What we don't yet know
Three open problems. First, whether parameter-efficient adaptation still suffices when the data used to teach dexterous skill approaches the scale of initial pretraining; the common-ground transfer bet rides on the answer. Second, whether contact experience that is physically consistent in simulation transfers to real contact; the sim-to-real gap for force-mediated tasks remains open, and tracking recorded human kinematics into a simulated hand is an open problem of its own. Third, what the scaling curve of dexterous skill versus data hours looks like. We believe there is one and intend to measure it. Scaling laws are things you verify, not invoke.
What we are building
Each piece of this work becomes a chapter:
- Recording feel: how we capture the fine-grained data of contact-rich skill from ego video and motion capture.
- Teaching feel: how the simulated human hand serves as common ground for transfer to robot morphologies.
- Rehearsing feel: how simulation completes the force channel and how policies cross back to real hardware.
- The sense beneath feel: what cameras miss about force and slip, and why the tactile channel matters for contact-rich work.
DexGEM (Dexterous Generalist Embodied Manipulation) is the research lab we are building around this thesis. The work is starting, and we would like to build it with you.
If you work on dexterous manipulation, human-data pipelines, simulation transfer, or tactile sensing, reach out at [email protected].
Citation
@misc{dexgem2026introducing,
title = {Hands That Know: Introducing {DexGEM} Lab},
author = {Lu, Irvine and Wu, Guangzheng and Chen, Siyuan and Liu, Chengyuan and Sun, JieQ and Zhang, Jiahong and Zhao, Bin},
year = {2026},
url = {https://dex-gem.ai/blog/introducing-dexgem-lab},
note = {Blog post}
}