Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

Abstract: How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.

• The paper introduces a novel two-stage pipeline that reconstructs 4D hand-object dynamics and robustly retargets them onto multi-fingered robotic hands.
• It leverages advanced vision models and dynamics-aware retargeting with warmup steps, force perturbations, and transition rewards to enhance robotic grasp success.
• The approach achieves significant SOTA improvements, boosting success rates from 25% to 71-81% in both simulation and real-world robotic tasks.

Scalable Dexterous Manipulation Dataset Generation from Monocular Human Videos with DO AS I DO

Problem Statement and Motivation

The primary challenge addressed is the scalable collection of robot-relevant dexterous manipulation data on human-like, multi-fingered robotic hands. While observational human video data is abundant, existing methodologies either require specialized sensor setups (e.g., MoCap, RGBD), rely on restrictive assumptions (predefined object classes, simple grasping), or perform poorly on in-the-wild monocular RGB videos due to the complexity of hand-object interaction estimation and profound embodiment gaps. This limits the usability of the vast corpus of monocular human videos for robotic learning, particularly for dexterous manipulation tasks.

Methodology

DO AS I DO comprises a two-stage pipeline: monocular hand-object 4D reconstruction, followed by robust, dynamics-aware retargeting onto dexterous robot hands.

Hand-Object Interaction Reconstruction

The pipeline exploits recent foundation models and robust optimization heuristics to resolve the hand and object state trajectories:

1. Hand Tracking: Utilizes HaWoR [45], a world-space hand pose tracker, to robustly estimate hand pose in diverse video conditions.
2. Object Reconstruction and Tracking: Employs SAM 3D [11], an image-conditioned 3D generative diffusion model, to reconstruct object shape and pose per-frame. Temporal consistency is enforced through a guided-diffusion tracking mechanism which anchors per-frame pose inference to previous-frame pose estimates.
3. Segmentation and Depth Estimation: Combines SAM 3 [64] for mask generation and MoGe [10] for monocular depth and camera intrinsic estimation, facilitating accurate spatial alignment.
4. Hand-Object Alignment: Aligns the reconstructed object to the tracked hand by a depth-consistent translation scaling method, achieving a consistent near-metric 4D trajectory.

Key novelties include the adaptive guidance of object pose estimation, using 2D point tracking to set dynamic guidance strengths for the guided-diffusion tracker, and an efficient pose selection phase using clustering in SE(3), which was empirically validated to match the performance of computationally intensive likelihood-based ranking.

Dynamics-Aware Dexterous Retargeting

Mapping reconstructed human hand motions to robotic actions is executed via a trajectory optimization approach built on parallelizable physics simulation (MuJoCo Warp [13]):

1. Sampling-Based Optimization: Adopts a Model Predictive Path Integral (MPPI)-style annealed sampler [15] to generate dynamically feasible robot hand trajectories tracking the noisy, reconstructed references.
2. Retargeting Innovations: Three key algorithmic innovations were introduced:
   • Warmup Steps: Initial trajectory warmup allows the robot hand to reach feasible pre-grasp configurations, mitigating irrecoverable states arising from reference noise.
   • Random Force Perturbations: Sim-to-real-style domain randomization via sampled forces during rollout, enforcing robustness to reference inaccuracies and simulating real-world uncertainties.
   • Transition Reward: Auxiliary rewards/penalties on critical hand-object contact transitions, specifically penalizing missed grasping or placement events.
3. Real-World Deployment: The resulting trajectories are mapped to the robot's workspace and executed on physical bimanual platforms comprising Sharpa Wave hands and UR3e arms.

Empirical Evaluation

Hand-Object Reconstruction

On benchmarks Dex YCB [71] and HOI4D [72], the method established new SOTA in F-5/F-10 score and Chamfer distance, outperforming image- and video-based SOTA methods ([48, 49, 50, 51, 54]). Human studies on 150 internet and synthetic videos indicated DO AS I DO was preferred 67% of the time over FoundationPose [17]. Specifically, the approach demonstrates superior object pose tracking under occlusions and motion blur, yielding temporally consistent reconstructions where prior methods fail.

Ablation Studies

Adaptive pose guidance and clustering-based sample selection contribute substantially to performance while achieving real-time inference, highlighting the necessity of temporal adaptation and robust sample consensus in 3D object tracking from monocular video.

Dexterous Retargeting

Experiments on both reconstructed in-the-wild references (n=655) and large-scale clean MoCap datasets (OakInk2 [73], n=1352) illustrate a success rate boost from 25% (annealed sampling baseline/SPIDER [15]) to 71% with DO AS I DO (and up to 81% on MoCap data). Key improvements stem from the warmup and transition reward terms. Notably, qualitative improvements in robustness and naturalness of grasps are apparent, with simulated and real-world executions spanning a range of tasks and object types.

Data Curation Insights

A systematic analysis on filtering human internet video datasets (e.g., 100DOH [76]) revealed that less than 5% of raw clips are suitable for dexterous manipulation learning, underscoring the importance of rigorous pre-processing and quality control in scaling human demonstration data for robotics.

Practical and Theoretical Implications

DO AS I DO materially broadens the applicability of monocular human video for generating physically executable robot policies, breaking free from the constraints of specialized capture hardware or curated datasets. By leveraging vision foundation models and robust optimization in physics simulation, the pipeline closes critical gaps in hand-object interaction perception and human-to-robot embodiment mapping. Practically, it enables the construction of large, diverse, and realistic robot manipulation datasets by simply "watching" everyday human activity, with minimal annotation or hardware overhead.

Theoretically, the work demonstrates that reliable 4D hand-object state estimation and behavior cloning is possible even under significant observation noise and model mismatch, provided that robust temporal priors and sim-to-real optimization techniques are employed. The modularity of the pipeline also allows future integration with generalist world models, richer parameterized object priors, or scene-level reasoning frameworks.

Limitations and Future Directions

Primary limitations include the rigid-object assumption and sensitivity to monocular depth ambiguities, which restrict applicability to non-deformable object manipulation and impact precise contact understanding. The approach currently does not reconstruct or reason about broader scene context (e.g., obstacles, articulated environments), which will be vital for generalizing to real-world tasks involving complex hand-scene interactions. Further, real-world deployment fidelity is inherently capped by the accuracy and expressiveness of contemporary physics simulators.

Future research directions include relaxing rigidity constraints (e.g., via generative models for articulated or deformable objects), improving monocular depth and contact estimation, scaling to continuous in-the-wild video streams, and integrating scene-level geospatial priors and affordance reasoning for holistic observation-to-action pipelines. Enhancements in physics engine realism and transfer learning across morphologically distinct robots will further elevate deployment efficacy.

Conclusion

DO AS I DO establishes a robust pipeline for extracting and retargeting dexterous manipulation trajectories from ordinary human videos to multi-fingered robotic hands, setting new empirical benchmarks and validating its effectiveness both in simulation and real-world rollouts. The framework offers substantial promise for dataset scaling and behavior learning in robotic manipulation, given continued advances in 3D perception, optimization under uncertainty, and simulation-to-reality transfer.