Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video

Abstract: Human manipulation videos are a convenient and intuitive source for robot learning. However, directly transferring human dexterity to robots remains challenging due to perception errors and embodiment gap. To address this, we introduce Video2Sim2Real, a full-stack framework for autonomous skill acquisition from a single human manipulation video. Our framework first uses off-the-shelf foundation models to reconstruct a simulator-ready digital twin and extract robot and object motion priors. Rather than treating the extracted robot motion as a reliable reference throughout execution, our key idea is to recover and leverage the most fundamental sources of supervision from the demonstrated skill: We identify object-centric keyframes to optimize the corresponding robot configurations using object information from the simulator, and use these configurations as anchors that refine the robot motion such that it ultimately has the desired impact on the environment. To bridge the remaining sim-to-real gap, we introduce a sim-to-real strategy that decouples robustness to noisy and incomplete perception from variations in hand-object interaction dynamics. Specifically, we learn to recalibrate robot configurations from noisy real-world point clouds via IL, and leverage residual RL to perform local finger-level adaptations to ensure for robust and effective interactions. Finally, a collision-aware motion planning module enables spatial generalization to novel object configurations. Across several everyday manipulation tasks, Video2Sim2Real improves simulated task success, safety, and trajectory coherence over numerous baselines, and achieves better sim-to-real transfer than existing techniques. These results demonstrate a promising path toward autonomous dexterous skill acquisition from human videos.

• The paper introduces a full-stack pipeline that autonomously extracts dexterous manipulation skills from one human demonstration, eliminating the need for robot-side data or manual rewards.
• It employs digital-twin scene reconstruction, object-centric keyframe-based trajectory refinement, and decoupled IL and RL to bridge perception and embodiment gaps.
• Experimental results demonstrate over 95% real-world task success with enhanced safety and smooth motion compared to standard sim-to-real and imitation learning baselines.

Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video

Introduction and Motivation

Dexterous robotic manipulation learning remains impeded by requirements for extensive task-specific supervision and the challenge of bridging the human–robot embodiment gap. "Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video" (2606.08828) presents a full pipeline that autonomously acquires dexterous manipulation skills from a single human demonstration video, without utilizing any robot-side demonstrations, manual rewards, or expert annotation. The framework leverages advancements in foundational vision models and sim-to-real adaptation techniques, achieving reliable real-world dexterous manipulation across several daily tasks.

Figure 1: Video2Sim2Real autonomously acquires dexterous manipulation skills end-to-end from human manipulation videos, without robot data or expert intervention, across different everyday manipulation tasks.

System Architecture

The Video2Sim2Real pipeline is modular and proceeds through four major stages: digital-twin scene reconstruction, object/hand motion estimation and retargeting, keyframe-based robot trajectory refinement, and a decoupled sim-to-real policy learning scheme. The pipeline introduces a keyframe-anchored object-centric optimization for robot trajectory correction and decouples geometric and physical gap adaptation by splitting sim-to-real policy learning into IL and RL components.

Figure 2: System overview—human video yields a simulator-ready digital twin and motion priors; keyframes are refined and interpolated into an executable robot trajectory; a learned decoupled sim-to-real policy adapts real-world execution; an optional planning module offers further generalization.

Scene Understanding and Motion Extraction:

The pipeline first employs foundation models (e.g., Gemini, SAM3, SAM3D, HaMeR, CoTracker) for digital-twin construction of the scene and segmentation/object mesh extraction. Both the human hand and manipulated object trajectories are then estimated in 3D space and retargeted to robot embodiment via differentiable inverse kinematics, with necessary coordinate unification.

Object-Centric Keyframe-Based Trajectory Refinement:

Rather than naively trusting retargeted trajectories, Video2Sim2Real identifies three manipulation keyframes per task—contact, interaction, and detachment—using object flow cues. At each keyframe, the robot configuration is optimized in the digital twin, aligning the robot's kinematics and contact to object-centric manipulation intent. Grasp optimization uses geometry-aware corrections; contact frames for non-prehensile tasks establish contact direction via object motion. Inter-keyframe motions are interpolated for temporal coherence.

Sim-to-Real Decoupled Policy Learning

Standard reinforcement or imitation-based sim-to-real approaches are shown to be non-robust under perception/modeling stochasticity, especially in high-DoF dexterous systems. Video2Sim2Real introduces a decoupled learning framework:

• Imitation Learning (IL): Handles geometry gap by mapping simulated/noisy real point clouds to keyframe robot hand poses. This global adaptation is deployed once per episode.
• Residual Reinforcement Learning (RL): Handles local physics gap via online residual corrections on finger joints during execution, trained under randomized simulation of robot/object physical parameters.

This architecture explicitly separates geometric and physical sim-to-real transfer concerns (addressing both domain and observation mismatch), leading to improved reliability, stability, and safety in real-world deployment.

Experimental Validation

Digital Twin and Policy Robustness

Video2Sim2Real is validated on seven real-world manipulation tasks (fruit placement, seasoning, object handover, etc.), using a 7-DoF Kinova arm with an anthropomorphic Leap Hand. Each experiment begins with only a single human demonstration video (RGB-D), and the pipeline operates without human-in-the-loop intervention.

Figure 3: Visualization of each task: human demonstration, robotic simulation in the digital twin, and final physical deployment. The digital twin closely aligns with the real scene; refined trajectories enable robust transfer.

Performance is compared against five categories of refinement/sim-to-real baselines, ranging from residual RL over retargeting to object-centric RL without human motion cues. Video2Sim2Real yields significantly higher simulated and real-world success rates, safety, and trajectory coherence metrics.

Quantitative Results

Figure 4: Average simulation results: Video2Sim2Real achieves dominant task success rate, competitive safety, shortest completion time, and lowest trajectory jerk under identical sim randomizations.

Key findings:

• Task Success: Video2Sim2Real demonstrates a mean simulated task success rate above 90% and real-world averaged success above 95% across tested tasks, outperforming strong RL and IL baselines.
• Safety and Smoothness: Achieves 100% safety rate (no unintended collisions) in simulation, with RMS trajectory jerk markedly below alternative approaches.
• Policy Ablations: Both IL and RL components are required for robust deployment—either alone are insufficient for general task reliability.
• Failure Analyses: Baselines (RL, pure-IL) suffer from unsafe/unnatural behaviors due to policy overfitting, poor embodiment bridging, or sim-to-real mismatch.

  Figure 5: Contact-frame robot configurations are materially improved after refinement, more precisely reflecting task intention and physical feasibility.

  

  Figure 6: Human-to-robot retargeting alone does not resolve embodiment mismatches or correct for hand–object occlusions—refinement is essential.

  

  Figure 7: RL-based baselines frequently yield unsafe or jerky behavior, causing low success rates and greater hardware risk.

Real-World Transfer and Generalization

Robust sim-to-real transfer is validated by shuffling object positions within perturbation bounds: Video2Sim2Real policies maintain high success rates where pose-estimation-based baselines collapse. The framework supports spatial generalization via collision-aware planning, leveraging the reconstructed digital twin for wider scene reconfiguration.

Figure 8: Robustness in the real world is systematically validated by local perturbation of object positions—Video2Sim2Real adapts without intervention.

Figure 9: Scene reconstructions for spatial generalization: obstacles and clutter are accurately encoded in the digital twin for planning.

Figure 10: Robust collision-free, feasible robot trajectories are generated for novel task configurations, not present in the demonstration video.

Implications and Future Directions

Practical Considerations

Video2Sim2Real demonstrates the viability of automating dexterous skill acquisition from passive human videos for everyday tasks, reducing human annotation bottlenecks and enabling scalable deployment of dexterous robots. The end-to-end pipeline is modular, supports broad task and scene diversity, and avoids brittle reliance on accurate pose estimation at execution time.

Theoretical Implications

The explicit decoupling of geometry- and physics-gap adaptation in sim-to-real policy learning could serve as a general template for transferring complex skills to robots across heterogeneous embodiments. Keyframe-based object-centric supervision strikes a balance between imitation of human priors and reward-driven reinforcement, mitigating exploration and embodiment mismatches typical in high-DoF dexterous manipulation.

Limitations and Prospects

• Keyframe-based refinement offers sparse temporal correction; continuous or variable-length in-hand manipulation remains a challenge.
• Keyframe identification is currently heuristic; foundation-model-based motion analysis could yield greater generality.
• The method assumes unarticulated manipulated objects in scene reconstruction—future pipelines incorporating articulated/deformable object assets are a natural extension.
• Integration of tactile feedback into residual RL or fully neural field-based scene representations may drive further sim-to-real performance.

Conclusion

Video2Sim2Real realizes autonomous dexterous skill acquisition from a single human video, efficiently bridging the perception and embodiment gaps inherent to human-to-robot transfer. The system's architectural contributions—object-centric keyframe refinement and decoupled sim-to-real policy learning—achieve robust, safe, and generalizable manipulation, pointing toward scalable learning-from-observation for real-world, high-DoF robots.

(2606.08828)