Dexterous Manipulation Policies from RGB Human Videos via 4D Hand-Object Trajectory Reconstruction

Abstract: Multi-finger robotic hand manipulation and grasping are challenging due to the high-dimensional action space and the difficulty of acquiring large-scale training data. Existing approaches largely rely on human teleoperation with wearable devices or specialized sensing equipment to capture hand-object interactions, which limits scalability. In this work, we propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos. Leveraging recent advances in computer vision, VIDEOMANIP reconstructs explicit 4D robot-object trajectories from monocular videos by estimating human hand poses, object meshes, and retargets the reconstructed human motions to robotic hands for manipulation learning. To make the reconstructed robot data suitable for dexterous manipulation training, we introduce hand-object contact optimization with interaction-centric grasp modeling, as well as a demonstration synthesis strategy that generates diverse training trajectories from a single video, enabling generalizable policy learning without additional robot demonstrations. In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand. In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand, outperforming retargeting-based methods by 15.87%. Project videos are available at videomanip.github.io.

• The paper introduces VideoManip, a pipeline that reconstructs metric 4D hand-object trajectories from single monocular RGB videos, retargets them to robotic hands, and trains manipulation policies without robot demonstrations or wearables.
• Contact optimization raises simulated grasp success from 30.7% to 63.75%, while added video viewpoints increase overall performance to 70.25% across 20 objects under force disturbances.
• The resulting policies achieve 62.86% average success across seven real-world tasks on a LEAP Hand, but fixed hand-object contact, static cameras, compounded vision errors, and limited force reasoning constrain broader deployment.

VideoManip is a framework for training multi-fingered robotic hand grasping and manipulation policies using only RGB human videos as supervision. Rather than relying on teleoperation rigs, wearables, motion-capture systems, or robot demonstrations, the system reconstructs explicit 4D (space-time) hand–object trajectories from monocular video, retargets them to a robotic hand embodiment, and uses the resulting data to train grasping and visuomotor policies. The central empirical claim is that this pipeline yields deployable dexterous behaviors from a single human video per object or task: 70.25% grasp success across 20 objects in simulation with an Inspire Hand, and an average 62.86% real-world success rate across seven manipulation tasks on a LEAP Hand, outperforming a retargeting-based generative-video baseline by 15.87 percentage points.

Reconstruction pipeline

The system operates under deliberately restrictive assumptions: one static-camera egocentric video per task or object, no pre-scanned meshes, no depth sensor, no camera intrinsics, and no robot data or wearable devices. Reconstruction proceeds in stages built entirely from off-the-shelf 3D vision components. MoGe-2 supplies metric depth maps and camera intrinsics, establishing a joint metric frame that anchors all subsequent alignment (2602.09013). SAM 2 produces object masks; MeshyAI generates a full 3D object mesh from masked crops — a deliberate departure from prior work that tracks only 2.5D point clouds — and HaMeR recovers per-frame human hand pose and shape parameters.

Two technical problems receive dedicated treatment. First, generated meshes lack metric scale, which breaks pose estimators such as FoundationPose. The paper addresses this with a two-stage scale estimation: GPT-4.1 provides a coarse physical dimension estimate, then candidate scalings between 0.5× and 2× are scored by rendering each against the SAM 2 masks under the predicted depth, selecting the minimum-error scale. Second, HaMeR's weak-perspective model introduces depth ambiguity; the authors correct hand depth by averaging MoGe-2 metric depths at the 2D hand keypoints so that hands and objects share a single depth reference. Retargeting follows AnyTeleop-style optimization matching robot link keypoints to human joints.

For in-the-wild videos, where no camera–robot extrinsic calibration exists, the pipeline estimates gravity direction from the first frame via GeoCalib and applies the recovered rotation ${}^{\text{grav}}R_{\text{cam}}$ to all meshes and configurations. This does not recover the full camera-to-world transform, but the ablation shows it is sufficient: without gravity alignment the in-the-wild Pour Tea task drops to 0% success, while with it, in-scene and in-the-wild versions of the task perform comparably. This is arguably the paper's most consequential calibration result, since it establishes that approximately egocentric internet-style videos can be aligned to a physically meaningful reference frame well enough for policy learning.

Contact optimization and interaction-centric grasping

Reconstructed trajectories capture correct gross motions but are frequently physically infeasible — interpenetration or missed contact arises from errors in mesh geometry and hand shape estimation. VideoManip addresses this by applying ContactOpt during the grasp stage $[t_1, t_2]$: differentiable contact maps over hand and object vertices are computed, target contact distributions are predicted, and the hand pose is optimized to minimize their discrepancy. The optimized poses are retargeted to robot configurations and used to train DRO, which learns a dense point-to-point distance matrix $\mathcal{D}(\mathcal{R}, \mathcal{O})$ between robot-hand and object point clouds; grasp poses are recovered from the predicted matrix via multilateration.

The quantitative impact of contact optimization is large. Training the same DRO model on unoptimized grasps yields only a 30.7% average success rate across 20 objects versus 63.75% with ContactOpt-refined grasps — roughly doubling grasp performance from the correction alone. This strongly supports the paper's position that explicit contact-aware modeling, rather than direct retargeting of raw reconstructed poses, is necessary for reconstruction-based supervision to be usable.

Demonstration synthesis and manipulation learning

Because one video yields only one trajectory, the authors decompose each trajectory into a grasp phase and a manipulation phase and apply DemoGen to synthesize additional demonstrations via SE(3) spatial randomization of the object, preserving hand–object contact and finger motions. Manipulation policies use DP3, taking hand point clouds at the grasp pose, proprioceptive state, and object point clouds as input, outputting configuration deltas executed closed-loop. A practical concession appears here: because the LEAP Hand occludes the grasped object's point cloud during rollout, the method assumes the hand–object relative pose established at grasp remains fixed throughout execution, propagating the object point cloud via the recorded grasp-time transformation. This approximation suffices for the evaluated tasks but would not cover tasks involving regrasping or significant in-hand slip.

Experimental results

Grasping. Trained on 20 single-view videos (3–6 s each, 30 fps), the DRO model achieves 82.13% success over the 15 successfully learned objects and 63.75% overall in IsaacGym, evaluated under six-directional force disturbances. Predicted grasps reproduce human grasp strategies (e.g., middle finger atop a spray-bottle trigger). Five objects fail, attributable to pose-estimation errors under heavy hand occlusion (Pan) and kinematically plausible but non-force-closure grasps (Glasses Case, Hand Bag). Adding two extra videos per failed object raises those five objects' aggregate success from 8.6% to 40.8% and overall performance from 63.75% to 70.25%, indicating that viewpoint diversity alone substantially mitigates reconstruction failures. Notably, the Hand Bag does not improve at all (0/100 in both conditions), showing that diversity-based augmentation is not uniformly effective when failure stems from disturbance sensitivity rather than reconstruction quality.

Real-world manipulation. On a LEAP Hand mounted on a 7-DoF xArm, seven tasks are trained from one video each — three from in-scene videos (Pour Tea, Close Drawer, Pick&Place Can) and four from in-the-wild videos (Pour Tea, Hang Hat, Screw Bulb, Move Jenga Box). DP3 policies trained on 1,000 DemoGen-synthesized trajectories achieve an average 62.86% success rate over ten trials per task, versus 7/10 best-case results for the LVP baseline and near-zero for zero-shot $\pi_{0.5}$ fine-tuned on disjoint LEAP Hand tasks. LVP exhibits characteristic generative-model failures — hallucinating grippers instead of hands, confusing left/right, producing infeasible grasps — particularly when the initial observation lacks a visible human hand, whereas VideoManip grounds supervision in explicit geometry. An ablation varying synthesized-trajectory count from 1 to 1,000 shows evaluation success climbing steadily from 1/15 to 13/15 on Pour Tea with randomized bowl placement, directly demonstrating that DemoGen augmentation drives spatial generalization. The authors concede that synthesis helps less for tightly coupled tasks such as Screw Bulb and Close Drawer, though location randomization still aids generalization.

Limitations and open questions

The paper is explicit about three constraints. First, the pipeline composes multiple imperfect off-the-shelf vision models whose errors compound; the current workaround — recording videos from viewpoints amenable to reconstruction — means the system does not yet handle arbitrary downloaded internet videos, despite the framing around in-the-wild data. A trajectory verification module for filtering bad reconstructions is proposed but not implemented. Second, the static-camera assumption excludes dynamic capture, and third, the point-cloud representation with fixed hand–object contact limits applicability to tasks requiring regrasping or handling severe occlusion; image-based trajectory representations are suggested as an alternative but left unexplored. An additional open question concerns force awareness: several grasp failures were kinematically valid yet lacked force closure, and the framework has no mechanism for reasoning about required grasp forces.

Conclusion

VideoManip demonstrates that contact-optimized, gravity-aligned 4D trajectory reconstruction from ordinary RGB videos can serve as sole supervision for multi-fingered dexterous manipulation, achieving competitive real-world performance without wearables, robot demonstrations, or object scans. The evidence indicates that explicit geometric reconstruction combined with contact refinement and synthetic demonstration generation provides more reliable training signal than video-generation-based retargeting, while leaving open whether the pipeline survives fully unconstrained internet video, dynamic cameras, and contact-rich tasks beyond its fixed-grasp execution model.