ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

Abstract: Learning robot dexterous manipulation from human manipulation videos requires reliably retargeting human intent to executable robot actions while maintaining stable hand-object contact, which remains a key challenge in embodied intelligence. Existing retargeting methods often ignore explicit contact modeling or rely on reinforcement learning, resulting in limited accuracy and generalization. To address this, we propose ObjRetarget, a human-to-robot motion retargeting framework for learning robot dexterous manipulation from human videos, which integrates anthropomorphic arm trajectory constraints with structured hand-object geometric modeling. For arm motion, reference trajectories extracted from human videos are used for initialization, followed by anthropomorphic constraints and redundancy-aware optimization to generate natural and accurate movements. For hand manipulation, ObjRetarget represents multi-finger contacts using polytope clusters and preserves contact structure through geometric invariants to improve stability. Experiments on real robots show that ObjRetarget improves manipulation success rates and contact stability across multiple dexterous tasks, and generalizes well to different demonstrations, object poses, and task settings.

• The paper’s main contribution is integrating anthropomorphic arm constraints with polyhedral hand modeling to robustly retarget human dexterous manipulation onto robots.
• It employs a two-stage process that refines RGB-D extracted trajectories with task-adaptive regularization, ensuring natural and accurate robot motions.
• Experimental evaluations across six tasks demonstrate superior success rates and stability metrics compared to prior methods, validating its robust design.

ObjRetarget: A Structured, Object-Centric Framework for Human-to-Robot Dexterous Manipulation Retargeting

Introduction and Motivation

Transferring dexterous manipulation skills from humans to robots demands more than mere kinematic pose replication; it necessitates precise transference of contact semantics, hand–object interaction geometry, and anthropomorphic motion patterns. Existing motion retargeting and imitation frameworks often either neglect contact stability or rely on data-driven RL that suffers from generalization gaps and intensive reward shaping requirements. The ObjRetarget framework addresses these deficiencies by integrating anthropomorphic arm constraints with explicit geometric contact modeling, facilitating robust and generalizable robot execution from human demonstration videos.

Core Framework Architecture

ObjRetarget systematically segments the retargeting problem into arm and hand subspaces, enabling tailored modeling and optimization for each with explicit coupling at the execution stage. The key design choices include:

• Arm motion retargeting leverages reference trajectories extracted from RGB-D human demonstration videos. These trajectories initialize robot arm motion, which is then refined using anthropomorphic geometric constraints and task-adaptive regularization. This two-tier strategy ensures the resulting motion is both physically feasible and semantically aligned with human intent.
• Hand manipulation retargeting employs a polyhedral (polytope-cluster) representation for hand–object contact states. Fine-grained multi-finger contacts are extracted, and geometric invariants enforce preservation of local contact semantics. Unlike previous simple pose mapping methods, this approach models the structure and meaning of contact, which is critical for robust manipulation in cluttered and varied task domains.
• Unified temporal scheduler synchronizes arm and hand motions, enabling coordinated, multi-modal dual-arm manipulation even in tasks requiring complex temporal and spatial synchronization.

Figure 1: Schematic of the motion retargeting framework, highlighting differences between conventional arm/hand mapping and the ObjRetarget polyhedral modeling with anthropomorphic constraints.

Figure 2: Overview of the ObjRetarget architecture, from pose and object extraction to refined, synchronized trajectories.

Anthropomorphic Arm Constraint Formulation

The framework employs a two-stage motion retargeting process. First, reference trajectories are derived via encoder–decoder mappings from the human pose manifold to the robot configuration space, with explicit loss terms ($L_{pos}$, $L_{ori}$, $L_{tip}$) enforcing end-effector and anthropomorphic fidelity. Second, a soft, task-adaptive arm-plane constraint regularizes the instantaneous configuration of the robotic arm. Specifically, the normal vector of the plane formed by the shoulder, elbow, and wrist is adaptively aligned with the local arm bending direction inferred from current motion trends rather than clinging to a static anthropomorphic prior. This soft constraint is weighted to prioritize anthropomorphism during large-scale movements, attenuating its influence during fine contact-sensitive phases for maximum versatility.

Figure 3: Illustration of the task-adaptive arm-plane regularization, depicting arm-plane normal calculation and its alignment with motion intent.

A task-oriented end-effector tracking loss further guarantees the spatial–temporal accuracy of wrist pose trajectories. The joint optimization ensures convergence to human-like, natural, and task-consistent robot motion even under significant hardware kinematic disparities and perception noise.

Polyhedral Hand–Object Contact Modeling

Unlike holistic hand-pose transfer approaches, ObjRetarget extracts finger-object adjacency graphs, representing each contact as a tetrahedral polytope consisting of the palm center, two fingertips, and the object contact point. This local, contact-centric model is robust to global pose errors and preserves manipulation semantics even with incomplete or ambiguous input. The use of edge-length and relative-pose invariants guarantees that robots preserve not only the kinematic structure but also the functional intent of human finger coordination during manipulation tasks.

Nonlinear optimization is employed to solve for robot hand joint configurations that minimize deviations from the human-derived geometric constraints, subject to hardware limits. Upon contact events, the system switches from initialized trajectory tracking to geometric-invariant optimization for precise, stable grasp and manipulation.

Experimental Evaluation

Comprehensive experiments were conducted with the RealMan dual-arm platform equipped with dexterous multi-DoF Inspire hands. ObjRetarget was assessed across a spectrum of six manipulation tasks, including sequential placement, soft object grasping, drawer closing, fruit transfer, liquid pouring, and bimanual coordination.

Figure 4: Qualitative visualization of ObjRetarget across six dexterous manipulation benchmarks; top: human, bottom: corresponding robot execution.

Key numerical findings include:

• Task success rate: ObjRetarget achieves a mean of 75.8% across all tasks, outperforming OKAMI (61.6%) and ORION (50.8%). Especially notable are gains in multi-stage and contact-critical tasks (e.g., Pour Water, Drawer Close, Bimanual Place), where explicit contact modeling is essential.
• Stability metrics: In ablation studies, the full ObjRetarget configuration exhibited the lowest object slip (0.012 m), lowest geometric inconsistency (0.081), and highest task reliability (75.8%). Removal of polyhedral contact models or anthropomorphic arm constraints led to significant performance drop-offs, including elevated slip, loss of geometric structure, and decreased task completion rates.
• Motion fidelity: ObjRetarget produces minimal mean per-joint position error (MPJPE: 0.088 m), quaternion error (0.192 rad), and Fréchet trajectory distance (0.124 m), substantiating its ability to accurately and consistently reproduce human-like motions.

Figure 5: Demonstration of ObjRetarget's generalization capability across demonstrators with diverse hand sizes and manipulation styles.

Strong generalization is confirmed by stable success rates when conditioned on distinct demonstrator styles and physique, a property not matched by baselines.

Implications and Future Directions

From a practical perspective, ObjRetarget enables reproducible, contact-robust, anthropomorphic robotic manipulation from unstructured human video, eliminating the need for specialized teleoperation or extensive RL rollout. Its modular, object-aware architecture readily adapts to varied object geometries, task complexity, and inter-individual demonstration variability, suggesting immediate applicability to industrial and service robots operating in dynamic, cluttered, or collaborative environments.

Theoretically, the hand–arm decoupling paradigm with explicit polyhedral contact modeling sets a foundation for more nuanced, semantic, and physically-grounded motion transfer in high-DoF systems. ObjRetarget's success demonstrates the efficacy of compositional, geometry-centric constraint integration versus monolithic, end-to-end RL for dexterous manipulation generalization.

Anticipated avenues for extension include:

• Dynamic feedback integration: Incorporation of tactile/force sensing for real-time correction during contact, further improving robustness under perception imprecision and non-rigid object interaction.
• Whole-body retargeting: Scaling from upper limb to full-body anthropomorphic transfer for mobile or humanoid platforms.
• Closed-loop learning: Leveraging ObjRetarget as an expert demonstrator to bootstrap RL or imitation learners with structured, contact-stable demonstrations.

Conclusion

ObjRetarget delivers a principled, generalizable solution for human-to-robot motion retargeting by structurally decoupling and optimizing arm and hand subspaces. Through polyhedral contact modeling and anthropomorphic regularization, it enables robust, natural, and contact-stable dexterous manipulation across diverse tasks, object configurations, and demonstrator styles. The framework materially advances the field toward practical, scalable robot skill learning from human video and provides a template for future research in explicit, geometry-aware retargeting (2607.03828).