Towards Human-level Dexterous Teleoperation

Abstract: Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.

• The paper introduces a novel hand-object co-tracking controller trained via single-stage RL that bridges human intent with contact-rich manipulation.
• It employs a two-stage optimization combining vector-based retargeting and mesh-aware contact optimization, achieving 75.2% average success across seven tasks.
• The approach enables scalable collection of rich dexterous manipulation data, though it remains limited by task-specific training and motion capture reliance.

Towards Human-level Dexterous Teleoperation: A Technical Essay

Motivation and Context

Teleoperation for dexterous robotic hands remains a critical bottleneck in achieving human-like manipulation—robust in-hand reorientation, adaptive tool use, and continuous finger gaiting—all within a single hand. While numerous systems leverage kinematic retargeting or learned action priors for translating human intent to robot commands, these approaches generally fail to capture the dynamic, contact-rich interaction modalities required for advanced dexterity. The paper "Towards Human-level Dexterous Teleoperation" (2607.11481) introduces a novel hand-object co-tracking controller, trained exclusively via single-stage RL, that bridges the gap between operator intent and physically feasible contact execution.

Method Overview

The proposed co-tracking system shifts teleoperation from hand joint tracking to synchronized tracking of both fingertip positions and object pose, allowing robust, zero-shot transfer from simulation to real-world hardware. Given human hand-object reference motions, the pipeline retargets these motions to robot hardware through a two-stage optimization—first vector-based retargeting, then mesh-aware contact optimization.

Figure 1: Method overview. Human hand-object reference motions are used to train a co-tracking controller via single-stage RL, which is deployed for zero-shot real-world teleoperation.

The RL agent is trained to reach consecutive co-tracking subgoals that span various dexterous manipulation modalities, rather than rigid trajectory replication. Hybrid rewards, comprising sparse subgoal-reaching signals and dense tracking terms, eliminate the dependence on manual reward design. Sim-to-real robustness is achieved via domain randomization and random action masking, regularizing the action space against dynamics mismatch and actuator compliance.

Task Suite and Experimental Setup

The evaluation suite comprises seven real-world dexterous manipulation tasks—three in-hand reorientation (CylinderReorient, CuboidReorient, BunnyReorient) and four long-horizon tool-use tasks (HammerUse, BrushSweep, ScrewdriverUse, BulbReplace). Each is decomposed into well-defined stages spanning initial grasp acquisition, in-hand reorientation, functional tool application, and final placement.

Figure 2: Seven dexterous tasks across two categories: reorientation (symmetric, edge/corner, irregular objects) and tool-use (hammer, brush, screwdriver, bulb), each with distinct stage decompositions.

Experiments leverage two dexterous robotic hands (LeapHand and SharpaWave) to validate morphological generalization, and employ a NOKOV MoCap system for real-time operator and object pose tracking.

Numerical Results and Comparative Analysis

The co-tracking controller achieves average success rates of 75.2% and task progress rates of 87.1% across all tasks—a performance unattainable by baselines employing kinematic retargeting or generative action priors. Baseline models consistently fail at the first manipulation stage requiring dynamic in-hand reorientation or finger gaiting, demonstrating the necessity of physically grounded action priors.

Stage-wise analysis Figure 3 reveals that most failures for the co-tracking controller occur at late, contact-intensive stages, while baselines collapse early, particularly at the onset of in-hand reorientation or tool-use transitions.

Figure 3: Stage-wise success curves for four long-horizon tool-use tasks. The co-tracking controller maintains high survival rates through contact-intensive stages, while all baselines collapse early.

Ablation studies highlight the critical impact of consecutive subgoal tracking (as opposed to dense frame-wise tracking) and random action masking. The former enables discovery of physically feasible contact strategies and longer manipulation sequences, while the latter prevents overfitting to simulated actuator response, ensuring robust sim-to-real deployment.

Figure 4: Training ablation curves on Hammer. Both reward and subgoals reached converge more efficiently with the curriculum and SAPG optimizer versus non-curriculum or PPO.

Data Collection and Policy Learning

A central claim is the scalability of the co-tracking controller for collecting rich, contact-intensive expert demonstrations. Behavioral cloning of Diffusion Policies trained on as few as 50 demonstrations achieves non-trivial closed-loop manipulation success rates, especially in long-horizon dexterous tasks like hammering, bulb installation, and sweeping. No existing teleoperation interface yields comparable quality or coverage of expert data.

Failure Modes and Limitations

Three dominant real-world failure modes are identified: interaction perturbation (unseen impact dynamics during tool-environment contact), contact transition jams (actuator compliance or geometric interlocking), and tracking stalls (policy failing to progress due to lack of tactile feedback).

Figure 5: Representative failure modes—interaction perturbation, contact transition jam, and tracking stall—highlight training and sensing limitations.

The framework currently requires object-specific training and retargeting; scaling to unified controllers capable of generalization across object categories is an explicit future direction. Reliance on motion capture also presents a deployment barrier that could be mitigated via vision-based pose estimation.

Practical and Theoretical Implications

Practically, the proposed pipeline establishes a scalable path for teleoperation-mediated collection of extensive dexterous manipulation datasets. Theoretically, the consecutive subgoal formulation serves as a general paradigm for RL-driven learning of dynamic contact strategies, addressing the exploration bottlenecks of dense imitation. The combination of geometrically grounded retargeting and robust RL policy design lays a foundation for future research in generalized dexterous skill acquisition and autonomous policy learning.

Figure 6: Motion capture setup and glove configurations for dense reference motion collection and real-time teleoperation.

Figure 7: Visualization of retargeted hand-object reference motions across morphologically distinct robotic hands, confirming generalization in contact-rich behavior transfer.

Conclusion

The paper advances the capabilities of dexterous teleoperation by introducing a hand-object co-tracking controller trained on geometrically retargeted human demonstrations, utilizing consecutive subgoal tracking, hybrid rewards, and action-space regularization for robust sim-to-real transfer. Experimental results demonstrate strong performance across a spectrum of challenging tasks and support the utility of the approach in collecting high-quality data for subsequent autonomous policy training. Key limitations center on task-specific training and sensing requirements, establishing clear targets for further research in unified dexterous skill learning and markerless deployment.