AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance

Abstract: Teleoperation is a key interface for controlling dexterous robotic hands and collecting demonstrations for imitation learning. Its effectiveness largely depends on kinematic retargeting, which maps operator hand motions to feasible and intuitive robot hand motions. Existing methods often require hand-crafted objectives, precise calibration, or global shape matching between human and robot hand spaces, making them sensitive to hand-specific tuning and less reliable across different dexterous hands. We propose AnyDexRT, a calibration-free retargeting method for intuitive dexterous teleoperation across human-like dexterous hands. AnyDexRT combines self-supervised fingertip correspondence learning with few-shot human guidance to anchor the mapping in task-relevant regions, and further refines pinch-related poses using a contact classifier. Experiments on diverse dexterous hands and real-world teleoperation tasks show that AnyDexRT improves retargeting quality, reduces manual tuning, and provides more intuitive and efficient control than prior retargeting methods. Project website: https://chenxi-wang.github.io/projects/anydexrt

• The paper presents AnyDexRT, a calibration-free framework that leverages few-shot human guidance and an asymmetric Chamfer loss to map human fingertip kinematics to robotic hands.
• It utilizes structure-preserving loss and a dedicated contact classifier to boost local motion consistency and pinch pose accuracy, achieving 90.2% LMC in simulations.
• The method reduces calibration effort and generalizes across diverse robotic hand morphologies, enabling efficient demonstration collection for imitation and reinforcement learning.

AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance

Problem Statement and Motivation

Dexterous teleoperation with anthropomorphic robotic hands remains a critical bottleneck in deploying manipulators to unstructured environments and collecting high-quality manipulation demonstrations for IL and RL frameworks. Conventional retargeting pipelines for translating human operator hand poses into feasible robot hand commands suffer from overreliance on precise calibration, manual tuning of objectives, and brittle global shape alignment. These limitations hinder generalization, increase development effort, and degrade operator intuitiveness.

The paper proposes AnyDexRT, a calibration-free teleoperation retargeting framework for human-like dexterous robotic hands. AnyDexRT leverages self-supervised fingertip-level shape correspondence regularized with a small set of user-provided anchors and employs a contact classifier to enhance pinch pose accuracy. The system aims to improve the intuitiveness, reduce calibration effort, and generalize across diverse dexterous end-effectors without morphology-specific manual retargeting.

System Overview

AnyDexRT decomposes the hand retargeting mapping into a composite of a geometric fingertip mapper and an IK-based or data-driven joint solver. The core insight is to learn the correspondence between the human and robot fingertip kinematic spaces using an asymmetric, structure-preserving loss without enforcing full global coverage. Few paired human-robot anchor poses are provided to resolve mapping ambiguities. A dedicated fingertip contact classifier further refines contact-critical (pinch) configurations, compensating for sensor and pose encoding limitations.

The training and deployment pipeline is illustrated in (Figure 1).

Figure 1: System overview of AnyDexRT, showing human-robot fingertip mapping learned with self-supervision and few-shot anchors, and contact classifier refinement for improved manipulation at deployment.

Retargeting Objectives and Algorithmic Innovations

AnyDexRT’s approach to the correspondence problem comprises several key innovations:

• Asymmetric Chamfer Loss: Partial Chamfer distance encourages surjective but not bijective mapping from human fingertip space to robot fingertip space, avoiding the pitfalls of forcing the entire (possibly redundant) robot workspace to be covered (Figure 2b).
• Pairwise Distance Preservation: A geometric regularizer ensures that local and global relational structure of the human fingertip manifold is preserved under the mapping (Figure 2c).
• Local Motion Consistency: By preserving motion directions in the local frame (rather than the global robot base), the mapping is robust to calibration error and maintains operator intent even under sensor/coordinate frame misalignment (Figure 2d).
• Few-Shot Anchor Alignment: Sparse paired anchor data from reference gestures are used to pin the mapped region in the robot’s workspace, overcoming the underdetermined solution space and resolving translation/scale ambiguities (Figure 2e).

The composition of these objectives and their impact on mapping quality are depicted in (Figure 2).

Figure 2: Impact of retargeting objectives on the quality of the human-to-robot fingertip mapping. Each ablation highlights the criticality of structure and anchor-based regularization.

Contact-critical poses (e.g., pinches) are handled by a classifier trained to detect thumb-fingertip contacts from the human hand data, then refine the corresponding robot hand pose via nearest-neighbor search in the robot’s pose database.

Experimental Results

Simulation Benchmarks

AnyDexRT is validated across seven kinematically diverse, human-like dexterous hands (6–20 DoFs) in simulation. Assessment focuses on local motion consistency (LMC; operator-aligned intent regardless of calibration), global motion consistency (GMC; for ideal calibration), retargeting stability, and calibration sensitivity. AnyDexRT achieves a mean LMC of $90.2\%$, a substantial improvement over reference methods (GeoRT $59.8\%$, optimization baseline $52.2\%$). It maintains mapping quality under significant coordinate perturbations, confirming its calibration-free property. Standard deviation analysis over random seeds shows AnyDexRT’s order-of-magnitude greater mapping stability relative to GeoRT.

Teleoperation Performance

In real-robot experiments using a Wuji Hand on a Flexiv manipulator with a vision-based Manus glove, AnyDexRT consistently outperforms baselines in operator task efficiency and pinch manipulation reliability. Across tasks involving fine control (bottle triggering, lightbulb screwing, spatula grasping, and tiny-object picking), AnyDexRT delivers lower task completion times and significantly higher pinch success rates (62.0% vs. 39.6%/29.2%). These results confirm improved intuitiveness, faster demonstration collection, and higher-fidelity contact pose reproduction.

Figure 3: Qualitative real-world results with AnyDexRT; intuitive pose retargeting is maintained across gesture types and teleoperation tasks.

Practical Implications and Theoretical Impact

The calibration-free retargeting paradigm of AnyDexRT addresses a longstanding issue in dexterous teleoperation—brittleness to hand-specific tuning and calibration artifacts. By separating structure-preserving self-supervised correspondence from user-anchored alignment, the method generalizes across robot embodiments with minimal design effort. Practically, this facilitates rapid deployment of new human-like hands, scalable demonstration collection, and operator customization via a handful of intuitive anchor gestures.

Theoretically, the demonstrated stability and invariance to calibration mismatch highlight the importance of local structure preservation and minimal supervision in correspondence learning for control, extending beyond prior global matching trajectories. The contact classifier’s success suggests a generalizable mechanism for robust teleoperation where state estimation is ambiguous or sensor coverage is incomplete.

Future Directions

Potential improvements include:

• Automating anchor selection via online policy adaptation or user feedback integration,
• Generalizing contact classification and pose refinement to richer multi-finger, multi-contact manipulation regimes,
• Downstream assessment by training manipulation policies (imitation, offline RL) directly from AnyDexRT-collected demonstrations.

These efforts will further close the gap between teleoperated control fidelity and effective autonomous generalization.

Conclusion

AnyDexRT establishes a robust, calibration-free hand retargeting pipeline that relies on self-supervised structure- and motion-preserving mapping, efficiently anchored by user-provided paired poses and refined through contact-aware inference. Empirical results demonstrate highly intuitive operator experience, strong generality across morphologies, high-quality demonstration efficiency, and strong dataset suitability for learning-based robotic dexterous manipulation.