AnyDexRT: Calibration-Free Hand Retargeting

• AnyDexRT is a calibration-free kinematic retargeting method that maps human fingertip positions to feasible robot joint configurations.
• It leverages self-supervised fingertip correspondence, few-shot human anchor guidance, and contact-aware pinch refinement to ensure intuitive and reliable teleoperation.
• Empirical evaluations on diverse human-like dexterous hands reveal improved local motion consistency, reduced tuning needs, and high real-time throughput.

AnyDexRT is a calibration-free kinematic retargeting method for dexterous hand teleoperation that maps operator hand motions to feasible and intuitive robot hand motions across human-like dexterous hands. Introduced in "AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance" (Wang et al., 9 Jul 2026), it is designed to address limitations of prior retargeting approaches that rely on hand-crafted objectives, precise calibration of coordinate frames, scale factors, and offsets, or global shape matching assumptions between human and robot hand spaces. The method combines self-supervised fingertip correspondence learning, few-shot human guidance, and contact-aware refinement for pinch, with the stated goals of calibration-free operation, intuitive control, generality across diverse human-like dexterous hands, and reduced hand-specific tuning.

1. Problem setting and design rationale

Teleoperation is defined as real-time control of a robot hand with an operator’s own hand motion while observing the robot’s behavior. Within that interface, kinematic retargeting is the mapping from human hand signals to robot hand joint commands. AnyDexRT is formulated around the observation that prior approaches often depend on hand-crafted objectives and task vectors, precise calibration, or global shape matching, each of which introduces failure modes under cross-hand variation or sensor misalignment (Wang et al., 9 Jul 2026).

The motivating failure modes are specific. Hand-crafted objectives and task vectors require tedious hand-specific tuning. Precise calibration of coordinate frames, scale factors, and offsets causes performance degradation under misalignment. Global shape matching assumptions force the robot’s entire fingertip space to be covered by the human space, which can distort task-relevant regions when the robot has redundant motion regions. Hand-specific tuning further limits transfer across dexterous hands. AnyDexRT is positioned against these constraints by requiring only human-to-robot coverage, not the converse, and by emphasizing local motion preservation rather than dependence on a shared global frame.

A central conceptual point is that the method operates on fingertips rather than explicit human joint angles. Human input is represented as 3D fingertip positions from a glove; robot output is ultimately a valid joint configuration. This suggests a deliberate abstraction: the retargeting interface is anchored at the end-effector level of each finger, while kinematic feasibility is enforced downstream through valid robot joint states.

A common misconception is that “calibration-free” implies the complete absence of operator-specific information. In AnyDexRT, calibration is avoided by design, aided by local-motion objectives and anchor guidance, but the method still requires few-shot human anchors. Another misconception is that the method seeks full global shape equivalence between human and robot fingertip spaces. The paper explicitly rejects that requirement and instead uses asymmetric partial coverage from human to robot.

2. Mathematical formulation and system decomposition

AnyDexRT decomposes retargeting into a fingertip mapping and a solver. For $F$ fingers, the human fingertip position space is $C^H \subset \mathbb{R}^{F \times 3}$, the robot fingertip position space is $C^R \subset \mathbb{R}^{F \times 3}$, and the robot joint space is $J^R \subset \mathbb{R}^{D}$ with joint limits $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$ (Wang et al., 9 Jul 2026).

The retargeting mapping is defined as

$f_m: C^H \rightarrow C^R,$

followed by a solver

$f_s: C^R \rightarrow J^R.$

The overall mapping is written as

$g_\theta = f_s \circ f_m,$

with

$$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$

Robot fingertip feasibility is described through forward kinematics. For finger $i$, forward kinematics $C^H \subset \mathbb{R}^{F \times 3}$0 yields fingertip position $C^H \subset \mathbb{R}^{F \times 3}$1, and the fingertip stack is

$C^H \subset \mathbb{R}^{F \times 3}$2

The paper states the generic chain form

$C^H \subset \mathbb{R}^{F \times 3}$3

with $C^H \subset \mathbb{R}^{F \times 3}$4. Exact chain and DH parameters depend on the target hand and are not provided.

In the reported implementation, kinematic feasibility is enforced implicitly by nearest-neighbor lookup in a database of valid robot joint configurations rather than by explicit IK. The practical inference problem is described as choosing joint commands consistent with target fingertips through nearest-neighbor search over precomputed valid joint states. An optional continuity term is described in the deep dive as

$C^H \subset \mathbb{R}^{F \times 3}$5

but the paper notes that no exact equation is provided in the paper text and that the core implementation uses nearest neighbor and the last joint state without an explicit $C^H \subset \mathbb{R}^{F \times 3}$6 term.

This decomposition is significant because it isolates correspondence learning from the robot-specific joint solver. A plausible implication is that the same fingertip-level correspondence machinery can be reused for multiple hands so long as a valid fingertip-to-joint realization mechanism is available.

3. Self-supervised fingertip correspondence and few-shot anchoring

The core learning problem is per-finger mapping from human fingertip samples to robot fingertip positions. AnyDexRT learns one mapper $C^H \subset \mathbb{R}^{F \times 3}$7 per finger using self-supervision and sparse paired anchors. Human fingertip samples are collected from the glove and normalized per finger by centralization and axis-range normalization to $C^H \subset \mathbb{R}^{F \times 3}$8 using the axis of maximum range, explicitly to avoid geometric distortion. Robot fingertip samples are generated in simulation by uniformly sampling robot joint configurations per finger to cover reachable fingertip spaces; these samples are also used to synthesize anchors and pinch templates (Wang et al., 9 Jul 2026).

The self-supervised correspondence objective has four components. The first is asymmetric partial Chamfer loss, defined for each finger as

$C^H \subset \mathbb{R}^{F \times 3}$9

Its asymmetry is methodologically central: it ensures human-to-robot coverage without forcing robot-only redundant regions to be matched.

The second component is pairwise distance preservation,

$C^R \subset \mathbb{R}^{F \times 3}$0

which regularizes mapped data geometry.

The third component is local motion preservation. Let $C^R \subset \mathbb{R}^{F \times 3}$1 be the local coordinate frame at position $C^R \subset \mathbb{R}^{F \times 3}$2, and let $C^R \subset \mathbb{R}^{F \times 3}$3 be a small perturbation around $C^R \subset \mathbb{R}^{F \times 3}$4. Then

$C^R \subset \mathbb{R}^{F \times 3}$5

and

$C^R \subset \mathbb{R}^{F \times 3}$6

If $C^R \subset \mathbb{R}^{F \times 3}$7 does not provide a local rotation, the rotation of its nearest neighbor in $C^R \subset \mathbb{R}^{F \times 3}$8 is used. The role of this term is to guard against calibration errors by enforcing consistency in local frames rather than a globally shared frame.

The fourth component is few-shot anchor alignment. Given $C^R \subset \mathbb{R}^{F \times 3}$9 paired anchors $J^R \subset \mathbb{R}^{D}$0,

$J^R \subset \mathbb{R}^{D}$1

The total mapping loss is

$J^R \subset \mathbb{R}^{D}$2

with the paper stating that the weights are not tuned and are summed as-is in implementation.

Few-shot human guidance is operationalized through two anchor types: lateral rotation and bending. Initial per-finger human anchors use $J^R \subset \mathbb{R}^{D}$3 samples per anchor type. Linear interpolation increases the anchor counts to $J^R \subset \mathbb{R}^{D}$4 for lateral rotation and $J^R \subset \mathbb{R}^{D}$5 for bending. Robot anchors are generated in simulation for the same gestures to form paired human-robot anchors. The paper states that these anchors remove global ambiguity, set desired scales, stabilize training, and personalize the mapping to operator habits.

The reported ablation on the Wuji Hand quantifies the contribution of these losses using LMC $J^R \subset \mathbb{R}^{D}$6: $J^R \subset \mathbb{R}^{D}$7 only gives $J^R \subset \mathbb{R}^{D}$8; adding $J^R \subset \mathbb{R}^{D}$9 yields $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$0; adding $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$1 yields $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$2; adding $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$3 yields $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$4 (Wang et al., 9 Jul 2026). The interpretation given in the paper is that pairwise distance preserves geometry, local motion preserves direction, and few-shot anchors resolve global ambiguity and yield the most intuitive mapping.

4. Contact-aware pinch refinement and runtime inference

AnyDexRT adds a dedicated contact classifier for pinch because pinch with tiny objects is sensitive to small geometric errors and glove measurements may fail to capture actual pinch due to sensor noise or interference (Wang et al., 9 Jul 2026). The classifier is per finger, comparing the thumb with each other finger.

For thumb index $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$5 and finger $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$6, the classifier $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$7 takes as input the concatenated 3D thumb and finger positions

$Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$8

and outputs a probability $Q = [q_{\min}, q_{\max}] \subset \mathbb{R}^{D}$9 of intended thumb-finger contact. The network is an MLP with topology $f_m: C^H \rightarrow C^R,$0 followed by Sigmoid. The training objective is

$f_m: C^H \rightarrow C^R,$1

At inference, the decision threshold is $f_m: C^H \rightarrow C^R,$2. When $f_m: C^H \rightarrow C^R,$3, the mapped thumb-finger pair is refined by snapping to the nearest pre-generated robot pinch template. Let $f_m: C^H \rightarrow C^R,$4 be the set of robot thumb-finger-$f_m: C^H \rightarrow C^R,$5 pinch templates. Given the raw mapped pair $f_m: C^H \rightarrow C^R,$6, the selected template index is

$f_m: C^H \rightarrow C^R,$7

The target fingertip pair is then replaced with $f_m: C^H \rightarrow C^R,$8 before conversion to joints.

The runtime pipeline is therefore feed-forward mapping by the fingertip mapper, optional contact-aware snapping, and nearest-neighbor conversion to valid robot joint states. The paper reports runtime of approximately $f_m: C^H \rightarrow C^R,$9 Hz for AnyDexRT, compared with $f_s: C^R \rightarrow J^R.$0 Hz for the GeoRT neural baseline and approximately $f_s: C^R \rightarrow J^R.$1 Hz for the optimization-based baseline.

The architectural choices are correspondingly lightweight. The fingertip mapper comprises $f_s: C^R \rightarrow J^R.$2 per-finger MLPs, each with topology $f_s: C^R \rightarrow J^R.$3. The contact classifier comprises $f_s: C^R \rightarrow J^R.$4 per-contact MLPs, each with topology $f_s: C^R \rightarrow J^R.$5 plus Sigmoid. Both $f_s: C^R \rightarrow J^R.$6 and $f_s: C^R \rightarrow J^R.$7 are trained independently for $f_s: C^R \rightarrow J^R.$8 epochs with learning rate $f_s: C^R \rightarrow J^R.$9 and batch size $g_\theta = f_s \circ f_m,$0. For $g_\theta = f_s \circ f_m,$1, random perturbations $g_\theta = f_s \circ f_m,$2 are added to compute $g_\theta = f_s \circ f_m,$3, and anchor minibatches of size $g_\theta = f_s \circ f_m,$4 are interleaved for $g_\theta = f_s \circ f_m,$5.

A practical misconception is that AnyDexRT solves contact-rich dexterous manipulation in a general sense. The paper’s contact-aware refinement is specifically centered on pinch. It improves pinch reliability without requiring precise glove geometry, but broader contact-rich behaviors would require richer contact models.

5. Empirical evaluation across hands and teleoperation tasks

The simulation evaluation covers Inspire Hand, Ability Hand, XHand, Wuji Hand, Allegro Hand, Leap Hand, and Shadow Hand, spanning $g_\theta = f_s \circ f_m,$6–$g_\theta = f_s \circ f_m,$7 DoFs with diverse kinematics (Wang et al., 9 Jul 2026). Evaluation metrics are motion consistency measures: GMC and LMC. GMC compares displacement directions in a shared global coordinate frame; LMC compares directions in local frames. The paper follows GeoRT for GMC and introduces LMC to reduce calibration dependence. The deep dive provides formalizations consistent with the method’s directional consistency, while noting that the formulas were not printed verbatim in the paper.

Average simulation performance over the seven hands is reported as follows.

Method	GMC	LMC
AnyDexRT	79.9	90.2
GeoRT	78.3	59.8
Optimization	62.0	52.2

These values are reported as $g_\theta = f_s \circ f_m,$8. Per-hand results show that AnyDexRT achieves particularly strong LMC across all tested hands: Inspire Hand $g_\theta = f_s \circ f_m,$9, Ability Hand $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$0, XHand $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$1, Wuji Hand $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$2, Allegro Hand $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$3, Leap Hand $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$4, and Shadow Hand $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$5. The paper interprets this as dramatic improvement in local motion consistency, i.e., operator intent preservation, while maintaining strong global consistency and showing small standard deviations across seeds.

The study also reports manual tuning effort and throughput.

Method	Hyperparameters	Throughput
Optimization baseline	$$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$6	93.4 Hz
GeoRT	4	281.7 Hz
AnyDexRT	3	293.0 Hz

The reported interpretation is lower tuning burden and higher real-time throughput for AnyDexRT, supporting calibration efficiency.

Calibration sensitivity is evaluated on the Wuji Hand by applying input rotations of $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$7, $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$8, $$g_\theta: \mathcal{A} \subset \mathbb{R}^{F \times 3} \rightarrow Q \subset \mathbb{R}^{D}.$$9, and $i$0 around the $i$1-axis. The paper states that these perturbations significantly degrade the baselines’ LMC, whereas AnyDexRT maintains consistent LMC across perturbations, confirming calibration-free capability. Training stability is also highlighted: over five seeds, AnyDexRT exhibits orders-of-magnitude lower standard deviation in retargeted positions than GeoRT across all tested hands, although exact numbers are not provided.

Real-world teleoperation uses a Manus glove for fingertip poses, an HTC Vive Tracker for arm pose, and a Flexiv Rizon 4 arm with a Wuji Hand. The protocol involves eight operators with varying experience. Four tasks are reported: Sprink for spray-bottle triggering, Screw for light-bulb screwing, Shovel for steak shoveling, and Pick-10 for picking ten small balls.

Task	Optimization / GeoRT	AnyDexRT
Sprink time per episode	29.0 s / 32.1 s	10.6 s
Screw time per episode	25.3 s / 22.8 s	17.0 s
Shovel time per episode	36.4 s / 38.5 s	28.0 s
Pick-10 time per episode	150.4 s / 220.4 s	105.8 s
Pick-10 pinch success rate	39.6% / 29.2%	62.0%

The paper interprets these results as faster, more intuitive teleoperation and markedly improved pinch reliability, especially for contact-critical control. A plausible implication is that the method is well suited not only for online teleoperation but also for demonstration collection, which the abstract identifies as a key downstream use case for imitation learning.

6. Assumptions, limitations, and deployment considerations

The method is subject to two explicit assumptions. First, it assumes structural similarity to the human hand: human-like robot hands in which fingertip positions plus reference joint angles sufficiently constrain IK, described as approximate one-to-one $i$2. Second, it assumes that the human fingertip motion space is covered by the robot fingertip space after suitable geometric transformation, with coverage required only from human to robot, not vice versa (Wang et al., 9 Jul 2026).

The principal limitation is that AnyDexRT still requires few-shot anchors. The paper characterizes anchor collection as lightweight, but it remains a necessary step. It also states that future work may automate anchor selection or collection or adapt online from feedback. A second limitation is that contact-aware refinement focuses on pinch; broader contact-rich behaviors may need richer contact models. A third limitation is that real-world validation centers on teleoperation performance, while downstream learning from collected data is left as an important next step.

For practical deployment, the setup requires a motion-capture glove for fingertip positions and tracking for the arm or hand base. The robot should be a human-like dexterous hand, and a simulator is useful for sampling robot joint configurations and generating robot fingertip datasets and pinch templates. Data collection proceeds through free-motion human fingertip sampling, per-finger normalization, robot fingertip sampling by uniform joint-space sampling in simulation, and sparse anchor collection. The anchor procedure is specified in three stages: collect $i$3 human anchors per finger per anchor type; interpolate to $i$4 rotation anchors and $i$5 bending anchors; and generate matching robot anchors in simulation. For bending, the paper states $i$6 with $i$7, and notes that the paper uses $i$8.

Extending the system to a new hand requires regenerating robot fingertip samples and pinch templates in simulation, collecting minimal few-shot human anchors, retraining the per-finger $i$9 and $C^H \subset \mathbb{R}^{F \times 3}$00 networks with the same losses, and deploying without further calibration. The paper also notes that default three hyperparameters are generally reusable across hands.

In methodological terms, the main contributions claimed for AnyDexRT are calibration-free dexterous retargeting that preserves operator intent in local frames, self-supervised shape correspondence using asymmetric partial Chamfer, pairwise distance preservation and local motion direction consistency, few-shot human guidance to anchor task-relevant regions and personalize scale and offset, contact-aware pinch refinement via a dedicated classifier and template snapping, and demonstrated generality across seven diverse dexterous hands with improved consistency, stability, reduced tuning effort, and superior real-world teleoperation performance (Wang et al., 9 Jul 2026).