UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

Abstract: Dexterous manipulation remains challenging due to the cost of collecting real-robot teleoperation data, the heterogeneity of hand embodiments, and the high dimensionality of control. We present UniDex, a robot foundation suite that couples a large-scale robot-centric dataset with a unified vision-language-action (VLA) policy and a practical human-data capture setup for universal dexterous hand control. First, we construct UniDex-Dataset, a robot-centric dataset over 50K trajectories across eight dexterous hands (6--24 DoFs), derived from egocentric human video datasets. To transform human data into robot-executable trajectories, we employ a human-in-the-loop retargeting procedure to align fingertip trajectories while preserving plausible hand-object contacts, and we operate on explicit 3D pointclouds with human hands masked to narrow kinematic and visual gaps. Second, we introduce the Function-Actuator-Aligned Space (FAAS), a unified action space that maps functionally similar actuators to shared coordinates, enabling cross-hand transfer. Leveraging FAAS as the action parameterization, we train UniDex-VLA, a 3D VLA policy pretrained on UniDex-Dataset and finetuned with task demonstrations. In addition, we build UniDex-Cap, a simple portable capture setup that records synchronized RGB-D streams and human hand poses and converts them into robot-executable trajectories to enable human-robot data co-training that reduces reliance on costly robot demonstrations. On challenging tool-use tasks across two different hands, UniDex-VLA achieves 81% average task progress and outperforms prior VLA baselines by a large margin, while exhibiting strong spatial, object, and zero-shot cross-hand generalization. Together, UniDex-Dataset, UniDex-VLA, and UniDex-Cap provide a scalable foundation suite for universal dexterous manipulation.

• The paper introduces a scalable foundation suite, UniDex, that leverages egocentric human videos to build a large-scale robot-centric dataset for dexterous manipulation.
• It introduces a novel unified action space (FAAS) that groups functionally analogous joints into an 82D parameterization, enabling robust cross-platform policy transfer.
• Empirical results show UniDex-VLA achieves up to 81% task progress and strong zero-shot hand transfer, outperforming baselines on diverse real-world tasks.

UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos

Motivation and Problem Scope

Dexterous manipulation via robotic hands remains a major open challenge, constrained by significant data bottlenecks and embodiment heterogeneity. Existing robot foundation models, particularly those leveraging vision-language-action (VLA) policies, predominantly target end-effectors with low DoF, such as grippers, ignoring the diversity and complexity of dexterous hands used in real-world tool-use. The presented work addresses three critical bottlenecks: (i) insufficient large-scale, high-quality robot-centric datasets for dexterous manipulation; (ii) a lack of unified action spaces to enable policy and data sharing across diverse hardware; and (iii) substantial kinematic and visual domain gaps between abundant cheap human data and robot embodiments.

UniDex System: Dataset, Unified Action Space, and VLA Foundation Model

Dataset: Transforming Egocentric Human Videos Into Robot-Centric Data

The cornerstone of UniDex is the construction of the UniDex-Dataset, a large-scale resource comprising 9 million paired image-pointcloud-action frames, over 50,000 trajectories, and eight distinct dexterous hand platforms spanning 6–24 DoFs. This is uniquely derived from four significant egocentric human manipulation datasets, with rigorous language annotation, fine-grained segmentation, and dataset cleaning.

Figure 1: The UniDex-Dataset spans diverse hand types and daily tasks, visualized via verb-object cloud and example episodes.

A central technical contribution is a robust human-to-robot transformation pipeline. This pipeline employs a two-stage, human-in-the-loop retargeting procedure:

• First, automatic fingertip-based inverse kinematics is solved to align robot and human fingertip trajectories, with additional treatment for mimic joints and morphology variance.
• Second, an interactive GUI allows manual adjustment of a base offset to ensure plausible object contact, significantly reducing contact modeling artifacts and bias.

  Figure 2: Complete human-to-robot transformation, including pointcloud hand-masking, kinematic retargeting, and mesh attachment.

For visual alignment, scene pointclouds are generated from RGB-D streams, and the human hand is masked. The robot hand mesh is attached in the correct contact pose, substantially narrowing the visual gap for downstream learning.

Unified Action Space: FAAS

Heterogeneity in hardware (different DoFs, joint layouts) severely impedes policy transfer, multi-hand scaling, and data aggregation. UniDex introduces the Function–Actuator–Aligned Space (FAAS), an 82D action parameterization grouping actuation by function rather than hardware index.

Figure 3: FAAS maps functionally analogous joints of different hands into a shared index space, as depicted for the thumb and ring fingers across Oymotion, Allegro, Inspire, and Wuji.

FAAS is instantiated through joint grouping, explicit wrist pose encoding, and reserved channels for underlying functional alignment and future hands, ensuring cross-platform policy operation and transfer.

UniDex-VLA: Vision-Language-Action Foundation Model

UniDex-VLA is a 3D egocentric policy that fuses pointcloud (via Uni3D ViT-based encoder), language (natural-language commands), and proprioception. Crucially, observation and action both operate in the FAAS space, ensuring geometric-perceptual and control abstraction alignment.

Figure 4: The UniDex-VLA model pipeline: colored pointcloud, instruction, and proprioception are encoded and fused to predict H-step action chunks in FAAS.

The policy is pretrained on UniDex-Dataset via a conditional flow-matching objective and finetuned with a limited set of task-specific demonstrations.

Benchmarking and Empirical Evaluation

Experiments were conducted on a 7-DoF Franka manipulator coupled with Inspire, Wuji, and Oymotion hands, across five real-world, long-horizon tool-use tasks requiring nuanced within-hand and between-finger coordination (e.g., grasping a kettle and pouring, using a sweeper, pressing a spray trigger, using scissors, and operating a mouse).

Figure 5: Real-robot benchmark tasks demand precise hand posture adaptation for tools with distinct affordances.

Performance was compared to Diffusion Policy (DP), 3D Diffusion Policy (DP3), and $\pi_0$ (a strong VLA baseline pretrained on gripper-based data). Finetuning was restricted to 50 demonstrations per task.

Figure 6: UniDex-VLA achieves 81.0% in average task progress across five tasks, with significantly higher final success rates than baselines.

Key results:

• UniDex-VLA achieves 81.0% average task progress and 76.0% final success rate, outperforming $\pi_0$ by 113% in task progress on the hardest task (Use Scissors to Cut Bags).
• Ablation with no pretraining (UniDex-VLA (No Pretrain)) degrades success to 32.5%, emphasizing dataset and model scaling benefits.

Generalization Analysis

Spatial Generalization

By editing scene pointclouds to place objects at out-of-distribution positions, UniDex-VLA demonstrates robust workspace coverage and performance persistence.

Figure 7: Task success persists under extensive OOD placement (spatial generalization) when policies are trained with DemoGen data augmentation.

Object Generalization

Replacing evaluation objects with unseen variants (differing in color/geometry/affordance) verifies strong transfer capacity.

Figure 8: UniDex-VLA retains proficiency with novel, visually and geometrically shifted objects.

Zero-Shot Hand Transfer

Policies trained on Inspire Hand transferred zero-shot to Wuji and Oymotion, demonstrating FAAS-enabled skill transfer. Baseline policies failed entirely.

Figure 9: Zero-shot cross-hand deployment achieves 60% (Oymotion) and 40% (Wuji) task progress, with baselines near zero.

Human-Robot Data Co-Training with UniDex-Cap

UniDex-Cap is a synchronized, portable, calibration-friendly capture system integrating Apple Vision Pro and Intel RealSense. It enables rapid, in-the-wild human demonstration collection, which is then transformed (kinematically and visually) to robot-executable trajectories.

Figure 10: UniDex-Cap system and example data transformations from human pose to robot-executable plan.

A quantitative study on human–robot data exchange rate reveals that two human demonstrations (collected $\sim$5.2x faster) effectively substitute one robot demonstration in finetuning, though some real-robot data remains necessary for performance.

Figure 11: Co-training curves highlight a $\approx$2:1 exchange rate between human and robot demos for reaching high-performance regimes.

Implications and Future Directions

UniDex introduces the first foundation suite for dexterous hands leveraging large-scale robot-centric datasets derived from egocentric human data. Its unified action space and practical transformation pipeline effectively bridge visual and kinematic domain gaps, enabling VLA models to exhibit robust skill, object, and hand generalization, as well as cost-efficient scaling via human-robot co-training.

The strong empirical results indicate that coupling functionally-aligned action parameterization with 3D perception and extensive pretraining is critical for universal dexterous manipulation. The FAAS approach provides a tractable scheme for aggregating and sharing dexterous data and policies, suggesting extensibility to new hardware, unseen tasks, and further scaling via unlabeled or weakly labeled data.

Integrating action-free or weak-supervision egocentric video streams, expanding to bimanual settings, and leveraging growing open-source 3D activity datasets represent promising future directions for increasing the generality and autonomy of dexterous robot learning.

Conclusion

UniDex establishes a scalable and practical route toward universal learning and control for dexterous robotic hands. Its blend of dataset scale, embodiment-unifying action space, robust VLA policy design, and data-efficient co-training offers a powerful foundation for subsequent research in general-purpose robotic manipulation, policy transfer, and real-world generalization (2603.22264).