UniDex-VLA: Universal Dexterous Control

• UniDex-VLA is a unified vision-language-action policy for dexterous robotic hand control that uses egocentric human demonstrations and a function-aligned action space.
• It fuses 3D visual data, task language, and proprioceptive states through a transformer backbone to generate FAAS-parameterized joint commands.
• The policy demonstrates state-of-the-art cross-hand generalization and high performance across diverse spatial layouts, embodiments, and object domains.

UniDex-VLA is a vision-language-action policy for universal dexterous robotic hand control. Integrated within the UniDex robot foundation suite, UniDex-VLA provides a unified framework for controlling diverse multi-fingered robotic hands using egocentric human demonstrations, functionally aligned actuation spaces, and scalable 3D vision-language-action modeling. The policy achieves high performance and strong generalization across embodiments, spatial layouts, and object domains, establishing a new standard for dexterous manipulation in robotics research (Zhang et al., 23 Mar 2026).

1. UniDex Suite and the Role of UniDex-VLA

UniDex is comprised of three primary modules:

• UniDex-Dataset: A large-scale, robot-centric dataset consisting of 9 million image–pointcloud–action frames and over 50,000 dexterous manipulation trajectories across eight robotic hands with 6–24 DoFs. The dataset leverages egocentric human video data, which is retargeted to robot embodiments via a human-in-the-loop procedure that ensures accurate fingertip contact and maintains realistic hand-object interaction.
• Function-Actuator-Aligned Space (FAAS): A functionally unified action parameterization mapping each hand’s actuators into a shared coordinate system, which is crucial for cross-device transfer.
• UniDex-Cap: A portable RGB-D and human-pose capture setup enabling efficient human-robot co-training and rapid acquisition of dexterous demonstrations.

UniDex-VLA is the vision-language-action policy trained on UniDex-Dataset and subsequently fine-tuned with both real and retargeted demonstrations. The policy takes as input task language and 3D visual observations and outputs FAAS-parameterized joint actions, enabling immediate deployment across a wide variety of dexterous hand platforms without architectural modifications (Zhang et al., 23 Mar 2026).

2. Input Modalities and Preprocessing

At every time step $t$, UniDex-VLA consumes:

• Visual Input: Single-view egocentric RGB-D images, transformed into colored point clouds $P_t \in \mathbb{R}^{N \times 6}$ (with xyz and RGB per point). Human hands in the recorded scene are masked out, and a retargeted robot hand mesh is inserted to ensure visual consistency during domain adaptation.
• Language Command: Task prompt $\ell$ is encoded using a pretrained text encoder, yielding a dense feature vector $e_{\mathrm{lang}}$.
• Proprioceptive State: Robot state $q_t$ mapped into FAAS coordinates.

This tri-modal representation enables robust fusion of perceptual, linguistic, and embodiment-specific signals for downstream policy learning.

3. Function–Actuator–Aligned Space (FAAS): Unified Action Parameterization

FAAS is central to UniDex-VLA's cross-embodiment generality. For each robot hand $i$ with $N_{\mathrm{act},i}$ actuators, FAAS defines a mapping: $$f_i: \mathbb{R}^{N_{\mathrm{act},i}} \longrightarrow \mathbb{R}^{82}$$ where the first 18 dimensions represent the 6D wrist pose and translation, and the remaining 64 dimensions encode finger actuation, allowing a standardized view across five fingers (5 DoF per finger, totaling 25 slots) plus reserved dimensions for future or custom DoFs. The inverse map $f_i^{-1}$ reconstructs native joint commands for each device.

Each action $a_t$ output by UniDex-VLA is in FAAS, ensuring any downstream system can project it onto the native joint space of an arbitrary hand. This functional alignment is the mechanism that enables zero-shot, cross-hand adaptation (Zhang et al., 23 Mar 2026).

4. UniDex-VLA Policy Architecture

UniDex-VLA's architecture consists of:

• 3D Vision Encoder (Uni3D): Lifts the colored pointcloud $P_t$ into a set of patch embeddings using transformer layers.
• Language Encoder: Tokenizes and embeds the language prompt into feature tokens.
• Proprioceptive Embedder: Projects the FAAS vector $q_t$ into the model's token space.
• Multimodal Backbone: Concatenates and fuses all modalities using a multi-layer transformer with multi-head self-attention: $$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$ where $Q$, $K$, and $V$ are linear projections of the fused token sequence.
• Action Decoder: Outputs an $H$-step chunk $$A_t = [a_t, ..., a_{t+H-1}] \in \mathbb{R}^{H \times 82}$$, each $a_{t+k}$ residing in FAAS and ready for direct mapping to the target device.

5. Losses, Training Protocol, and Data Co-Training

• Pretraining: UniDex-VLA is pretrained on the UniDex-Dataset using a conditional flow-matching objective.
• Post-Training (Finetuning): Performed via $L_2$ behavior cloning loss: $$\mathcal{L}_{\mathrm{BC}} = \mathbb{E}_t \left\lVert\, a_t^{\mathrm{pred}} - a_t^{\mathrm{demo}} \right\rVert_2^2$$ with optional velocity smoothness,

$$\mathcal{L}_{\mathrm{smooth}} = \lambda_{\mathrm{smooth}}\, \mathbb{E}_t \left\lVert a_t - a_{t-1} \right\rVert_2^2 \quad (\lambda_{\mathrm{smooth}} = 0.01)$$

and contact consistency regularization to keep fingertips proximate to the manipulated object.

• Human–Robot Co-Training: UniDex-Cap allows rapid capture and retargeting of human demonstrations. Empirically, two human demos are as effective as one robot teleoperation, and human collection is approximately five times faster, substantially reducing overall data acquisition cost (Zhang et al., 23 Mar 2026).

6. Generalization, Evaluation Results, and Comparison

To quantify cross-hand transfer and generalizability:

• Cross-Hand Zero-Shot Transfer: After training on Inspire Hand (6 DoF) "Make Coffee" tasks, UniDex-VLA is deployed without finetuning on Wuji (20 DoF) and Oymotion (6 DoF) hands; results are 40% and 60% average task progress, respectively, with baselines scoring near 0%.
• Spatial and Object Generalization: With test-time spatial layouts or unseen tools, UniDex-VLA surpasses 80% progress (vs. ~30% for next-best), and maintains ~75% on novel-object generalization (vs. ~35%).
• Baseline Comparison: Across all evaluated tool-use tasks, UniDex-VLA achieves 81% average task progress and substantially outperforms prior VLA baselines.

These results establish the efficacy of large-scale dataset pretraining, FAAS parameterization, and mixed-modality modeling for dexterous hand control (Zhang et al., 23 Mar 2026).

7. Runtime Inference and Deployment Mechanism

The control loop for UniDex-VLA is as follows:

1. Acquire $(I_t, D_t)$ and mask human points to reconstruct $P_t$.
2. Embed language $\ell$ and proprioceptive state $q_t$ (in FAAS).
3. Form token sequence via Uni3D, language encoder, and linear embedding.
4. Forward all tokens through the transformer backbone and output $A_t$ via an MLP.
5. Transform the leading action $a_t \in \mathbb{R}^{82}$ via $f_i^{-1}$ to yield robot-specific joint angles $q_t^{(i)}$.
6. Dispatch $q_t^{(i)}$ as commands to the hardware hand.
7. Iterate at 4–10 Hz.

Because $a_t$ is always in FAAS, the policy is agnostic to the hardware's precise embodiment, and only the $f_i^{-1}$ mapping needs adaptation.

---

UniDex-VLA operationalizes large-scale pretraining of visuomotor policies with function-aligned action spaces and flexible human-robot data integration. Its transformer-based, 3D vision-language-proprioception architecture and explicit functional calibration across manipulators yield state-of-the-art generalization, establishing UniDex-VLA and the UniDex suite as central resources for research on universal dexterous manipulation (Zhang et al., 23 Mar 2026).