Being-H0: Dexterous VLA Model

• Being-H0 is a large-scale, dexterous Vision-Language-Action model that learns fine-grained 3D hand motion trajectories from web-scale human videos for complex robotic manipulation tasks.
• It employs a novel part-level motion tokenization and physical instruction tuning, aligning visual, language, and motion data in a unified 3D space with millimeter-level precision.
• The model leverages comprehensive data curation and robust cross-domain transfer, significantly improving instruction following and dexterous control across varied robotic tasks.

Being-H0 refers to a large-scale, dexterous Vision-Language-Action (VLA) model trained predominantly on human video data, with the explicit objective of capturing the fine-grained, physically accurate hand motion required for complex robotic manipulation tasks (Luo et al., 21 Jul 2025). Unlike prior VLAs that rely primarily on synthetic environments or limited-scale teleoperated demonstrations, Being-H0 leverages the extensive dexterity present in human hands—combined with the broad diversity and volume of web-scale human video—to produce a system that excels at instruction following, hand motion generation, and generalizes robustly to new real-world tasks via physical instruction tuning.

1. Vision-Language-Action Training with Human Videos

Being-H0 is pre-trained to associate high-resolution visual observations and natural language instructions with 3D hand motion trajectories. Each training instance consists of a video frame or sequence, a corresponding language instruction, and the associated hand pose, represented as MANO parameters (joint angles, global wrist pose, and optionally shape). The continuous hand trajectories are discretized into structured motion tokens. These tokens are embedded along with visual and textual tokens into a shared transformer backbone, which uses cross-attention layers to fuse information across modalities.

Physical instruction tuning is the core paradigm: instead of conventional vision-language pre-training, Being-H0 explicitly learns from human manipulation videos to map instructions and observations to physical action trajectories. The training pipeline is staged as:

• Large-scale VLA pretraining from annotated human videos.
• Physical space alignment, unifying data captured under heterogeneous viewpoints and camera models in a consistent 3D metric space, critical for geometric reasoning.
• Post-training adaptation, projecting the learned representation into specific robot morphologies and environments for downstream tasks. This sequential approach enables transfer of nuanced human dexterity—learned at scale—from pretraining into real robotic systems.

2. Data Curation Strategies and Source Integration

A foundational contribution is the creation of UniHand, a comprehensive unified dataset. The curation pipeline incorporates:

• Motion capture data and VR-tracked sequences (high-fidelity hand pose with known camera and metric calibration).
• RGB-only web videos, processed by state-of-the-art 3D hand pose estimation models to recover per-frame MANO representations.
• View-invariant augmentation, ensuring balanced coverage across hand pose, camera viewpoint, and task type by reweighting or synthetically expanding underrepresented configurations.

All trajectories are normalized into a canonical 3D space using a weak-perspective projection alignment: for each source and target camera, scale factors $s_x = f'_x/f_x$ and translation offsets $\Delta x = c'_x - s_x c_x$ are computed; each pixel $(u,v)$ is mapped via $u' = s_x u + \Delta x$, $v' = s_y v + \Delta y$. This alignment allows accurate geometry and hand-object interaction statistics to be learned despite varying camera models and scene depth cues.

The effective pretraining set comprises over 2.5 million instances from 440,000 task trajectories, extending to more than 130 million frames (over 1,100 hours), covering a broad manipulation task spectrum.

3. Part-Level Motion Tokenization and Model Architecture

Precisely capturing dexterous human hand motion for learning is achieved using part-level motion tokenization, a technical innovation of Being-H0:

• Motions are separated into global wrist movements and local finger articulations, each tokenized independently.
• Grouped Residual Quantization (GRQ) is used: for each motion chunk $z^{(g)}$, residuals $r_0 = z^{(g)}_i$, quantizer codewords $$q_\ell = \arg\min_{c\in \mathcal{C}^{(g)}} ||r_{\ell-1} - c||_2$$, residual updates $r_\ell = r_{\ell-1} - q_\ell$, and reconstructed tokens $\hat{z}^{(g)}_i = \sum_{\ell=1}^L q_\ell$.
• This approach yields millimeter-level accuracy in reconstructing original hand motion, essential for reproducing subtle manipulations.

The transformer-based VLA model is trained using an autoregressive next-token loss:

$$\theta^* = \arg\min_\theta \sum_{i=1}^N L(\theta) = -\sum_{j=1}^L \log P_\theta(y_j \mid X_Q, \hat{y}_{1:j-1})$$

where $X_Q$ represents the visual-language context, and $\hat{y}_{1:j-1}$ are the previously predicted tokens. The architecture scales up to 14B parameters, with explicit gains found as model and data size increase.

4. Physical Space Alignment and 3D Reasoning

Physical instruction tuning necessitates explicit reasoning about 3D spatial relationships, a nontrivial challenge given the heterogeneity of camera intrinsics and viewpoints across human video sources. Being-H0 introduces a weak-perspective projection alignment technique:

• For each source/target pair, camera intrinsic matrices are used to compute scaling and translation required to remap pixel spaces into a common canonical frame.
• This transform is given by

$s_x = f'_x / f_x, \quad s_y = f'_y / f_y,$

$$\Delta x = c'_x - s_x c_x, \quad \Delta y = c'_y - s_y c_y,$$

$u' = s_x u + \Delta x, \quad v' = s_y v + \Delta y$

where $f_x, f_y$ and $c_x, c_y$ are the original camera's focal lengths and principal points, and $f'_x, f'_y, c'_x, c'_y$ are those of the target canonical camera.

• Alignment is applied jointly to all vision and trajectory data, enabling the model to learn geometric invariances alongside semantic grounding.

Additionally, data spanning underrepresented viewpoints are oversampled to ensure uniform coverage, directly improving spatial generalization.

5. Empirical Performance, Scalability, and Applications

Performance is evaluated across multiple axes:

• Vision-language-mapped hand motion generation, with metrics such as Mean Per Joint Position Error (MPJPE), Mean Wrist Translation Error (MWTE), Procrustes-Aligned MPJPE (PA-MPJPE), and semantic retrieval accuracy.
• Part-level motion tokenization enables valid motion reconstruction at millimeter precision, with block-formatted and soft-formatted decoding balancing physical plausibility and flexibility.
• The larger model variants consistently achieve lower error and higher semantic consistency, with performance improving as pretraining data increases, especially on out-of-distribution (“tail”) splits.

For physical instruction tuning, Being-H0 is adapted to actual robot systems by mapping proprioceptive and state representations to the pretrained embedding space using a lightweight MLP. Real-world deployments cover tasks requiring fine finger control: pick-and-place, articulated object closure, pouring, and unfolding deformables. These adaptations exhibit marked improvements in dexterity and precision over non-physically instructional-tuned baselines, and require fewer teleoperated demonstrations for equivalent performance.

6. Scientific and Practical Implications

Being-H0 advances the paradigm of learning dexterous manipulation from web-scale human videos. By using human hand motion as a “foundation manipulator” and aligning it into a unified 3D space, the model achieves cross-domain transfer—bridging the sim-to-real gap and endowing robots with nuanced dexterous capabilities without reliance on datasets scarce in scope and diversity. The physical instruction tuning pipeline offers a blueprint for transferring knowledge from rich human experience to robotic agents, with empirical evidence for gains attributable to both scale and curation strategy.

In terms of methodology, the part-level tokenization and explicit space alignment present new standards for encoding action in VLA systems, demonstrating state-of-the-art millimeter-level action modeling. This model architecture, paired with the associated data pipeline, is scalable and broadly applicable for future generalist robots performing high-DOF manipulation tasks that require nuanced spatial, visual, and language reasoning.