---
title: 'Wh0: Post-Training for Dexterous Manipulation'
url: https://www.emergentmind.com/topics/wh0
type: topic
---

# Wh0: Post-Training for Dexterous Manipulation

Searching arXiv for the Wh0 paper and closely related works mentioned in the provided data.
Wh0 is a post-training framework for dexterous robot manipulation that uses generative video world models as scalable and controllable sources of egocentric human-hand manipulation data. It is designed to address a central bottleneck in dexterous manipulation: scaling data while keeping it aligned with the robot’s real deployment conditions. Conditioned on language, objects, and scenes, Wh0 uses a generative world model to produce **WM-H**, a **50k-episode** dataset of egocentric human-object interaction videos, then converts those videos into robot-trainable supervision through hand motion reconstruction and visual editing. Co-trained with a limited amount of real robot data, WM-H adapts pretrained dexterous VLA models to dexterous manipulation deployment, improving zero-shot success on unseen real-world tasks from **8.3%** to **38.9%** across **18 dexterous manipulation tasks** [2606.22136].

## 1. Problem setting and motivation

Wh0 is motivated by the observation that dexterous manipulation policies must generalize across many objects, many scenes and layouts, many task instructions, and a difficult embodiment gap between human hands and robot hands. Existing data sources each incur a distinct limitation. Teleoperation data is highly aligned with deployment because it comes from the target robot in the target workspace, but it is expensive, slow to collect, and platform-specific. Simulation is scalable, but dexterous manipulation is especially sensitive to contact, geometry, and visual realism, so sim-to-real transfer remains difficult. Real egocentric human video scales well and contains rich human manipulation priors, but it is misaligned with robot deployment through both a **scene gap** and an **embodiment gap** [2606.22136].

Wh0 proposes generative video world models as a new point in this trade-off. Rather than using them primarily as robot dynamics simulators, the framework uses them as controllable engines for generating egocentric human-hand manipulation videos. These videos are conditioned on language, object identity, and scene context, making the data source scalable with compute rather than human collection labor. The central operational idea is to generate deployment-aligned egocentric manipulation video with human hands, reconstruct explicit 3D hand motion from those videos, adapt the visuals toward robot embodiment, and co-train this synthetic-but-aligned supervision with a small amount of real teleoperated robot data to improve a pretrained dexterous VLA [2606.22136].

The paper’s argument is not that generated video replaces teleoperation or explicit robot data. Instead, it positions generated human-hand data as a middle ground whose usefulness depends on two alignment mechanisms: **scene alignment**, which makes generated videos resemble the actual robot workspace, and **embodiment alignment**, which exposes the policy to the same trajectory rendered with robot-hand appearance. This suggests that the framework is best understood as a mechanism for bridging between abundant human-centric visual priors and robot execution constraints rather than as a standalone synthetic-data pipeline [2606.22136].

## 2. Generative data pipeline and WM-H dataset

The Wh0 generation pipeline is conditioned on **language instruction**, **object specification**, and **scene layout / workspace image**. The generated data is intended to vary across object nouns, object attributes, scene arrangements, and manipulation types such as grasping, picking, and placing. The pipeline has three major stages: **instruction generation**, **scene- and embodiment-aligned video synthesis**, and **hand motion extraction and conversion to robot-trainable supervision** [2606.22136].

A notable design choice is the explicit balancing of lexical diversity and frequency coverage during instruction generation. The authors use a **dual-agent system** in which one LLM-based agent continually expands a pool of object nouns and attribute adjectives relevant to manipulation, while another agent preferentially samples underrepresented words and assembles instructions using templates such as “pick the {adj} {noun}.” A database tracks word counts, rejects duplicates, and expands vocabulary only when existing words have met a minimum usage threshold. The reported coverage is **noun h-index: 201** and **adjective h-index: 117**, meaning at least 201 nouns each appear in at least 201 samples, with the analogous interpretation for adjectives [2606.22136].

The output of this synthesis pipeline is **WM-H**, a dataset of **50k egocentric manipulation episodes**, each with a language instruction, a generated egocentric human-hand video, reconstructed 3D hand motion annotations, and optionally embodiment-aligned edited frames with robot-hand appearance. WM-H is therefore neither pure teleoperation data nor passive human video. It is a synthetic egocentric human-hand dataset engineered to become trainable for robot policy adaptation. Compared with standard robot datasets, it contains generated egocentric human-hand manipulation videos, language prompts, reconstructed hand trajectories in MANO space, deployment-aligned scene visuals, and appearance-edited robot-hand variants [2606.22136].

After processing, the effective supervision includes current visual observations, the language instruction, a FoV token, current hand state, future hand-action chunks, reconstructed wrist translations, reconstructed wrist rotations, reconstructed MANO joint angles, and edited versions of frames with robot-hand appearance. In this respect, WM-H is close to action-labeled manipulation data rather than weakly labeled video [2606.22136].

## 3. Scene alignment and embodiment alignment

Scene alignment is implemented by first capturing **real background images** from the target robot workspace using the same deployment camera, the same viewpoint, and the same resolution as policy input. A human hand is intentionally placed in the scene during capture as a **scale anchor**, preserving realistic relationships among hand size, object size, and camera-to-workspace distance. For each instruction, a workspace image is chosen and objects are inserted using **Qwen-Image-Edit**, with object insertion spatially localized using a rectangular guide region sampled in the reachable workspace [2606.22136].

From the edited initial frame, Wh0 uses **Wan-I2V-A14B** to generate an egocentric human-hand manipulation video conditioned on the instruction. To improve temporal correctness, **Qwen3-VL** is first used to infer an expected description of hand-object state changes from the image and instruction, and that generated dynamics description is appended to the final video prompt. For efficiency, the framework uses **LightX2V LoRA adapters**, reducing generation to **four inference steps**. The reported generation cost is approximately **5.44 GPU-hours per 1k videos**, a figure that is important because the paper’s central claim is scalability through compute [2606.22136].

Embodiment alignment addresses the fact that generated videos still contain human hands, while the deployed system uses an **Inspire dexterous hand**. Wh0 performs visual editing on sparsely sampled frames using **Qwen-Image-Edit** to replace the human hand with a realistic robot hand while preserving hand pose, hand position, hand scale, object motion, and scene composition. The stated purpose is not to synthesize true robot kinematics. Rather, the same manipulation trajectory is shown under a different appearance so that the model learns that the **action semantics** should remain stable across executor identity [2606.22136].

These two alignment mechanisms define the method. Scene alignment preserves deployment-relevant viewpoint and workspace geometry; embodiment alignment reduces the visual discrepancy between human-hand observations and robot-hand deployment. The ablations show that both materially affect transfer. Without scene alignment, success drops to **20.0%**; without embodiment alignment, success drops from **38.9%** to **34.7%** [2606.22136].

## 4. Hand reconstruction, action representation, and shared MANO space

A key step in Wh0 is converting generated videos into explicit action labels. Following VITRA, the framework reconstructs 3D hand poses from human-hand videos using **HaWoR**, described as “world-space hand motion reconstruction from egocentric videos.” HaWoR detects hands in each frame and regresses **MANO parameters** plus wrist pose. The retained representation is wrist pose in camera space and articulated hand pose in MANO parameter space. If needed, camera tracking from **MegaSAM** associates framewise predictions with the camera trajectory [2606.22136].

The policy action space is defined in the camera coordinate frame of the current observation \(o_t\). The action is

```latex
a_t = [\Delta t^l, \Delta r^l, \theta_h^l, \Delta t^r, \Delta r^r, \theta_h^r] \in \mathbb{R}^{102},
```

where \(\Delta t, \Delta r \in \mathbb{R}^3\) are relative wrist translation and wrist rotation between consecutive frames, \(\theta_h \in \mathbb{R}^{15 \times 3}\) are local joint rotations of the **15-DoF MANO hand model**, and superscripts \(l, r\) indicate left and right hands. In practice, the work focuses on **right-hand manipulation**. This gives \(51\) dimensions per hand and \(102\) dimensions total for two hands [2606.22136].

The paper assumes that robot joint configurations can be **retargeted into MANO space**, allowing both human and robot trajectories to be expressed in a common hand-action representation. Robot demonstrations are converted by transforming robot states and actions from the robot base frame to the camera frame, correcting wrist rotations to align with MANO conventions, retargeting robot joints into the MANO hand space, and normalizing robot actions and states using **human-data normalization statistics precomputed by VITRA**. The authors explicitly avoid robot-specific normalization from only **400 demos**, arguing that reuse of large-scale human normalization keeps robot actions aligned with the pretrained model’s representation [2606.22136].

This shared MANO-space formulation is central to the framework. It provides the bridge through which generated human video can become executable robot-learning signal. A plausible implication is that Wh0 depends less on direct visual imitation of robot embodiments than on representational compatibility between human and robot trajectories.

## 5. Policy architecture and post-training procedure

Wh0 uses a **VITRA-style vision-language-action policy** built on a **PaliGemma2-3B** vision-language backbone. The backbone encodes the current image observation, the language instruction, a **2D FoV token** projected by an MLP, and a learnable **cognition token**. The final hidden state of the cognition token serves as the conditioning representation for the action decoder. The decoder is a diffusion-based **DiT-B** network that predicts **16 future action steps per chunk** in the MANO-based action space [2606.22136].

The current hand state is represented in the camera frame by wrist translation, wrist Euler angles, and **15-DoF MANO joint rotations** for each hand. Internally, the model uses the unified VITRA state/action space with **state dimension 212** and **action dimension 102**, although only the human-hand-related dimensions are active here. The diffusion decoder is trained using the standard noise-prediction MSE objective,

```latex
\mathcal{L}_{\mathrm{MSE} = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, 1),\, i} \left[ \left\| \hat{\epsilon}_i - \epsilon \right\|_2^2 \right],
```

where \(\epsilon\) is Gaussian noise sampled from \(\mathcal{N}(0, 1)\), \(i\) is the sampled diffusion timestep, and \(\hat{\epsilon}_i\) is the model’s predicted noise at step \(i\) [2606.22136].

The method is explicitly a **post-training** approach rather than training from scratch. Initialization comes from **VITRA**, pretrained on **Ego4D**, **Epic-Kitchens**, **EgoExo4D**, and **Something-Something-V2**. The robot adaptation set contains **400 expert teleoperated trajectories** collected using **Apple Vision Pro** on seen tasks and backgrounds. Wh0 uses a dataset-scale ratio of approximately **125:1** between WM-H and teleoperation data, corresponding to **50k WM-H samples** and **400 teleop demonstrations**. Batch sampling deliberately oversamples robot data, with each batch consisting of **28% teleop**, **68% WM-H**, and **4% WM-H EA**, written as

```latex
R:W\text{-EA}:W = 0.28:0.04:0.68
```

[2606.22136].

During post-training, the **vision encoder** is frozen, while the remaining VLM backbone and diffusion action decoder are updated. The stated optimization details are: **4 NVIDIA H200 GPUs**, per-GPU batch size **64**, total batch size **256**, learning rate **\(1\times10^{-5}\)**, weight decay **0.1**, Adam betas **\((0.9, 0.95)\)**, gradient clipping **1.0**, max steps **40k**, diffusion steps during training **100**, a squared-cosine noise schedule, diffusion training repeated **8 times per batch** with independent noise and timestep sampling, and **no image augmentation**. At inference, a single **RTX 4090** is used with **DDIM sampling**, **10 denoising steps**, and classifier-free guidance scale **5.0** [2606.22136].

## 6. Experimental evaluation, baselines, and results

All experiments are conducted on a **Unitree G1 humanoid** equipped with **Inspire dexterous hands** and a **head-mounted egocentric camera**. The training data consists of **400 teleoperated expert trajectories**, mainly pick-and-place over seen objects and backgrounds. The evaluation suite contains **18 real-world dexterous manipulation tasks** across four scenes, including tasks such as “grasp the tripod,” “put the coke can into the black box,” “grasp the remote controller,” and “put the tissue into the yellow basket.” Evaluation is **zero-shot** on unseen tasks, environments, and instructions, with **20 trials per task**, randomized object poses and scenes, success rate as the primary metric, and no manual action correction during execution. For multi-stage tasks, the evaluator provides the next stage instruction when the previous phase is completed but does not manually intervene in behavior [2606.22136].

There is also a deployment-time **grasping prior**: before contact, finger joints are constrained to close monotonically until stable grasp. The same rule is applied to all methods. This matters because the reported performance differences are not attributable to a method-specific execution heuristic [2606.22136].

The main baselines are \(\pi_{0.5}\), a robot-data-pretrained VLA adapted using teleop data only; **VITRA**, a human-video-pretrained dexterous VLA adapted using the same teleop data only; **VITRA Real Version**, which replaces WM-H with real egocentric human-hand videos from **HOI4D**; and **Wh0**, which uses teleop plus WM-H during co-fine-tuning. The appendix notes that HOI4D is processed into **5,511 annotated episodes** by grouping every 100 frames into one episode and assigning one language annotation [2606.22136].

The main quantitative result is the improvement in zero-shot real-world dexterous manipulation success:

| Method | Success rate |
|---|---:|
| \(\pi_{0.5}\) | **7.78% ± 15.6** |
| VITRA | **8.3% ± 8.6** |
| VITRA Real Version | **21.4% ± 23.4** |
| Wh0 | **38.9% ± 19.8** |

Wh0 therefore improves zero-shot success on unseen tasks from **8.3%** for the teleop-only VITRA adaptation to **38.9%**, a **4.7× improvement** [2606.22136]. The main empirical pattern is that teleop-only adaptation is weak, adding extra human video during post-training helps, and generated, deployment-aligned WM-H helps substantially more than real but misaligned egocentric human video. This suggests that alignment matters more than merely using “real” data [2606.22136].

The paper also introduces **Hand-Object Distance** as a proxy for language-conditioned object grounding. Lower HO is better. The ablations probe scene alignment, embodiment alignment, and dataset scale. Reported values include: **Teleop only** with HO(human) **16.2**, HO(robot) **16.2**, success **8.3**; **w/o scene alignment** with HO(human) **14.9**, HO(robot) **14.3**, success **20.0**; **w/o embodiment alignment** with HO(human) **10.2**, HO(robot) **13.8**, success **34.7**; **WM-H 5k** with HO(human) **11.9**, HO(robot) **10.5**, success **27.8**; **WM-H 25k** with HO(human) **11.4**, HO(robot) **9.9**, success **32.5**; and **Wh0 (50k)** with HO(human) **10.6**, HO(robot) **9.6**, success **38.9** [2606.22136].

These ablations indicate that scene alignment, embodiment alignment, and scale each contribute materially. Performance improves steadily from **5k** to **25k** to **50k** generated episodes. The authors also report that embodiment alignment stabilizes action features under hand-appearance changes, measured by action-feature cosine similarity [2606.22136].

## 7. Interpretation, limitations, and relation to adjacent directions

One of the paper’s most explicit interpretive claims concerns “unlocking pretraining priors.” The analysis compares models with and without strong human-video pretraining. The reported table gives **PaliGemma pretrain, Teleop** with Hand-Object Distance **14.3** and task success **0.8**; **PaliGemma pretrain, Teleop + WM-H** with **12.7** and **0.6**; **Human pretrain only** with **13.1** and **0.0**; **Human pretrain, Teleop** with **16.2** and **8.3**; and **Wh0** with **10.6** and **38.9** [2606.22136]. The paper’s conclusion is that WM-H does not teach dexterous manipulation from scratch; rather, it activates capabilities already latent in the pretrained VLA. Without strong human-video pretraining, adding WM-H does almost nothing; human pretraining alone gives some grounding but no deployable success; teleop alone partially adapts but overfits and undergeneralizes; and only the combination of human-video pretraining, teleop data, and WM-H yields large gains [2606.22136].

The framework includes several quality-control mechanisms: duplicate instruction filtering, controlled object insertion and non-overlap sampling, manual filtering of incorrect episodes for Hand-Object Distance evaluation, user studies on realism and training suitability, and appendix discussion of failure cases. In a user study with **72 AI practitioners**, **37.7%** of synthetic videos were judged as real. Average quality scores were **3.97 ± 1.22** for object correctness, **4.18 ± 1.09** for instruction alignment, **3.95 ± 1.19** for hand-object interaction, **3.78 ± 1.30** for physical plausibility, and **3.57 ± 1.31** for training suitability. After robot-hand editing, scores were **4.30 ± 0.85** for pose consistency and **4.25 ± 0.84** for contact preservation [2606.22136]. These values support the narrower claim that the generated data is noisy but usable.

The paper is also explicit about failure modes. Reported issues include **image editing errors** such as wrong object placement and cropping artifacts; **physically implausible interactions** such as hand passing through objects and unrealistic grasps; **temporal inconsistencies** such as mismatch between pre- and post-interaction visuals; **invalid instructions** that are not executable; and **robot-hand editing issues** in which pose is not perfectly preserved or scene objects are unintentionally modified [2606.22136]. At a broader level, the method is limited by video generation quality, hand reconstruction accuracy, human-robot morphology mismatch, dependence on strong human-video pretraining, and task scope. The experiments mostly focus on **single-arm pick-and-place style manipulation**, and the paper explicitly notes that extending to bimanual tasks, more complex tool use, and longer-horizon manipulation remains future work [2606.22136].

In relation to broader research directions, Wh0 resembles sim-to-real transfer in that it creates synthetic training data for real deployment, but it does not rely on explicit physics or precise robot dynamics. It uses generative video instead of a simulator. It also resembles imitation learning from demonstrations, but it reconstructs human hand motion and aligns robot action space to human MANO space rather than directly cloning robot actions. Within egocentric video learning and robot foundation models, Wh0 suggests a path in which large pretrained VLAs are adapted using **compute-generated post-training corpora**, not only more human labor or more teleoperation [2606.22136].

The most novel aspect of the framework is the reframing of world models as controllable data engines for deployment-targeted manipulation supervision. A plausible implication is that its significance lies less in any single architectural component than in the coordinated combination of controllable generation, scene alignment, embodiment alignment, shared action representation, and limited real-robot co-training.

Source: https://www.emergentmind.com/topics/wh0