Wh0: Post-Training for Dexterous Manipulation

• Wh0 is a post-training framework that generates synthetic egocentric human-hand videos conditioned on language and scene context to create scalable, deployment-aligned supervision for robot manipulation.
• It employs generative video world models and explicit hand motion reconstruction to bridge the gap between human video data and robot execution, ensuring scene and embodiment alignment.
• Co-training with limited real teleoperation data raised zero-shot success on unseen tasks from 8.3% to 38.9% across 18 dexterous manipulation tasks.

Searching arXiv for the Wh0 paper and closely related works mentioned in the provided data. Wh0 is a post-training framework for dexterous robot manipulation that uses generative video world models as scalable and controllable sources of egocentric human-hand manipulation data. It is designed to address a central bottleneck in dexterous manipulation: scaling data while keeping it aligned with the robot’s real deployment conditions. Conditioned on language, objects, and scenes, Wh0 uses a generative world model to produce WM-H, a 50k-episode dataset of egocentric human-object interaction videos, then converts those videos into robot-trainable supervision through hand motion reconstruction and visual editing. Co-trained with a limited amount of real robot data, WM-H adapts pretrained dexterous VLA models to dexterous manipulation deployment, improving zero-shot success on unseen real-world tasks from 8.3% to 38.9% across 18 dexterous manipulation tasks (Chen et al., 20 Jun 2026).

1. Problem setting and motivation

Wh0 is motivated by the observation that dexterous manipulation policies must generalize across many objects, many scenes and layouts, many task instructions, and a difficult embodiment gap between human hands and robot hands. Existing data sources each incur a distinct limitation. Teleoperation data is highly aligned with deployment because it comes from the target robot in the target workspace, but it is expensive, slow to collect, and platform-specific. Simulation is scalable, but dexterous manipulation is especially sensitive to contact, geometry, and visual realism, so sim-to-real transfer remains difficult. Real egocentric human video scales well and contains rich human manipulation priors, but it is misaligned with robot deployment through both a scene gap and an embodiment gap (Chen et al., 20 Jun 2026).

Wh0 proposes generative video world models as a new point in this trade-off. Rather than using them primarily as robot dynamics simulators, the framework uses them as controllable engines for generating egocentric human-hand manipulation videos. These videos are conditioned on language, object identity, and scene context, making the data source scalable with compute rather than human collection labor. The central operational idea is to generate deployment-aligned egocentric manipulation video with human hands, reconstruct explicit 3D hand motion from those videos, adapt the visuals toward robot embodiment, and co-train this synthetic-but-aligned supervision with a small amount of real teleoperated robot data to improve a pretrained dexterous VLA (Chen et al., 20 Jun 2026).

The paper’s argument is not that generated video replaces teleoperation or explicit robot data. Instead, it positions generated human-hand data as a middle ground whose usefulness depends on two alignment mechanisms: scene alignment, which makes generated videos resemble the actual robot workspace, and embodiment alignment, which exposes the policy to the same trajectory rendered with robot-hand appearance. This suggests that the framework is best understood as a mechanism for bridging between abundant human-centric visual priors and robot execution constraints rather than as a standalone synthetic-data pipeline (Chen et al., 20 Jun 2026).

2. Generative data pipeline and WM-H dataset

The Wh0 generation pipeline is conditioned on language instruction, object specification, and scene layout / workspace image. The generated data is intended to vary across object nouns, object attributes, scene arrangements, and manipulation types such as grasping, picking, and placing. The pipeline has three major stages: instruction generation, scene- and embodiment-aligned video synthesis, and hand motion extraction and conversion to robot-trainable supervision (Chen et al., 20 Jun 2026).

A notable design choice is the explicit balancing of lexical diversity and frequency coverage during instruction generation. The authors use a dual-agent system in which one LLM-based agent continually expands a pool of object nouns and attribute adjectives relevant to manipulation, while another agent preferentially samples underrepresented words and assembles instructions using templates such as “pick the {adj} {noun}.” A database tracks word counts, rejects duplicates, and expands vocabulary only when existing words have met a minimum usage threshold. The reported coverage is noun h-index: 201 and adjective h-index: 117, meaning at least 201 nouns each appear in at least 201 samples, with the analogous interpretation for adjectives (Chen et al., 20 Jun 2026).

The output of this synthesis pipeline is WM-H, a dataset of 50k egocentric manipulation episodes, each with a language instruction, a generated egocentric human-hand video, reconstructed 3D hand motion annotations, and optionally embodiment-aligned edited frames with robot-hand appearance. WM-H is therefore neither pure teleoperation data nor passive human video. It is a synthetic egocentric human-hand dataset engineered to become trainable for robot policy adaptation. Compared with standard robot datasets, it contains generated egocentric human-hand manipulation videos, language prompts, reconstructed hand trajectories in MANO space, deployment-aligned scene visuals, and appearance-edited robot-hand variants (Chen et al., 20 Jun 2026).

After processing, the effective supervision includes current visual observations, the language instruction, a FoV token, current hand state, future hand-action chunks, reconstructed wrist translations, reconstructed wrist rotations, reconstructed MANO joint angles, and edited versions of frames with robot-hand appearance. In this respect, WM-H is close to action-labeled manipulation data rather than weakly labeled video (Chen et al., 20 Jun 2026).

3. Scene alignment and embodiment alignment

Scene alignment is implemented by first capturing real background images from the target robot workspace using the same deployment camera, the same viewpoint, and the same resolution as policy input. A human hand is intentionally placed in the scene during capture as a scale anchor, preserving realistic relationships among hand size, object size, and camera-to-workspace distance. For each instruction, a workspace image is chosen and objects are inserted using Qwen-Image-Edit, with object insertion spatially localized using a rectangular guide region sampled in the reachable workspace (Chen et al., 20 Jun 2026).

From the edited initial frame, Wh0 uses Wan-I2V-A14B to generate an egocentric human-hand manipulation video conditioned on the instruction. To improve temporal correctness, Qwen3-VL is first used to infer an expected description of hand-object state changes from the image and instruction, and that generated dynamics description is appended to the final video prompt. For efficiency, the framework uses LightX2V LoRA adapters, reducing generation to four inference steps. The reported generation cost is approximately 5.44 GPU-hours per 1k videos, a figure that is important because the paper’s central claim is scalability through compute (Chen et al., 20 Jun 2026).

Embodiment alignment addresses the fact that generated videos still contain human hands, while the deployed system uses an Inspire dexterous hand. Wh0 performs visual editing on sparsely sampled frames using Qwen-Image-Edit to replace the human hand with a realistic robot hand while preserving hand pose, hand position, hand scale, object motion, and scene composition. The stated purpose is not to synthesize true robot kinematics. Rather, the same manipulation trajectory is shown under a different appearance so that the model learns that the action semantics should remain stable across executor identity (Chen et al., 20 Jun 2026).

These two alignment mechanisms define the method. Scene alignment preserves deployment-relevant viewpoint and workspace geometry; embodiment alignment reduces the visual discrepancy between human-hand observations and robot-hand deployment. The ablations show that both materially affect transfer. Without scene alignment, success drops to 20.0%; without embodiment alignment, success drops from 38.9% to 34.7% (Chen et al., 20 Jun 2026).

4. Hand reconstruction, action representation, and shared MANO space

A key step in Wh0 is converting generated videos into explicit action labels. Following VITRA, the framework reconstructs 3D hand poses from human-hand videos using HaWoR, described as “world-space hand motion reconstruction from egocentric videos.” HaWoR detects hands in each frame and regresses MANO parameters plus wrist pose. The retained representation is wrist pose in camera space and articulated hand pose in MANO parameter space. If needed, camera tracking from MegaSAM associates framewise predictions with the camera trajectory (Chen et al., 20 Jun 2026).

The policy action space is defined in the camera coordinate frame of the current observation $o_t$. The action is

$\Delta t, \Delta r \in \mathbb{R}^3$5

where $\Delta t, \Delta r \in \mathbb{R}^3$ are relative wrist translation and wrist rotation between consecutive frames, $\theta_h \in \mathbb{R}^{15 \times 3}$ are local joint rotations of the 15-DoF MANO hand model, and superscripts $l, r$ indicate left and right hands. In practice, the work focuses on right-hand manipulation. This gives $51$ dimensions per hand and $102$ dimensions total for two hands (Chen et al., 20 Jun 2026).

The paper assumes that robot joint configurations can be retargeted into MANO space, allowing both human and robot trajectories to be expressed in a common hand-action representation. Robot demonstrations are converted by transforming robot states and actions from the robot base frame to the camera frame, correcting wrist rotations to align with MANO conventions, retargeting robot joints into the MANO hand space, and normalizing robot actions and states using human-data normalization statistics precomputed by VITRA. The authors explicitly avoid robot-specific normalization from only 400 demos, arguing that reuse of large-scale human normalization keeps robot actions aligned with the pretrained model’s representation (Chen et al., 20 Jun 2026).

This shared MANO-space formulation is central to the framework. It provides the bridge through which generated human video can become executable robot-learning signal. A plausible implication is that Wh0 depends less on direct visual imitation of robot embodiments than on representational compatibility between human and robot trajectories.

5. Policy architecture and post-training procedure

Wh0 uses a VITRA-style vision-language-action policy built on a PaliGemma2-3B vision-language backbone. The backbone encodes the current image observation, the language instruction, a 2D FoV token projected by an MLP, and a learnable cognition token. The final hidden state of the cognition token serves as the conditioning representation for the action decoder. The decoder is a diffusion-based DiT-B network that predicts 16 future action steps per chunk in the MANO-based action space (Chen et al., 20 Jun 2026).

The current hand state is represented in the camera frame by wrist translation, wrist Euler angles, and 15-DoF MANO joint rotations for each hand. Internally, the model uses the unified VITRA state/action space with state dimension 212 and action dimension 102, although only the human-hand-related dimensions are active here. The diffusion decoder is trained using the standard noise-prediction MSE objective,

$\Delta t, \Delta r \in \mathbb{R}^3$6

where $\epsilon$ is Gaussian noise sampled from $\mathcal{N}(0, 1)$, $i$ is the sampled diffusion timestep, and $\hat{\epsilon}_i$ is the model’s predicted noise at step $\Delta t, \Delta r \in \mathbb{R}^3$0 (Chen et al., 20 Jun 2026).

The method is explicitly a post-training approach rather than training from scratch. Initialization comes from VITRA, pretrained on Ego4D, Epic-Kitchens, EgoExo4D, and Something-Something-V2. The robot adaptation set contains 400 expert teleoperated trajectories collected using Apple Vision Pro on seen tasks and backgrounds. Wh0 uses a dataset-scale ratio of approximately 125:1 between WM-H and teleoperation data, corresponding to 50k WM-H samples and 400 teleop demonstrations. Batch sampling deliberately oversamples robot data, with each batch consisting of 28% teleop, 68% WM-H, and 4% WM-H EA, written as

$\Delta t, \Delta r \in \mathbb{R}^3$7

(Chen et al., 20 Jun 2026).

During post-training, the vision encoder is frozen, while the remaining VLM backbone and diffusion action decoder are updated. The stated optimization details are: 4 NVIDIA H200 GPUs, per-GPU batch size 64, total batch size 256, learning rate $\Delta t, \Delta r \in \mathbb{R}^3$1, weight decay 0.1, Adam betas $\Delta t, \Delta r \in \mathbb{R}^3$2, gradient clipping 1.0, max steps 40k, diffusion steps during training 100, a squared-cosine noise schedule, diffusion training repeated 8 times per batch with independent noise and timestep sampling, and no image augmentation. At inference, a single RTX 4090 is used with DDIM sampling, 10 denoising steps, and classifier-free guidance scale 5.0 (Chen et al., 20 Jun 2026).

6. Experimental evaluation, baselines, and results

All experiments are conducted on a Unitree G1 humanoid equipped with Inspire dexterous hands and a head-mounted egocentric camera. The training data consists of 400 teleoperated expert trajectories, mainly pick-and-place over seen objects and backgrounds. The evaluation suite contains 18 real-world dexterous manipulation tasks across four scenes, including tasks such as “grasp the tripod,” “put the coke can into the black box,” “grasp the remote controller,” and “put the tissue into the yellow basket.” Evaluation is zero-shot on unseen tasks, environments, and instructions, with 20 trials per task, randomized object poses and scenes, success rate as the primary metric, and no manual action correction during execution. For multi-stage tasks, the evaluator provides the next stage instruction when the previous phase is completed but does not manually intervene in behavior (Chen et al., 20 Jun 2026).

There is also a deployment-time grasping prior: before contact, finger joints are constrained to close monotonically until stable grasp. The same rule is applied to all methods. This matters because the reported performance differences are not attributable to a method-specific execution heuristic (Chen et al., 20 Jun 2026).

The main baselines are $\Delta t, \Delta r \in \mathbb{R}^3$3, a robot-data-pretrained VLA adapted using teleop data only; VITRA, a human-video-pretrained dexterous VLA adapted using the same teleop data only; VITRA Real Version, which replaces WM-H with real egocentric human-hand videos from HOI4D; and Wh0, which uses teleop plus WM-H during co-fine-tuning. The appendix notes that HOI4D is processed into 5,511 annotated episodes by grouping every 100 frames into one episode and assigning one language annotation (Chen et al., 20 Jun 2026).

The main quantitative result is the improvement in zero-shot real-world dexterous manipulation success:

Method	Success rate
$\Delta t, \Delta r \in \mathbb{R}^3$4	7.78% ± 15.6
VITRA	8.3% ± 8.6
VITRA Real Version	21.4% ± 23.4
Wh0	38.9% ± 19.8

Wh0 therefore improves zero-shot success on unseen tasks from 8.3% for the teleop-only VITRA adaptation to 38.9%, a 4.7× improvement (Chen et al., 20 Jun 2026). The main empirical pattern is that teleop-only adaptation is weak, adding extra human video during post-training helps, and generated, deployment-aligned WM-H helps substantially more than real but misaligned egocentric human video. This suggests that alignment matters more than merely using “real” data (Chen et al., 20 Jun 2026).

The paper also introduces Hand-Object Distance as a proxy for language-conditioned object grounding. Lower HO is better. The ablations probe scene alignment, embodiment alignment, and dataset scale. Reported values include: Teleop only with HO(human) 16.2, HO(robot) 16.2, success 8.3; w/o scene alignment with HO(human) 14.9, HO(robot) 14.3, success 20.0; w/o embodiment alignment with HO(human) 10.2, HO(robot) 13.8, success 34.7; WM-H 5k with HO(human) 11.9, HO(robot) 10.5, success 27.8; WM-H 25k with HO(human) 11.4, HO(robot) 9.9, success 32.5; and Wh0 (50k) with HO(human) 10.6, HO(robot) 9.6, success 38.9 (Chen et al., 20 Jun 2026).

These ablations indicate that scene alignment, embodiment alignment, and scale each contribute materially. Performance improves steadily from 5k to 25k to 50k generated episodes. The authors also report that embodiment alignment stabilizes action features under hand-appearance changes, measured by action-feature cosine similarity (Chen et al., 20 Jun 2026).

7. Interpretation, limitations, and relation to adjacent directions

One of the paper’s most explicit interpretive claims concerns “unlocking pretraining priors.” The analysis compares models with and without strong human-video pretraining. The reported table gives PaliGemma pretrain, Teleop with Hand-Object Distance 14.3 and task success 0.8; PaliGemma pretrain, Teleop + WM-H with 12.7 and 0.6; Human pretrain only with 13.1 and 0.0; Human pretrain, Teleop with 16.2 and 8.3; and Wh0 with 10.6 and 38.9 (Chen et al., 20 Jun 2026). The paper’s conclusion is that WM-H does not teach dexterous manipulation from scratch; rather, it activates capabilities already latent in the pretrained VLA. Without strong human-video pretraining, adding WM-H does almost nothing; human pretraining alone gives some grounding but no deployable success; teleop alone partially adapts but overfits and undergeneralizes; and only the combination of human-video pretraining, teleop data, and WM-H yields large gains (Chen et al., 20 Jun 2026).

The framework includes several quality-control mechanisms: duplicate instruction filtering, controlled object insertion and non-overlap sampling, manual filtering of incorrect episodes for Hand-Object Distance evaluation, user studies on realism and training suitability, and appendix discussion of failure cases. In a user study with 72 AI practitioners, 37.7% of synthetic videos were judged as real. Average quality scores were 3.97 ± 1.22 for object correctness, 4.18 ± 1.09 for instruction alignment, 3.95 ± 1.19 for hand-object interaction, 3.78 ± 1.30 for physical plausibility, and 3.57 ± 1.31 for training suitability. After robot-hand editing, scores were 4.30 ± 0.85 for pose consistency and 4.25 ± 0.84 for contact preservation (Chen et al., 20 Jun 2026). These values support the narrower claim that the generated data is noisy but usable.

The paper is also explicit about failure modes. Reported issues include image editing errors such as wrong object placement and cropping artifacts; physically implausible interactions such as hand passing through objects and unrealistic grasps; temporal inconsistencies such as mismatch between pre- and post-interaction visuals; invalid instructions that are not executable; and robot-hand editing issues in which pose is not perfectly preserved or scene objects are unintentionally modified (Chen et al., 20 Jun 2026). At a broader level, the method is limited by video generation quality, hand reconstruction accuracy, human-robot morphology mismatch, dependence on strong human-video pretraining, and task scope. The experiments mostly focus on single-arm pick-and-place style manipulation, and the paper explicitly notes that extending to bimanual tasks, more complex tool use, and longer-horizon manipulation remains future work (Chen et al., 20 Jun 2026).

In relation to broader research directions, Wh0 resembles sim-to-real transfer in that it creates synthetic training data for real deployment, but it does not rely on explicit physics or precise robot dynamics. It uses generative video instead of a simulator. It also resembles imitation learning from demonstrations, but it reconstructs human hand motion and aligns robot action space to human MANO space rather than directly cloning robot actions. Within egocentric video learning and robot foundation models, Wh0 suggests a path in which large pretrained VLAs are adapted using compute-generated post-training corpora, not only more human labor or more teleoperation (Chen et al., 20 Jun 2026).

The most novel aspect of the framework is the reframing of world models as controllable data engines for deployment-targeted manipulation supervision. A plausible implication is that its significance lies less in any single architectural component than in the coordinated combination of controllable generation, scene alignment, embodiment alignment, shared action representation, and limited real-robot co-training.