Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

Abstract: Action-conditioned world models allow robots to predict the future consequences of candidate actions without additional physical interaction, supporting policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model. The dynamics model predicts future segmentation masks from past masks and 23-DoF action sequences. The rendering model maps the predicted masks to photorealistic RGB using a ControlNet-augmented Stable Video Diffusion backbone. The smaller sim-to-real gap in segmentation space enables the dynamics model to benefit from large-scale pretraining on over 50 h of synthetic simulation data, followed by fine-tuning on fewer than 2.5 h of real demonstrations. Experiments on a dexterous pick-and-place benchmark show that mask conditioning and simulation pretraining are both required for per-DoF action controllability across all 23 degrees of freedom. In contrast, monolithic baselines capture broad hand and end-effector trajectories but do not reliably reflect fine-grained, per-joint action effects.

• The paper introduces a two-stage model that decouples segmentation mask prediction from RGB rendering to bridge the sim-to-real gap.
• It leverages large-scale synthetic pretraining with minimal real-world data to achieve high fidelity and robust per-DOF controllability on a 23-DOF system.
• Empirical evaluations demonstrate improved out-of-distribution performance and independent joint actuation compared to monolithic baseline models.

Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

Introduction

Mask2Real-WM proposes a two-stage action-conditioned world model for dexterous robotic manipulation that introduces a structured decomposition: first, future segmentation masks are predicted via dynamics modeling; then, these masks are rendered as photorealistic RGB through a diffusion-based renderer. This division is motivated by the reduced sim-to-real domain gap in the segmentation space, enabling large-scale simulation pretraining of the dynamics component while requiring only minimal real-world data for the appearance renderer. The study evaluates the approach on a 23-DOF Franka Emika Panda–ORCA hand system and demonstrates enhanced controllability, out-of-distribution (OOD) robustness, and fine-grained action-conditioning compared with monolithic baselines.

Figure 1: Mask2Real-WM consists of a dynamics world model (WM1) that predicts segmentation masks and a rendering model (WM2) that maps masks to RGB frames, with the former pre-trained on simulation and the latter trained on real data.

Related Work

The action-conditioned video prediction and robotic manipulation literature is rapidly moving toward multi-modal, diffusion-based world models. While prior methods like Ctrl-World (Guo et al., 11 Oct 2025) and Cosmos (NVIDIA et al., 7 Jan 2025, NVIDIA et al., 28 Oct 2025) have achieved strong results with parallel-jaw grippers or low-DOF manipulators, extending these techniques to high-DOF systems like dexterous hands has been challenging. Furthermore, segmentation and structured intermediate representations in image/video diffusion pipelines (e.g., ControlNet (Zhang et al., 2023), Mask World Model (Lou et al., 21 Apr 2026), BridgeV2W (Chen et al., 3 Feb 2026)) have shown promise for improved spatial fidelity. Mask2Real-WM expands on these by leveraging mask-based sim-to-real transfer and introduces rigorous evaluation of per-joint controllability in complex dexterous settings.

Methodology

Problem Formulation and Factorization

Action-conditioned video prediction is formulated as modeling the distribution over future images, conditioned on past images, segmentation masks, and a sequence of robot actions ($6$-DoF end-effector + $17$-DoF hand joints, totaling $23$). Mask2Real-WM factors this predictive distribution into:

• Dynamics model (WM1): predicts future segmentation masks, $p(m_{t+1:t+H} \mid m_{t-k:t}, a_{t-k:t+H})$.
• Rendering model (WM2): generates RGB frames conditioned on masks, $$p(I_{t+1:t+H} \mid I_{t-k:t}, m_{t-k:t+H}, a_{t-k:t+H})$$.

This split exploits the near-invariance of mask dynamics across sim and real, decouples appearance from structure, and empowers WM1 with large-scale synthetic data while confining real-data collection to WM2.

Figure 2: The left branch (WM1) models dynamics in mask-space, while the right branch (WM2) renders photorealistic video from predicted masks; inference chains both modules autoregressively.

Simulation and Real Data Pipelines

WM1 is pretrained on 50+ hours of IsaacLab (Mittal et al., 2023) synthetic data, using both MimicGen-driven (Mandlekar et al., 2023) demonstrations and per-joint sinusoidal exploration, spanning a wide action space and object set. WM2 is trained on less than 2.5 hours of real-world manipulation demonstrations with RGB images and segmentation masks obtained via SAM-3 (Carion et al., 20 Nov 2025).

Figure 3: Simulated data includes both structured demonstrations and exploratory motion, allowing broad coverage of the hand's joint space absent in the limited real data.

WM1: Segmentation-Dynamics Model

WM1 is a video diffusion model, parameterized as a denoising autoencoder (SVD-based (Blattmann et al., 2023), LoRA-adapted (Hu et al., 2021)), conditioned on past segmentations and actions. Training objectives are applied in mask-latent space, exploiting the structured, low-entropy nature of rasterized segmentation images to close the sim-to-real gap via synthetic pretraining, with final fine-tuning on real data.

WM2: ControlNet-Augmented RGB Renderer

WM2 employs the same SVD backbone as WM1, operating in RGB space and incorporating predicted masks as structure via a ControlNet branch (Zhang et al., 2023). Masks are encoded using a CNN that specializes in boundary preservation (as opposed to the VAE encoder), injected into the diffusion U-Net via ControlNet-style conditioning. Action and past-image cues are embedded with cross-attention, and classifier-free guidance is applied during inference.

Experimental Evaluation

Action Controllability

Controllability is measured by applying isolated sinusoids to each action component, evaluating the resulting motion on ID and OOD examples. Mask2Real-WM, benefitting from simulation pretraining and fine-tuning, achieves a per-DOF mean score of 0.95 (ID) and 0.87 (OOD), exceeding baselines which show pronounced coupling and lower OOD generalization.

Figure 4: Sinusoidal perturbations to individual action dimensions reveal Mask2Real-WM's ability to independently actuate every DOF, with substantially higher mean controllability scores than monolithic architectures.

Qualitative analysis further demonstrates independent per-joint responses to action input, with Mask2Real-WM producing structurally coherent, sharp motion across all joints and both camera views.

Figure 5: Model rollouts for each action dimension verify independent controllability of all 23 DOF, with clear separation of hand and finger motions.

Video Prediction and Sharpness

Mask2Real-WM, especially the sim+real-trained variant, maintains high-fidelity, sharp predictions both in-distribution and under distribution shift (background, object, lighting). While pure pixel-level metrics can favor smooth, blurry outputs produced by monolithic baselines, the Laplacian variance analysis demonstrates consistently higher spatial detail and edge-fidelity in Mask2Real-WM's predictions.

Figure 6: Perceptual quality (PSNR, SSIM, LPIPS) as a function of WM1 training: sim-then-real pretraining offers both strong ID and improved OOD performance.

Figure 7: Video sharpness, measured via Laplacian variance, confirms that Mask2Real-WM preserves high-frequency content, whereas end-to-end models progressively blur over time.

Ablations: Mask and Action Conditioning

Ablation studies show that ControlNet mask conditioning is the dominant driver of video spatial quality, restoring frame sharpness and spatial consistency. Action conditioning further tightens motion correspondence, especially in ambiguous wrist perspectives.

Figure 8: LPIPS is lowest when both mask and action conditioning are present, but mask-only conditioning already restores the majority of spatial detail.

Failure Modes and Generalization

Primary limitations arise from mask dynamics ambiguity: during occlusion, objects may vanish or be duplicated, underlining the need for depth or object-ID conditioned mask modeling. WM1 mask fidelity limits ultimate rendering quality.

Figure 9: Example of object vanishing when occluded by the hand; object reappears post-occlusion.

Figure 10: Example of object duplication, reflecting ambiguity in segmentation dynamics under occlusion.

Mask2Real-WM generalizes to unseen objects at test-time without dynamics retraining; rendering generalization is determined solely by the capacity of WM2 to map novel mask structures into plausible RGB.

Figure 11: Model produces coherent video rollouts for previously unseen objects (banana), confirming the transferability of segmentation-space dynamics.

Practical Modularity and Scalability

The two-stage factorization not only matches or outperforms monolithic baselines in controllability and OOD robustness, but also provides practical modularity: WM1 can function as a fast, lightweight checker of mask-level dynamics, while WM2 can be updated separately to adapt appearance to new environments. The framework's limited dependency on real-world data is critical for scaling dexterous world modeling when real data collection is prohibitive.

Implications and Future Work

Mask2Real-WM demonstrates that segmentation masks, as structured intermediate representations, effectively bridge the sim-to-real gap for high-DOF robotic systems. This structured approach provides a principled path to scalable simulation-augmented training for both policy learning and downstream planning or model-based control.

The main limitation is computational—two sequential diffusion passes are expensive. Future directions include distilling the pipeline into consistent or accelerated models (Song et al., 2023) to enable real-time inference. Enhanced mask modeling (depth, object ID, textural priors), trajectory optimization-driven simulation data, and conditioning on variable camera intrinsics/extrinsics can further broaden the applicability of the approach to multi-object, multi-view, and policy-learning scenarios.

Conclusion

Mask2Real-WM establishes a sim-to-real paradigm for action-conditioned world modeling in dexterous manipulation, leveraging segmentation masks as the structural substrate for transferable, controllable dynamics. Its structured factorization yields strong per-DOF controllability, OOD robustness, and modularity—facilitating the adoption of data-driven dexterous world models in scenarios where scalable real-world data acquisition is infeasible. The segmentation-space bridge, underpinned by large-scale simulation and minimal real fine-tuning, is well positioned to drive further advances in robotics and embodied AI.