Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control

Abstract: World models are rapidly becoming a core infrastructure for embodied intelligence and interactive agents: they provide controllable simulators in which agents can perceive, act, forecast, and acquire scalable experience. Yet current video generation world models are still organized around isolated control interfaces, such as camera trajectories, robot actions, or hand-joint signals. This fragmentation is increasingly a scaling bottleneck. The central challenge is not the absence of controllable generators, but the lack of a unified and extensible learning framework that can absorb heterogeneous action supervision while preserving a shared model of world dynamics. In this work, we introduce Worldscape-MoE, a Mixture-of-Experts world model built on Diffusion Transformers for scalable heterogeneous action control. Our key observation is that different controls specify different interfaces to the same underlying world: although their representations differ, they constrain shared physical regularities, scene dynamics, and interaction semantics. Worldscape-MoE operationalizes this observation through modality-aware control injection, shared and control-specific experts, and a progressive MoE tuning strategy that supports continual extension to new action modalities. Experiments across locomotion, robotic manipulation, and egocentric hand control show that heterogeneous supervision improves rather than interferes with individual control capabilities. Worldscape-MoE achieves strong results on WorldArena, improves locomotion and hand-control metrics, exhibits robust out-of-distribution generalization, and demonstrates scaling behavior as additional control data and experts are integrated.

• The paper introduces Worldscape-MoE, a unified MoE-based world model that integrates diverse control modalities using a shared expert and modality-specific pathways.
• It employs a diffusion-transformer backbone with progressive tuning, achieving state-of-the-art performance on locomotion, manipulation, and hand-motion benchmarks.
• Experimental results demonstrate improved temporal coherence, physical plausibility, and scalable extensibility through sparse expert routing and deterministic gating.

Worldscape-MoE: A Unified MoE World Model for Scalable Heterogeneous Action Control

Motivation and Problem Formulation

World models have become critical infrastructure for embodied intelligence due to their ability to simulate visuomotor dynamics under explicit action controls. Current video-based world models, however, are fragmented along control modality boundaries—separately modeling camera motion, robot actions, or egocentric hand pose trajectories. This architectural siloing severely restricts scalability, reuse of visual priors, and cross-modal transfer.

"Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control" (2607.03964) addresses this fragmentation by proposing a unified world modeling framework capable of absorbing heterogeneous controls without sacrificing per-modality performance or cross-control interference. The central hypothesis is that diverse control signals (e.g., camera trajectory, robot action, hand pose) constrain a shared latent world and should be factorized—not isolated—in a unified generative architecture.

Worldscape-MoE Architecture

The core of Worldscape-MoE is a diffusion-transformer (DiT) backbone, where modality-specific control signals are injected through dedicated pathways, and the dense FFN layers are replaced by a Mixture-of-Experts (MoE) structure comprising a shared expert and modality-specific experts.

Modality-Aware Control Injection

Each control pathway is tailored to its data structure and semantics:

• Locomotion (Camera trajectories): Camera pose sequences are mapped via a Control Adapter into a space aligned with VAE-compressed video tokens, then fused at the patch-token level.
• Hand-Object Manipulation: Low-dimensional action sequences (e.g., robot joint commands) are embedded by a trainable MLP and injected through temporal modulation into DiT blocks.
• Hand Motion (Egocentric/Action Map): Dense action maps (spatially aligned to video frames) are VAE-encoded and fused as conditional latents.

Figure 1: Worldscape-MoE supports three mainstream control modalities (locomotion, manipulation, hand-motion) in a unified framework, extensible to additional control types.

Each sample activates a shared expert (accumulating world priors) and a single modality-specific expert (for action-specific specialization), enforced by deterministic gating and sparse routing. Newly-introduced experts are initialized from the current shared expert to promote knowledge transfer.

Figure 2: Worldscape-MoE architecture integrating modality-dependent control pathways and MoE blocks, supporting both ego-centric exploration and embodied task simulation.

Progressive Training and Extensibility

Worldscape-MoE introduces a progressive tuning strategy. Instead of introducing all experts/controls at once, new control branches and their corresponding experts are added incrementally. The shared expert receives a low learning rate to preserve pretrained priors and absorb cross-modality information, while modality-specific branches train at standard rates for efficient adaptation.

Figure 3: The shared expert is conservatively tuned; modality-specific experts undergo faster adaptation for specialization.

This schedule enables continual extension: new control modalities can be incorporated post-hoc, with only transient degradation in prior modalities and rapid recovery of original capabilities, as substantiated by empirical convergence tracking.

Experimental Analysis

Unified Performance Across Modalities

Worldscape-MoE is evaluated on three regimes: trajectory-following locomotion (iWorld-Bench), dual-arm robotic manipulation (WorldArena), and egocentric hand-motion (EgoDex/Ego4D). It consistently achieves the top scores against baselines specialized to each control type:

• Locomotion: Highest overall generation and smoothness metrics; motion smoothness score of 0.9941.
• Manipulation: EWM score 62.84 (best among all evaluated models, including CtrlWorld and Wan 2.6).
• Hand Motion: FID-VID 3.80, FVD 110.94—substantially stronger than action-specific or appearance-driven baselines.

Qualitative comparisons reveal superior temporal coherence, physically plausible dynamics, and control fidelity across all control types.

Figure 4: Worldscape-MoE generates more reliable and physically coherent world dynamics than control-specific baselines across all control modalities.

Specialization and Routing in MoE

Gate-weighted analysis reveals that the shared expert is responsible for 69.48% of computation, with modality-specific experts handling the remainder (20-48% depending on control type). Locomotion relies more on the shared expert, manipulation and hand-motion require greater per-modality specialization, validating the architectural factorization design.

Figure 5: Expert routing and workload: the shared expert dominates, but each control leverages its dedicated expert according to inherent modeling difficulty.

Ablation (removing MoE, using dense FFN) leads to significant cross-control interference and degraded metrics, underscoring the criticality of the sparse expert factorization.

Figure 6: MoE substantially improves temporal stability and control fidelity versus direct mixed training.

Scalability and Continual Extension

When new control modalities are added, performance on prior modalities initially dips (notably in locomotion under large visual shifts), but recovers with further training. The transient nature of this degradation demonstrates the stability and extensibility of the progressive tuning regime.

Figure 7: Transient degradation of a modality on expansion, followed by efficient recovery as shared and specialized parameters re-equilibrate.

Generalization to out-of-distribution (OOD) settings (unseen tasks, objects, or compositions) is robust—qualitative outputs in all action domains are physically plausible, temporally smooth, and well-controlled.

Figure 8: Out-of-distribution rollouts demonstrating coherent responses across three motion domains.

Data Pipeline and Augmentation

Worldscape-MoE leverages a high-diversity, multi-source dataset:

• Camera control from RealEstate10k, DL3dv-10k, Sekai, and iWorld-Bench synthetic environments.
• Manipulation from RoboTwin 2.0 and Seedance 2.0 augmentation (modifying objects, backgrounds without altering action semantics).

Figure 9: Seedance 2.0 augmentation diversifies visual inputs but preserves action-conditioned dynamics, combating overfitting to specific appearances.

• Egocentric hand data from EgoDex and Ego4D, unified via monocular 3D hand mesh recovery (HaMeR) and projected 2D action maps for robust pixel alignment.

Comparative and Qualitative Evaluation

Extensive cross-baseline comparisons (Matrix-Game, HY-World, CamI2V, CameraCtrl, HunyuanVideo, Cosmos-Predict, MimicMotion, LOME, etc.) on long-horizon video prediction illustrate that Worldscape-MoE outputs are not only visually consistent but maintain action semantics and physical regularities across diverse scene domains.

(Figure 10-16)

Figures 11-16: Worldscape-MoE yields sharper, more temporally consistent locomotion rollouts compared to trajectory-controlled baselines.

(Figure 11-22)

Figures 17-22: Dual-arm manipulation predictions maintain object interaction, end-effector geometry, and background consistency exceeding existing models.

(Figure 12-25)

Figures 23-25: Egocentric hand motion: Worldscape-MoE exhibits high-fidelity control response and scene plausibility relative to state-of-the-art baselines.

Implications and Future Directions

Worldscape-MoE demonstrates that joint, MoE-mediated training on heterogeneous control data is not only viable but enhances overall capability, reinforcing rather than degrading individual control specialization. This enables unified infrastructure for scalable embodied intelligence, transfer to new tasks (e.g., single-arm LIBERO manipulation with minimal finetuning), and cross-modal generalization.

Potential future directions include:

• Distillation and model compression for real-time deployment.
• Incorporation of broader control modalities (multi-agent, language-programmed policies).
• Safeguards for misuse (deepfake detection, provenance) alongside increasing simulation fidelity and agent-level data generation.

Conclusion

Worldscape-MoE provides a principled solution to the long-standing scalability bottleneck in video-based world modeling. By architecturally disentangling shared world dynamics from control-specific specialization and leveraging Mixture-of-Experts for sparse, modular computation, the framework achieves unified, extensible, and high-performance world generation across diverse embodied and interactive control regimes. This establishes a technical foundation for scalable, heterogeneous action-conditioned simulation required by the next generation of generalist AI agents.