Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

Abstract: Despite the recent promise in robot control, video generative models suffer from a domain mismatch due to their primary focus on content creation. For example, their design inherently prioritizes visual fidelity and creativity over computational efficiency and physical realism. In this work, we present LingBot-Video, a DiT-based video pretraining paradigm specifically tailored for embodied intelligence. From the architecture perspective, we adopt the Mixture-of-Experts (MoE), instead of dense, framework to achieve a better trade-off between modeling capacity and inference efficiency, and manage to scale it up from scratch. From the data perspective, we construct a data profiling engine that augments standard internet videos with extensive robot-oriented footage, encompassing manipulation, navigation, and egocentric perspectives, to equip the base model with an intrinsic understanding of actions and world dynamics. From the training perspective, we develop a multi-dimensional reward system to enforce the alignment regarding physical rationality and task completion, going beyond standard criteria such as aesthetics, prompt-following, and motion consistency. Comprehensive evaluations validate its performance and efficiency as a video foundation model. We contribute LingBot-Video as the inaugural large-scale, open-source MoE video foundation model to the community, in a pioneering effort to bridge digital creativity and physical actuation.

• The paper demonstrates a unified diffusion transformer with a sparsely-activated Mixture-of-Experts backbone for scalable, embodied video modeling.
• It introduces a data pipeline with a curated, graph-based curriculum to address data sparsity in robotics and embodied applications.
• Post-training with multi-aspect RLHF refines physical realism and task completion, outperforming dense models on key embodied benchmarks.

Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

Introduction and Motivation

The paper "Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence" (2607.07675) presents a comprehensive framework for large-scale video foundation model pretraining, specifically targeting the requirements of embodied AI and robot-centric applications. The main objective is to align generative video modeling not only with visual fidelity and text-prompt alignment, but also with physical realism and controllability—key prerequisites for downstream robotics and embodied intelligence.

Conventionally, state-of-the-art video diffusion and autoregressive models have been optimized for content creation, focusing on perceptual quality rather than accurate physical interactions or computational efficiency. This introduces a critical domain mismatch when employing such models in embodied settings, where grounding in physical dynamics and the ability to serve as environment simulators or predictive world models are essential. The paper addresses three major interdependent constraints that have previously hindered progress: (1) dense model architectures bottleneck scalability and efficiency; (2) training corpora lack sufficient embodied and robotics-specific data; (3) loss functions remain limited to aesthetic and prompt-based objectives, neglecting physical plausibility and task completion.

Unified Single-Stream Diffusion Transformer and MoE Scaling

The core architectural contribution is a unified, single-stream diffusion transformer with a scalable, sparsely-activated Mixture-of-Experts (MoE) backbone. This DiT-based architecture accepts both visual and multimodal condition tokens, using Qwen3-VL-4B for instruction encoding and Wan2.1-VAE for visual compression. All conditioning and visual tokens are processed jointly via a shared transformer stack leveraging 3D Multi-Modal Rotary Position Embedding (3D-MM-RoPE), enabling the simultaneous handling of Text-to-Image (T2I), Text-to-Video (T2V), and Text-and-Image-to-Video (TI2V) tasks within a unified computation graph.

Figure 1: Overview of the task-unified single-stream diffusion transformer, illustrating attention and Sparse MoE pathways.

The adoption of sparse MoE is central to the model's scalability and efficiency. Standard dense FFNs are replaced with routed sparse MoE layers, where each token is dispatched to a subset of specialized experts based on learned affinity scores and group-limited routing. This design allows for decoupling total model capacity (number of parameters) from per-token compute, a critical property for high-resolution, long-video generation. Experiments demonstrate clear and predictable scaling laws: increasing the expert pool size under a fixed compute budget yields consistent improvements in both training and validation loss, and models with greater expert specialization (more, smaller experts per token) outperform those using coarser routing.

Figure 2: Training/validation loss comparison for Dense 1.3B vs. MoE 13B-A1.4B, with MoE providing a notable loss improvement at fixed active compute.

Figure 3: Dense and sparse scaling curves substantiating MoE parameter efficiency.

Inference benchmarks confirm that MoE-based DiT exhibits minimal overhead compared to dense models at equal active compute, while enabling faster inference than larger dense baselines for million-token sequences.

Data Infrastructure: Profiling, Graph-Based Curation, and Curriculum

The data pipeline is meticulously engineered to overcome the lack of physically grounded, action-centric footage in existing Internet-scale corpora. A Data Profiling Engine extracts structural, semantic, kinematic, cinematic, and quality attributes for every sample, enabling sophisticated filtering, rebalancing, and captioning.

Figure 4: Data Profiling Engine schematic showcasing multi-dimensional sample annotation.

Samples are further organized within a World-Knowledge Topological Graph, integrating a semantic concept tree (for both images and videos) and an action tree (for videos) to enable targeted curriculum sampling, upweighting rare actions and underrepresented concepts. Distribution-aware sampling, informed by per-node denoising loss statistics from early training, drives balanced inclusion of challenging and long-tail phenomena.

Figure 5: The World-Knowledge Topological Graph as a semantic-action control surface for data mixing.

Training follows a staged curriculum that ramps up sample complexity and resolution in a principled manner across five phases, introducing robot-centric, navigation, egocentric, and high-resolution refinement datasets incrementally to promote optimization stability.

Figure 6: Progressive data curriculum through five pre-training stages with evolving image/video source composition.

All samples are annotated with dense, structured captions encoding scene composition, object attributes, actions, and camera properties, with a train/inference gap addressed via a dedicated Caption Rewriter.

Post-Training: Multi-Aspect RLHF and Action-Conditioned World Modeling

To align the generative model towards physical realism and robust task completion, post-training with Reinforcement Learning from Human Feedback (RLHF) is performed using multi-aspect reward modeling. Six orthogonal reward functions are constructed: vision quality, text-video alignment, dynamic degree, motion coherence, human motion consistency, and physical plausibility. The multi-reward signals are aggregated and optimized via Group Relative Policy Optimization (GRPO) using a single-step stochastic exploration protocol with coefficients-preserving sampling, enabling efficient advantage estimation and credit assignment.

Figure 7: Qualitative improvement in video quality after multi-aspect RLHF post-training, with resolution of hand synthesis and text rendering artifacts.

Figure 8: Marked enhancement of physical plausibility and reduction of non-physical errors post RLHF.

The model is also adapted as an action-conditioned world model for robot planning by integrating future action sequences through a learnable ActionEmbedder, aligned with visual latents for future trajectory rollouts.

Figure 9: Action-to-Video world model architecture—integrating future action sequences and visual latents with temporal alignment.

Quantitative and Qualitative Evaluation

Extensive evaluation demonstrates superiority across both generic video quality and robotics/embodied sub-domains. On internal and public benchmarks (RBench, Physics-IQ Verified), the model achieves state-of-the-art scores among open-source systems, particularly in manipulation, spatial reasoning, and action follow-through categories.

Figure 10: T2V quality score highlighting generative performance.

Figure 11: RBench task and embodiment-specific scores—model outperforms across manipulation and multi-entity categories.

Notably, in robot-embodied and physical simulation tasks, the model demonstrates robust performance, both with and without initial image conditioning—indicating strong intrinsic physical priors. User studies confirm consistent human preference for generated outputs compared to other prominent open-source and commercial video generation baselines, especially in the TI2V configuration.

Comprehensive qualitative results reaffirm fine-grained motion prediction, object permanence, and physical law preservation in both text-to-video and image-conditioned scenarios.

Implications and Future Directions

The architectural and data-centric innovations presented enable video foundation models to address the critical requirements of embodied AI: efficient scaling, action-centric world modeling, and physically accurate simulation. The modular sparse MoE design and progressive data curriculum collectively facilitate not only superior generative performance at scale, but also mitigate domain mismatch for robotics and embodied control.

Practically, such models are poised to serve as (a) synthetic data engines for robot learning at scale, (b) safe policy evaluators and simulators, and (c) predictive planners for closed-loop actuator control. The open-source release establishes a robust baseline and reproducible testbed for continued research in embodied video foundation models.

Theoretically, this work catalyzes further investigation into task-specific expert specialization, curriculum-driven pretraining, and the extension of multi-aspect RL post-training to high-dimensional generative spaces. Future work may extend action-conditioning to more granular interactive policies, tighter integration with real-world robot learning pipelines, and broader coverage of dynamic environmental conditions.

Conclusion

The framework in "Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence" constitutes a significant advancement in physically grounded, scalable, and efficient video foundation modeling for embodied intelligence. The integration of sparse MoE architectures, curated action-centric video corpora, structured semantic-action annotation, and multi-task RLHF post-training delivers robust embodied capabilities with high generalization. The systematic design principles detailed here will inform future embodied AI and robotic world modeling paradigms.