Rethinking Video Generation Model for the Embodied World

Abstract: Video generation models have significantly advanced embodied intelligence, unlocking new possibilities for generating diverse robot data that capture perception, reasoning, and action in the physical world. However, synthesizing high-quality videos that accurately reflect real-world robotic interactions remains challenging, and the lack of a standardized benchmark limits fair comparisons and progress. To address this gap, we introduce a comprehensive robotics benchmark, RBench, designed to evaluate robot-oriented video generation across five task domains and four distinct embodiments. It assesses both task-level correctness and visual fidelity through reproducible sub-metrics, including structural consistency, physical plausibility, and action completeness. Evaluation of 25 representative models highlights significant deficiencies in generating physically realistic robot behaviors. Furthermore, the benchmark achieves a Spearman correlation coefficient of 0.96 with human evaluations, validating its effectiveness. While RBench provides the necessary lens to identify these deficiencies, achieving physical realism requires moving beyond evaluation to address the critical shortage of high-quality training data. Driven by these insights, we introduce a refined four-stage data pipeline, resulting in RoVid-X, the largest open-source robotic dataset for video generation with 4 million annotated video clips, covering thousands of tasks and enriched with comprehensive physical property annotations. Collectively, this synergistic ecosystem of evaluation and data establishes a robust foundation for rigorous assessment and scalable training of video models, accelerating the evolution of embodied AI toward general intelligence.

• The paper introduces two key contributions—RBench and RoVid-X—that address the limitations in physically plausible robotic video generation.
• It proposes a multi-level evaluation framework combining physical-semantic plausibility, task-adherence, stability, and motion smoothness to assess model performance.
• Empirical results demonstrate that scaling and domain-specific data enhance physical reasoning, revealing a significant media-simulation gap in general video models.

Rethinking Video Generation Models for the Embodied World: A Systematic Benchmark and Dataset for Robotic Video Generation

Introduction and Motivation

The intersection of video generation models and embodied AI has intensified interest in leveraging foundation video models as simulators for robot-world interaction, with the potential to address the scarcity of large-scale teleoperated robot datasets and provide an alternative source of data for perception, reasoning, and action. However, progress in robotic video generation is bottlenecked by two systemic issues: the inadequacy of standardized evaluation frameworks that are sensitive to physical realism and task alignment, and the lack of sufficiently diverse, high-fidelity robot-centric video datasets capturing the breadth of real-world interactions.

This work addresses these gaps by introducing RBench, a comprehensive benchmark for the assessment of robot-world video models, and RoVid-X, the largest curated open-source robotic video dataset for training generative video models. The paper systematically quantifies the limitations of current foundation video models when applied to physically plausible robotic tasks and provides a scalable infrastructure for benchmarking and model development (2601.15282).

Figure 1: Overview of RBench and RoVid-X: RBench enables embodiment-aware, task-level evaluation using reproducible automated metrics, validated against human judgment; RoVid-X provides 4 million annotated clips capturing robot skill and physical attributes.

RBench: Embodiment-Centric Benchmark for Robotic Video Generation

Benchmark Structure

RBench is designed as a task- and embodiment-diversified evaluation suite, containing 650 meticulously curated image–text pairs distributed across five key robotic task categories—Common Manipulation, Long-horizon Planning, Multi-entity Collaboration, Spatial Relationship, and Visual Reasoning—and spanning four embodiment classes: Dual-arm, Humanoid, Single-arm, and Quadruped robots. Each evaluation sample is verified for prompt/task alignment and filtered to remove any overlap with training data, ensuring robust, fair assessment. The benchmark is further stratified by object categories and environment types to enable analysis of model behavior along relevant axes of physical generalization.

Figure 2: RBench statistics ensure broad coverage over robot types, manipulation skills, and diverse physical contexts, crucial for robust embodied model evaluation.

Evaluation Metrics

Contrary to existing perceptual or physics-only metrics, RBench incorporates multi-level automated metrics:

• Physical-Semantic Plausibility: Penalizes violations such as floatation, interpenetration, object detachment without contact, and spurious entity emergence.
• Task-Adherence Consistency: Enforces correct action ordering, key action presence, and prompt responsiveness.
• Robot-Subject Stability: Assesses both robot morphology and object identity consistency across video frames.
• Motion Amplitude / Smoothness: Separates true subject motion (robot articulation) from background/camera motion, countering the “static-yet-smooth” failure case.

A multi-modal LLM-based VQA protocol underpins assessment, combining high-level reasoning with low-level computational indicators.

Figure 3: RBench captures failure cases—structural distortion, floating artifacts, omitted actions—missed by traditional perceptual metrics.

RoVid-X: Large-Scale, Physically-Annotated Robotic Video Dataset

Data Construction Pipeline

RoVid-X is assembled via a four-stage pipeline:

1. Robot Video Collection: Aggregates data from >20 open-source robot datasets and large-scale web video scraping, using LLM-based filtering to assure relevance.
2. Quality Filtering: Automated screening enforces standards in clarity, motion, aesthetics, and robot content using multi-dimensional video quality scorers.
3. Task Segmentation and Captioning: Action-centric segmentation creates short clips annotated with temporally precise, model-generated task captions, enabling fine-grained supervision.
4. Physical Attribute Annotation: Enhances videos with optical flow (AllTracker), depth maps, and super-resolved frames (FlashVSR), supporting physical realism and downstream analysis.

RoVid-X encompasses 4 million annotated clips covering >1,300 task types, with high-resolution imagery and dense physical attribute labeling.

Figure 4: RoVid-X construction stages and dataset statistics: ablation by frame intervals, skill breadth, and object taxonomy validates coverage for training physical world models.

Dataset Properties and Comparative Scope

Compared to prior datasets—which are typically limited in both task diversity and scale—RoVid-X is unique in its breadth of robot morphologies, task diversity, and standardized physical annotation. The corpus supports tasks ranging from basic object manipulation to long-horizon reasoning and multi-agent interaction, capturing physical priors for embodied learning. When models are finetuned on RoVid-X, substantial gains are observed in both task and embodiment-specific metrics, validating its utility for scalable foundation model training.

Empirical Evaluation and Insights

Large-Scale Model Assessment

Twenty-five state-of-the-art video generation models, comprising open-source, commercial, and robotics-specific lines, are evaluated on RBench. The results reveal several important trends:

1. Visual Fidelity vs. Physical Intelligence

Top commercial models (e.g., Wan 2.6, Seedance 1.5 Pro) excel in physically grounded video generation, outstripping both open-source and consumer (media-oriented) models by a wide margin. The Sora series, despite its prominence in generic video synthesis, performs markedly worse on physically plausible robotic video generation tasks—exposing a clear "media-simulation" domain gap.

2. Iterative Scaling and World Modeling

Model performance increases monotonically with scale and iteration within product lines. Wan demonstrates a leap in physical reasoning and control fidelity from v2.1 to v2.6; similarly, Seedance upgrades yield substantial numerical improvements—underscoring the role of scaling in learning physical priors, not just pixel-level features.

3. General Foundation Models vs. Robotics-Specific Models

Generalist foundation models dominate in aggregate scores, but Cosmos 2.5, designed for robotics, demonstrates superior robustness relative to similarly sized open-source generalists, confirming the utility of domain-specific physical data. Overly specialized models, such as those trained only for a single embodiment or with limited task scope, underperform, highlighting that balance of physical, domain, and world knowledge is necessary for transferability.

4. Cognitive Reasoning and Fine Motor Control Bottlenecks

The most prominent bottlenecks across all models are visual reasoning tasks (logic, attribute matching, key action sequencing) and fine-grained manipulation requiring precise object-robot interaction. The “manipulation gap” (vs. gross locomotion) and “cognitive gap” (vs. execution-only tasks) remain persistent challenges for deep generative video models.

Figure 5: Qualitative comparison—representative tasks reveal correct vs. spurious reasoning, incomplete action sequences, and physical implausibilities unique to robotic video synthesis.

Metric Validity and Human Alignment

Automatic RBench metrics are validated against a 30-participant human preference study, yielding a Spearman rank correlation of 0.96; Bland-Altman analysis further confirms absolute agreement post-calibration. This establishes that the automated benchmarks—focusing on higher-level physical and semantic realism—track human perception sufficiently for reliable model comparison.

Figure 6: User study questionnaire interface used for human preference validation against RBench metrics.

Implications and Future Directions

The introduction of RBench and RoVid-X directly addresses critical bottlenecks in embodied video modeling evaluation and training. The benchmark’s ability to surface nuanced model failures—missed by standard FID or aesthetic scores—sets a new bar for evaluating video world models designed for control and simulation. RoVid-X provides the prerequisite data scale and physical fidelity for learning embodied priors beyond what is available in existing open robot datasets.

Practical implications include:

• Policy Extraction: Video models evaluated on RBench can serve as a substrate for policy learning via IDM or latent action models, moving toward closed-loop control or robot imitation from synthesized videos.
• Physics-Informed Model Development: The observed "media-simulation" gap demonstrates the need for physically-informed architectures and training regimens targeting embodied domains, not just visual quality.
• Open-Source Deficit: Substantial performance gaps remain between closed and open foundation models, motivating further efforts in dataset scaling, physical annotation, and domain-aware architectures.

Theoretically, expanded benchmarks like RBench may prompt new research in disentangled world modeling, better physical reasoning within diffusion and transformer architectures, and multi-modal evaluation protocols for embodied agents.

Conclusion

By systematizing benchmark-driven evaluation (RBench) and scalable data curation (RoVid-X), this work establishes a robust infrastructure for advancing video generation in embodied robotics. The gap between task-consistent, physically plausible video generation and the requirements of real-world robot learning is now quantitatively measurable, setting a foundation for iterative progress. Future directions include closed-loop control evaluation, further metric automation, and the development of generative models capable of both aesthetic and physically grounded performance across diverse robot-world interactions.