WorldModelBench for Video World Modeling

• WorldModelBench is a benchmark suite for video generation models, assessing instruction execution, physical consistency, and commonsense plausibility.
• It utilizes a diverse dataset with 350 condition pairs and over 67,000 high-density annotations, combined with an automated judger for scalable assessment.
• The framework enables reward-based fine-tuning to improve model performance in robotics, autonomous driving, industrial manipulation, and simulation gaming.

WorldModelBench is a benchmark suite developed to rigorously evaluate the world modeling capabilities of modern video generation models in application-driven domains such as robotics, autonomous driving, industrial manipulation, simulation gaming, and animation. Unlike legacy video generation metrics that emphasize visual fidelity or generic alignment, WorldModelBench explicitly incorporates instruction-following, adherence to fundamental physics, and commonsense plausibility as core axes of evaluation. By crowd-sourcing high-density human annotations and developing an automated judger model aligned to these preferences, it provides a standardized, fine-grained, and scalable protocol to assess—and improve—the ability of video models to simulate physically consistent and task-accurate futures (Li et al., 28 Feb 2025). This comprehensive evaluative tool addresses the longstanding gap between visually compelling synthesis and true world-model realism required for embodied or decision-critical AI.

1. Motivation and Distinctive Evaluation Criteria

WorldModelBench was conceived in response to fundamental shortcomings in existing video generation benchmarks. Standard metrics—such as Fréchet Video Distance (FVD) or text-video alignment scores (CLIPSIM)—quantify appearance and temporal coherence at the pixel or feature level but fail to detect crucial violations relevant to robust world modeling, such as failures of physics (e.g., objects floating in mid-air) or neglect of task instructions (e.g., the agent does not carry out the directed manipulation). The benchmark is designed to reveal subtle but consequential errors, including violations of mass conservation or solid mechanics, which are crucial for safety and reliability in autonomy and robotics (Li et al., 28 Feb 2025).

WorldModelBench evaluates two primary dimensions:

• Instruction-Following: Does the model execute the intended high-level action as instructed in the conditional prompt (e.g., robot arm inserts block into box)? Graded on a 0–3 ordinal scale reflecting null, incorrect, partial, or fully correct execution.
• Physics Adherence: Does the generated video respect foundational physical laws (inertia, mass conservation, fluid mechanics, impenetrability, gravity)? Each is scored as a binary check, totaling 0–5 per sample.
• Commonsense Plausibility: Captures obvious visual/temporal failures such as frame corruption or flicker, with two discrete flags.

This triad goes beyond coarse yes/no or pairwise ratings, providing finer discrimination of nuanced world model breakdowns.

2. Dataset Composition and Annotation Protocol

The WorldModelBench dataset comprises 350 diverse conditioning pairs, each specification comprising a text prompt and a first-frame image. These are drawn from seven major domains—autonomous driving, robotics, human activities, industrial settings, natural scenes, gaming simulation, and animation—subdivided into 56 specific task categories. For each condition, candidate models generate short videos (typically 3–5 seconds at 16 fps) under text-to-video or image-to-video settings (Li et al., 28 Feb 2025).

To enable robust, scalable assessment, 65 crowd annotators generated over 67,000 individual ratings—eight judgments per video, capturing all sub-dimensions (instruction, five physics laws, two commonsense checks). Strong quality control was enforced through duplicate annotation (average 1.7 votes per video), cross-protocol pairwise consistency checks (70% agreement with arena-style ranking), and grounding in an expert-validated standard (96.2% of expert scores within one standard deviation of the crowd mean).

3. Judger Model: Design, Training, and Performance

To facilitate efficient and objective scoring at scale, the benchmark includes a fine-tuned vision-language "judger" model. Based on a 2B-parameter architecture (VILA-2B or Qwen2-VL), the judger is supervised to replicate human ratings using cross-entropy loss over individual question-answer pairs derived from the multi-factor annotation protocol. The model demonstrates 8.6% higher average accuracy than GPT-4o (2B) at predicting world model violations on held-out videos. This performance extends across all evaluation axes: reducing error rates on instruction, physics, and commonsense by 8–12 percentage points compared to baseline LLM-based judges (Li et al., 28 Feb 2025).

4. Reward-Based Fine-Tuning and Model Response

WorldModelBench is not only an evaluative suite but also a source of reward signals for improving video generation models. By summing sub-rewards $R_g(x,c)$ for each grading criterion $g$ to construct an overall reward $R(x,c)$, the judger enables reward-based fine-tuning of generators. The objective function for generator parameters $\theta$ is

$$J(\theta) = \mathbb{E}_{c\sim D, x\sim\pi_\theta(\cdot|c)}[R(x, c)].$$

Gradients can be approximated through the discrete logits corresponding to correct/incorrect answers, e.g., maximizing $$\log p_R(\mathrm{Yes}|x,c) - \log p_R(\mathrm{No}|x,c)$$. Empirical application to OpenSora-v1.2 demonstrates an increase in aggregate WorldModelBench score from 6.17 to 6.56 and qualitative improvements such as reduced flicker, fewer gravity violations, and improved instruction execution (Li et al., 28 Feb 2025).

5. Results: Model Rankings, Domain-Level Insights, and Violation Profiles

Benchmarked models include both closed-source (KLING, Minimax, Mochi-official) and open-source representatives (OpenSoraPlan-T2V, Mochi). Performance is summarized in the following table:

Model	Total Score (/10)	Correct Instruction Execution (%)	Mass Violation Rate (%)	Gravity Violation Rate (%)
KLING (closed)	8.82	61	~12	~7
Minimax (closed)	8.59	—	—	—
Mochi-official	8.37	—	—	—
Mochi (open)	7.62	—	—	—
OpenSoraPlan-T2V	7.61	—	—	—

Across all evaluated models, three physics categories predominate as sources of error: mass conservation violation (12%), object interpenetration (11%), and gravity violation (7%). Performance stratifies by domain, with the hardest scenarios being autonomous driving maneuvers, complex human activities (such as throwing or gymnastics), and robotic manipulation (such as opening latches). Easier categories include natural scenes and animation/gaming, likely due to simpler backgrounds and fewer dynamic interactions (Li et al., 28 Feb 2025).

6. Design Principles and Comparison to Related Benchmarks

WorldModelBench uniquely centers fine-grained semantic and physics-based evaluation, whereas prior video generation benchmarks chiefly track fidelity or alignment:

• Instruction-following and physics: Unlike existing metrics that evaluate visual realism, WorldModelBench can detect subtle violations (e.g., shape or volume drift, unnatural object trajectories) that impact agent reliability in embodied systems.
• Human-aligned scoring: Its massive annotated corpus enables both high trust in results and the development of efficient, high-accuracy automated scoring.
• Cross-domain generality: The coverage of multiple high-level application domains permits broad conclusions about the generalization and robustness of learned video world models.

Compared to alternatives such as VideoPhy (coarse physics checks) or SimWorld (simulator-conditioned scene generation for perception), WorldModelBench sets a new standard in linking video generative quality directly to actionable world-model fidelity (Li et al., 18 Mar 2025, Chen et al., 4 Jun 2025).

7. Resources, Impact, and Future Directions

All dataset resources, judger model code, pre-trained weights, and evaluation scripts are openly available at https://worldmodelbench-team.github.io. The results suggest significant headroom: even state-of-the-art models complete correct instructions only 61% of the time, and physics violations are frequent.

This benchmark lays the empirical foundation for the next generation of world models suitable for embodied AI, autonomous vehicles, and decision-making pipelines by making behavioral and physical fidelity a first-class evaluation criterion. Further advances are expected through integration with reward-based fine-tuning, more challenging physical scenarios, and unified hybrid protocols that test both open-domain generalization and domain-specific competence (Li et al., 28 Feb 2025).