WorldReasonBench: Video World-State Prediction

• WorldReasonBench is a benchmark that reframes video generation as world-state prediction, assessing physical, social, logical, and informational consistency.
• It comprises 436 curated test cases with structured QA annotations spanning four reasoning dimensions and 22 subcategories to diagnose failures masked by visual quality.
• The evaluation employs Process-aware Reasoning Verification and Multi-dimensional Quality Assessment to measure outcomes, dynamics, and align with human preferences.

WorldReasonBench is a benchmark for evaluating video generators as future world-state predictors rather than as systems optimized primarily for perceptual quality. It formalizes the task as: given an initial image $x_0$ and an instruction or action $a$, a generator produces $V = G(x_0, a)$, and evaluation asks whether the generated future states remain physically, socially, logically, and informationally consistent with the intended evolution. The benchmark contains 436 curated test cases with structured ground-truth QA annotations spanning four reasoning dimensions and 22 subcategories, and it pairs this task formulation with a human-aligned evaluation stack built to expose failures in temporal dynamics, causality, and information preservation that can be masked by visually convincing outputs (Wu et al., 11 May 2026).

1. Problem formulation and conceptual scope

WorldReasonBench reframes video generation as world-state prediction. Its central question is not whether a model can synthesize plausible pixels, but whether a model that “sees” an initial visual state and receives an action or instruction can roll the world forward in a way that is temporally coherent, causally valid, and information-preserving. The benchmark is motivated by the observation that commercial systems such as Seedance2.0 and Veo3.1 have rapidly improved in visual quality and control, yet videos can still violate gravity, object permanence, causal links, or exact information fidelity while remaining superficially convincing (Wu et al., 11 May 2026).

The benchmark targets open-domain, image-conditioned video generation under TI2V conditions. Two instruction regimes are defined: implicit, which provides high-level intent only, and hinted, which provides explicit transition guidance. The assistance benefit is reported as the gap between scores under the hinted versus implicit regime. This design separates failures of world reasoning from failures that can be partially compensated by stronger textual guidance.

A common misconception is that improved realism or motion smoothness is sufficient evidence of world simulation competence. WorldReasonBench explicitly rejects that equivalence. Its evaluation is built to show that a video may look correct in still frames while failing process reasoning, temporal ordering, mechanism fidelity, or information preservation. The benchmark therefore treats visual plausibility as only one component of assessment, not the operative definition of success.

2. Benchmark construction, taxonomy, and ground truth

WorldReasonBench contains 436 curated test cases with structured ground-truth QA annotations. It spans four high-level reasoning dimensions and 22 concise subcategories (Wu et al., 11 May 2026).

Dimension	Cases	Subcategories
World Knowledge	127	Material Change; Public Systems; World Mechanics; Cultural Life; Everyday Living; Earth Cycles; Living World
Human-Centric	78	Object Handling; Social Scenes; Skilled Action; Personal Routine; Public Conduct
Logic Reasoning	131	Quantitative Math; Spatial Geometry; Experimental Science; Logic Puzzles; Pattern Discovery
Information-Based	100	Data Reading; Process Timeline; Visual Editing; Knowledge Media; Creative Expression

Each case includes 5–7 QA pairs spanning four question types: factual (28.4%), reasoning (27.1%), detail (24.7%), and temporal (19.7%). Difficulty labels are easy, medium, or hard. The QA pairs are designed to be answerable from the generated video alone and to probe both outcome and process. The QA is provided in structured JSON with question text, type, difficulty, evaluation criteria, and ground-truth answers, and answers are checked with a strict binary judge.

The examples in the taxonomy illustrate the breadth of the task. World Knowledge cases include prompts such as “Using the input image as the first frame, after the person lets go, what happens?” and “After many sunny days in this salt field, what will happen?” Human-Centric cases include “The barista is flattening the coffee powder, what series of actions will he do next?” Logic Reasoning includes experiment completion and solution-process generation, while Information-Based cases include chart interpretation and transformations such as changing a dot plot to a histogram.

The data pipeline relies on VLM-assisted drafting for captioning, prompt generation, and QA generation, followed by human audit. A stratified QA audit of approximately 300 QAs yields Cohen’s $\kappa = 0.78$ with a 7.8% rejection rate, after which rejected items are rewritten or removed. This suggests that the benchmark is designed as an auditable stress test rather than as a purely synthetic evaluation set.

3. Process-aware Reasoning Verification

The benchmark’s primary evaluation layer is Process-aware Reasoning Verification (PRV), which uses phase-linked QA with dynamic versus static diagnostics. PRV operates through a two-stage QA chain. In Stage 1, a VLM answers each video-grounded question strictly from visible evidence, with no guessing. In Stage 2, a separate LLM judge compares the predicted answer against the ground truth and returns a strict $0/1$ score (Wu et al., 11 May 2026).

Question types are mapped to four phases: state (factual), process (temporal), fidelity (detail), and mechanism (reasoning). Phase scores are mean binary accuracies: $S_{\text{state}}$, $S_{\text{proc}}$, $S_{\text{fidel}}$, and $S_{\text{mech}}$. Overall QA accuracy is the equal-weighted mean across the four phases.

PRV distinguishes static success from dynamic success through the following diagnostics:

• $$s_{\text{out}} = (S_{\text{state}} + S_{\text{fidel}})/2$$
• $a$0
• $a$1

A large positive ARG indicates outcome-hacking: the video looks correct in still frames but fails in temporal or causal reasoning. The headline PRV score is process-aware and multiplicative:

$a$2

where $a$3 is the equal-weighted mean over the four phases and $a$4. The exponent $a$5 is described as validated via human alignment. Additional diagnostics include the process-completeness ratio $a$6, and optional difficulty-weighted and bottleneck variants are provided for analysis.

The significance of PRV lies in its insistence that world reasoning is not reducible to endpoint correctness. By separating state, process, fidelity, and mechanism, the benchmark makes it possible to localize failures such as incorrect temporal ordering, broken causal chains, or loss of exact informational content.

4. Multi-dimensional Quality Assessment and WorldRewardBench

The second major evaluation layer is Multi-dimensional Quality Assessment (MDQA), a continuous per-video scoring protocol. MDQA rates each video on three axes, each on a 1–5 scale: Reasoning Quality, Temporal Consistency, and Visual Aesthetics. The automatic aggregate is

$a$7

which emphasizes reasoning while remaining comparable to human annotation (Wu et al., 11 May 2026).

Two MDQA protocols are defined. In the point-wise protocol, each video is scored independently to obtain $a$8, and pairwise preferences are induced with a $a$9 tie threshold. In the pair-wise protocol, two videos are judged jointly with a priority order of reasoning $V = G(x_0, a)$0 temporal $V = G(x_0, a)$1 aesthetics, yielding $V = G(x_0, a)$2, $V = G(x_0, a)$3, or tie. The point-wise form is intended for reward-model training and score-based ranking; the pair-wise form is intended for close comparisons and stronger ordinal supervision.

A companion benchmark, WorldRewardBench, extends the framework to preference calibration. It contains approximately 6,000 expert-annotated pairs over 1,432 unique videos, covering 11 generators with approximately 8 videos sampled per case. Fifteen trained annotators score each video on the same three axes and rank videos per case to construct pairwise preferences. Near-equal pairs are relabeled as ties, and left/right order is randomized to reduce bias. Aggregated human overall scoring is defined as

$V = G(x_0, a)$4

after disagreement-aware re-annotation.

Human and judge preferences are compared via a Bradley–Terry model with Davidson ties, and Elo mapping is reported as

$V = G(x_0, a)$5

The benchmark reports accuracy with and without ties, Spearman’s $V = G(x_0, a)$6 and Kendall’s $V = G(x_0, a)$7 versus Human Elo, and tie or close-pair diagnostics. A plausible implication is that WorldRewardBench is not merely an auxiliary dataset, but an explicit calibration layer for automatic judges and reward models.

5. Empirical findings, model behavior, and diagnostic value

WorldReasonBench evaluates five closed-source systems—Sora2 (8s/12s), Kling, Wan2.6, Seedance2.0, and Veo3.1-Fast—and six open-source systems—LTX2.3, Wan2.2-14B, UniVideo, Hunyuan Video-1.5, Cosmos-Predict2.5, and LongCat-Video. Automatic judging uses Qwen3.5-27B at 4 FPS by default, which is reported as the best cost–accuracy trade-off in ablation (Wu et al., 11 May 2026).

The main empirical result is a robust two-tier split. Closed-source models achieve overall $V = G(x_0, a)$8–$V = G(x_0, a)$9 and $\kappa = 0.78$0–$\kappa = 0.78$1, whereas open-source models achieve overall $\kappa = 0.78$2–$\kappa = 0.78$3 and $\kappa = 0.78$4–$\kappa = 0.78$5. Confidence intervals show no overlap between tiers on overall metrics. Logic Reasoning is the hardest dimension, with the best closed-source $\kappa = 0.78$6 only approximately $\kappa = 0.78$7 and most open-source models below $\kappa = 0.78$8. Information-Based tasks are the next major bottleneck, especially for world mechanics, material change, and exact text or data preservation.

Static-versus-dynamic attribution is central to the analysis. The process-completeness ratio $\kappa = 0.78$9 is $0/1$0–$0/1$1 for closed-source systems and $0/1$2–$0/1$3 for open-source systems, indicating that open-source deficits concentrate in dynamic reasoning rather than static appearance. ARG frequently flags outcome-hacking. This is consistent with the reported qualitative failures: domino chains that do not topple correctly, grabbing-machine videos that move the wrong object, and electromagnetic setups in which the wrong component moves.

Prompting assistance improves performance, but not enough to remove the underlying reasoning gap. Under hinted prompts, open-source systems gain $0/1$4 to $0/1$5 absolute QA points, corresponding to $0/1$6–$0/1$7, while the closed-source example Sora2-8s gains $0/1$8 points, or $0/1$9. The paper cautions that hint-gain should not be over-interpreted as latent world reasoning without corroborating process-aware diagnostics.

PRV also aligns strongly with human preferences. Against Human Elo from WorldRewardBench, $S_{\text{state}}$0 achieves Spearman $S_{\text{state}}$1, compared with $S_{\text{state}}$2 for ACCQA and $S_{\text{state}}$3 for a pairwise VLM judge. On WorldRewardBench, pair-wise judging reaches best agreement without ties at $S_{\text{state}}$4 for Qwen3.5-9B-Thinking, while point-wise scoring achieves best Spearman $S_{\text{state}}$5. Information-Based is reported as the hardest dimension to judge and also the most discriminative for future reward models.

6. Reproducibility, positioning, and relation to WR-Arena

WorldReasonBench releases its taxonomy, prompts, structured QA pairs with ground truth, expert preference pairs, and evaluation scripts under CC-BY 4.0, while not releasing model weights. The dataset structure is explicit: WorldReasonBench cases include per-case JSON with id, dimension, subcategory, image reference, implicit and hinted prompts, and 5–7 evaluation QA pairs with type, difficulty, criteria, and answers; WorldRewardBench contains video pairs from the same case with human per-video dimension scores, aggregated scores, tie labels, and pairwise preference labels. Reproducibility settings include default video sampling at 4 FPS, case-level bootstraps with $S_{\text{state}}$6 for 95% confidence intervals and rank stability, fixed seeds for preference post-processing, and dependencies including PyTorch, Transformers, and vLLM (Wu et al., 11 May 2026).

The recommended PRV workflow is procedural and auditable: generate videos under implicit and hinted prompts; run Stage 1 video QA with Qwen3.5-27B at 4 FPS; run Stage 2 strict binary judging; then aggregate to $S_{\text{state}}$7, $S_{\text{state}}$8, $S_{\text{state}}$9, $S_{\text{proc}}$0, ACCQA, $S_{\text{proc}}$1, $S_{\text{proc}}$2, ARG, and $S_{\text{proc}}$3. MDQA is likewise divided into point-wise and pair-wise protocols, and reward-model evaluation is anchored to agreement, rank correlation, and Human Elo comparisons.

Relative to prior video benchmarks such as VBench, VBench-2.0, EvalCrafter, and T2V-CompBench, the benchmark is positioned around open-domain world-state prediction with case-wise ground-truth QA that interrogates both outcomes and processes. Relative to narrower reasoning-oriented efforts such as WorldSimBench, V-ReasonBench, Gen-ViRe, VIPER, and VideoVerse, it combines open-domain coverage, multi-phase diagnostics, and calibrated preference data in a single protocol.

A useful contrast can be drawn with WR-Arena, or World Reasoning Arena, which evaluates world models as internal simulators along three dimensions: Action Simulation Fidelity, Long-horizon Forecast, and Simulative Reasoning and Planning (Team et al., 26 Mar 2026). WR-Arena centers on action-conditioned future-state simulation, multi-round forecasting, counterfactual rollouts, and planner-in-the-loop decision making, including metrics such as Transition Smoothness, Multi-round Smoothness, and penalized Generation Consistency. WorldReasonBench, by contrast, focuses on single-transition, initial-state-conditioned prediction with structured QA and human-aligned stress testing of video generators. This suggests that the two benchmarks are complementary rather than interchangeable: WorldReasonBench emphasizes auditable diagnosis of temporal, causal, logical, and informational consistency in TI2V generation, whereas WR-Arena emphasizes action-conditioned simulation competence over longer horizons and in planning loops.

The stated limitations clarify the current scope. WorldReasonBench does not yet cover counterfactuals, multi-agent social dynamics beyond two actors, physics with numerical ground truth, or long-horizon multi-event chains. Data construction and automatic judging depend on VLMs, and close-pair disagreements remain despite QA audits, cross-family judge comparisons, and calibration to Human Elo. The benchmark therefore functions less as a final measure of world reasoning than as a human-aligned stress test for identifying where visually persuasive systems still fail to model how worlds should evolve.