WoW-World-Eval Protocol

• WoW-World-Eval is a comprehensive protocol assessing video foundation models’ ability to generate perceptually accurate, physically plausible, and actionable video rollouts.
• It evaluates models across five core competencies—perception, planning, prediction, execution, and generalization—using 22 metrics and human preference judgments.
• The benchmark bridges simulated video generation with real-world robotics by emphasizing long-horizon planning and physical consistency.

WoW-World-Eval (Wow, wo, val) is a comprehensive evaluation protocol and benchmark suite designed to measure the capabilities of video foundation models as embodied world models in robotics. Developed as an “Embodied Turing Test,” WoW-World-Eval provides a standardized framework for assessing whether video foundation models possess the generative fidelity, physical grounding, and robustness required for real-world embodied AI tasks. Specifically, it targets two pivotal questions: (1) Can these models generate future video rollouts that maintain perceptual fidelity to the satisfaction of human observers? (2) Are they sufficiently robust and physically plausible to serve as priors for real-world robotic agents? WoW-World-Eval operationalizes evaluation across five core competencies—perception, planning, prediction, generalization, and execution—using a multi-dimensional metric suite, extensive human preference judgments, and real-robot action replay, thereby establishing a rigorous protocol for benchmarking the current frontier and limitations of generative world models in embodied AI (Fan et al., 7 Jan 2026).

1. Motivations and Evaluation Philosophy

WoW-World-Eval was motivated by the increasing reliance on video foundation models as world models for downstream embodied tasks, including 3D prediction and interactive video generation. Unlike preceding video evaluation benchmarks, which primarily focused on low-level pixel fidelity or isolated perceptual dimensions, WoW-World-Eval adopts a robotics-centric approach: it aligns its evaluation metrics with the functional requirements of embodied agents. This alignment mandates assessment not only of visual plausibility but also of coherent planning, physical simulation, execution feasibility, and generalization outside of the training distribution. WoW-World-Eval thus bridges the gap between generative video modeling and the actionable needs of autonomous robotic control, facilitating pre-deployment benchmarking for real-robotic systems (Fan et al., 7 Jan 2026).

2. Core Competency Dimensions

WoW-World-Eval is structured around five orthogonal dimensions representing the essential capabilities of an embodied world model:

1. Perception Understanding: Assesses a model’s ability to infer scene semantics, including recognition of object attributes (e.g., color, shape, size, count), spatial relationships (e.g., “cup is left of plate”), and affordances (e.g., graspable handles), given a single initial frame and natural language instruction. Accurate scene representation is foundational for subsequent planning and action selection in robots.
2. Decision-Making and Planning: Requires models to generate multi-step video rollouts for long-horizon, causally structured instructions (e.g., “pick up the block, place it in the drawer, then close the drawer”). This dimension probes temporal coherence, subgoal decomposition, and the correct sequencing of atomic actions.
3. Predictive Reasoning (Physical Simulation): Evaluates whether models simulate environment evolution under specified actions, exhibiting object permanence, collision dynamics, and realistic trajectories. The world model must function as a physics simulator, approximating $s_{t+1}\mid s_t, a_t$.
4. Interactive Execution: Measures whether generated video rollouts are compatible with execution on physical robots using an Inverse Dynamics Model (IDM), which infers actionable motion commands from the visual output of the world model.
5. Generative Generalization: Tests model robustness and compositional generalization by introducing out-of-distribution inputs, such as in-house style-transferred or artistic initial frames, reflecting the requirement for reliable deployment in visually novel scenarios (Fan et al., 7 Jan 2026).

3. Evaluation Metrics and Scoring Protocol

WoW-World-Eval leverages a suite of 22 metrics, mapping raw measurements $x_{i,m}$ into bounded scores $s_{i,m} \in (0, 100)$ by pre-scaling to $[0,1]$, applying a monotone parametric transform $f_m(\cdot;\theta_m)$, and rescaling to $[0,100]$. Scores are aggregated by weighted arithmetic mean within five metric groups: Video Quality (VQ), Instruction Understanding (IU), Planning Reasoning (PL), Physical Law (PR), and Execution Accuracy (EA). In the standard protocol, group weights are uniform.

The following table summarizes the five groups and selected metrics:

Group	Key Metrics (examples)	Measure Purpose
Video Quality (VQ)	FVD, PSNR, SSIM, DINO, DreamSim	Distributional realism, pixel/structural/semantic fidelity, human-aligned perceptual similarity
Instruction Understanding (IU)	Caption Score, SeqMatch, Execution Quality	Semantic and procedural adherence to instructions
Planning Reasoning (PL)	Long-Horizon DAG Score	Coherence and completeness of complex action plans
Physical Law (PR)	MRC, Trajectory L2Norm/DTW/FD, ATE/RPE, Physical Common-Sense	Physical consistency: region/trajectory similarity, physical common-sense
Execution Accuracy (EA)	Real-World Success Rate	Executability of inferred actions on real robots

The protocol encompasses detailed instantiations for each metric, such as Fréchet Video Distance (FVD) using I3D features, DreamSim incorporating human-finetuned multimodal encoders, and Mask-Guided Regional Consistency (MRC) for spatial-temporal stability in robot, object, and background regions. Instruction adherence is measured using automated VLM and LLM scoring of decomposed video captions, and long-horizon planning leverages DAG-based action graph comparisons. Physical Law metrics combine region, trajectory, and camera motion calculations, along with AI-judged common-sense physicality using a Qwen-2.5-VL MLLM fine-tuned with GRPO on human ratings (Fan et al., 7 Jan 2026).

4. Dataset and Experimental Setup

WoW-World-Eval draws on a curated collection of 609 robot-manipulation video samples, sourced from RoboMIND, DROID, in-house recordings, and GPT-5-generated out-of-distribution sequences. Samples are distributed across Prediction (50.6%), Perception (40.9%), Planning (25 tasks), Execution (9 tasks), and Generalization domains. Automated assignment to core competency areas is performed using GPT-4o, with subsequent human verification and keypoint annotation for trajectory-based metrics.

The primary generation protocol requires models to produce a 5-second video conditioned on an initial image and text instruction. Evaluation proceeds under a two-alternative forced choice (2AFC) Human Turing Test, where 15 domain experts judge real vs. generated videos across the four core dimensions, yielding the Deceive-Human Ratio—proportion of synthetic videos judged as real. This human evaluation anchors the correlation analyses for overall model performance (Fan et al., 7 Jan 2026).

5. Human and Machine Correlation: Turing Tests

WoW-World-Eval validates metric alignment with human perception by correlating the aggregated overall score $O_i$ for model $i$ (across all groups) with the human Deceive-Human Ratios $H_i$. The reported Pearson correlation $r > 0.93$ (Spearman $\rho = 0.91$) signifies very strong agreement between metric-based evaluation and expert human judgment, confirming the validity of the multi-metric aggregation as a surrogate for direct Turing-like human discrimination (Fan et al., 7 Jan 2026).

To test actionability, the Inverse Dynamics Model (IDM) Turing Test is implemented: a Gripper-Centric IDM is trained on paired real videos and action sequences. Generated videos are input to the IDM, which infers action commands that are replayed on a physical robot, with success/failure recorded. Reported results indicate that most models exhibit near-zero replay success (Kling: 9.88%, Hailuo: 2.47%, CogVideoX, Cosmos-Predict1, Wan2.1: 0%), whereas video world models trained with real-robot data (WoW-wan: 40.74%, WoW-cosmos2: 18.52%) substantially outperform generic video models (Fan et al., 7 Jan 2026).

6. Empirical Findings and State of the Art

Comprehensive benchmarking on WoW-World-Eval reveals several trends and limitations in generative world models for embodied AI:

• Long-Horizon Planning Remains Difficult: Top-performing models achieve only modest scores (Hailuo: 17.27, Cosmos-Predict2: 13.41, WoW-cosmos2: 12.27 out of 100), reflecting challenges in generating temporally coherent, causally plausible action sequences.
• Physical Consistency Is Incomplete: The closed-source upper bound for Physical Law is ~68.0 (Kling); open-sourced models reach ~66.2 (WoW-cosmos2). These results, while high, reveal persistent artifacts such as jitter, trajectory drift, and mild physical law violations.
• Visual Realism Does Not Imply Executability: Success in perceptual and physical metrics fails to guarantee real-world action success—most generic models collapse at the execution stage, highlighting a significant gap between imagined and actionable environments.
• Human–Metric Concordance: The strong correlation between automated metrics and human preference supports their use for large-scale benchmarking.

These findings demonstrate the structural limitations of current video foundation models as world models, especially in closing the gap between simulated and executable robotic behaviors (Fan et al., 7 Jan 2026).

7. Implications for Embodied AI and Future Directions

WoW-World-Eval underscores the urgent need for embodied world models to move beyond pixel-space generation towards representations that support explicit planning, physical reasoning, and robustness to domain shift. The protocol’s findings highlight the deficiency of visual realism as a proxy for actionability, urging the integration of physics inductive biases, structured task decomposition, and real-world fine-tuning. A plausible implication is that future progress in embodied AI requires tight coupling of generative video modeling with multi-modal, real-robot data and explicit grounding in physical causality, as validated by both human and autonomous execution tests (Fan et al., 7 Jan 2026). WoW-World-Eval establishes a rigorous, reproducible protocol and benchmark suite intended to catalyze these advances by enabling systematic, multi-faceted evaluation of world models prior to their deployment in embodied interactive settings.