WorldOlympiad: Can Your World Model Survive a Triathlon?

Abstract: We introduce WorldOlympiad, a benchmark for diagnosing video-based world models across physical faithfulness, geometric consistency, and interaction fidelity. While existing benchmarks often focus on visual quality, semantic alignment, or short-term temporal coherence, they provide limited insight into whether generated videos obey physical rules, preserve coherent 3D structure, and sustain controllable interactions over long horizons. To address this gap, WorldOlympiad decomposes world-model evaluation into three complementary dimensions. The physical track uses object segmentation and MLLM-as-judge to assess whether generated videos follow interpretable rules in mechanics, thermal phenomena, and material properties. The geometry track reconstructs generated videos with Gaussian splatting and evaluates structural consistency, cross-view coherence, and camera-trajectory alignment. The interaction track assesses whether generated rollouts follow complex action prompts and maintain smooth, coherent transitions across consecutive video chunks. WorldOlympiad further covers three major downstream scenarios, including gaming, robotics, and general real-world videos, capturing diverse challenges from interactive control and embodied manipulation to open-domain motion and camera dynamics. Together, these tracks and scenarios form a scalable and interpretable evaluation suite that exposes failure modes beyond generic video quality. Experiments on state-of-the-art models reveal substantial gaps in physical reasoning, 3D consistency, and long-horizon interaction, underscoring the need for more structured evaluation protocols for generative world models.

• The paper introduces a 1,000-video benchmark that evaluates physical faithfulness, geometric consistency, and long-horizon interaction across gaming, robotics, and real-world domains.
• The paper finds that LingBot-World leads overall with a score of 0.683, while geometric consistency remains weak across all models, peaking at 0.424, and thermodynamic reasoning is especially unreliable.
• The paper’s automatic rankings closely match human preferences with Spearman correlation ρ = 0.95, while revealing recurring failures involving implausible dynamics, viewpoint-dependent geometry, and state drift across video chunks.

WorldOlympiad is a benchmark for evaluating video-based world models along three capability axes that existing evaluation suites largely leave unmeasured: physical faithfulness, geometric consistency, and interaction fidelity. The authors assemble 1,000 long videos spanning gaming, robotics, and real-world domains, evaluate eight publicly available long-video generation pipelines, and show that current models—despite strong visual quality—remain far from reliable world simulators. The most striking quantitative finding is that even the best model achieves an overall score of only 0.683, and geometric consistency tops out at 0.424 across all evaluated pipelines.

Motivation and positioning

The paper argues that prevailing benchmarks such as VBench, VBench++, and VBench 2.0 center on visual quality, aesthetics, motion smoothness, and semantic alignment over short clips, while task-oriented benchmarks such as MIND, WorldArena, EWMBench, and WorldEval each target a single downstream domain. Two gaps follow: physical plausibility, 3D structure preservation, and long-horizon interactive control are under-assessed; and metrics built around perceptual fidelity are fundamentally insensitive to violations of physical law or control-signal misalignment—failures invisible to those metrics. WorldOlympiad is designed to close both gaps within a single unified protocol covering gaming, robotics, and general real-world scenarios.

Benchmark construction

The test set comprises 400 robotics videos drawn from RoboCOIN (bimanual manipulation), 400 gaming videos sampled from GameGen-X and split into 60-second chunks, and 200 real-world videos from LVD-2M filtered by duration above 60 seconds and motion score above 50. Captions are produced by a three-stage chunk–caption–refine pipeline using Gemini-3-Pro-Preview: temporal chunking into at most six contiguous intervals, chunk-level captioning with a camera-movement action field mapped to WASD-style controls, and a full-video refinement pass that corrects hallucinations and standardizes terminology across adjacent chunks. The refinement stage matters because adjacent chunks share objects and scene context; inconsistent captions would corrupt interaction and long-context assessment.

Evaluation tracks

Physical track. An MLLM identifies moving or deforming entities, SAM3 produces object-centric masks and trajectories, and a two-step judging procedure applies: a relevance judge first determines whether each rule is applicable to the reference video, then a compliance judge compares generated against reference video on rules spanning mechanics (gravity, buoyancy, compression, impact), thermodynamics (melting, sublimation, vaporization, condensation, deposition, freezing), and material properties (color mixing, solubility, hardness, combustibility). Scores are averaged over applicable metrics per subset and across subsets.

Geometry track. Up to 32 frames are sampled, dynamic foreground objects are masked out, Depth Anything 3 reconstructs a Gaussian scene with camera parameters, and a Gaussian-Splat renderer produces a reconstruction video and a meta-view from the pose farthest from the reconstruction origin. A calibrated MLLM judge scores both artifacts in $[0,1]$, optionally replaced by LPIPS for the reconstruction video. Camera-trajectory agreement combines translation path-shape, motion extent, and camera-center error with rotation geodesic error, final-frame error, and rotation-extent agreement, aggregated adaptively according to the reference motion profile. The final geometry score averages the three subscores.

Interaction track. Under chunk-by-chunk rollout, a CLIP-based semantic-adherence score compares frame embeddings to chunk captions, calibrated with fixed thresholds $\tau_{\min}=0.20$, $\tau_{\max}=0.40$ so new models do not shift previously reported scores. A structured MLLM judge operates at three levels—chunk (visual quality, text alignment), adjacent-pair transitions (smoothness, identity continuity), and global (long-range consistency, global text alignment)—and the final interaction score blends the MLLM average with the calibrated CLIP term at weight $\lambda=0.1$. The overall score equally weights the three tracks.

Main results

Eight pipelines are evaluated through OpenWorldLib: gaming-centric Matrix-Game 2.0 and LingBot-World; robotics-centric Cosmos-Predict-2.5 and WoW; and general models Rolling Forcing, LongLive, Yume-1.5, and Hunyuan-WorldPlay.

Model	Physical	3D Cons.	Interact.	All	Rank
LingBot-World	0.942	0.373	0.734	0.683	1
Cosmos-Predict-2.5	0.906	0.399	0.707	0.671	2
Rolling Forcing	0.873	0.321	0.636	0.610	3
Yume-1.5	0.863	0.301	0.649	0.604	4
LongLive	0.863	0.363	0.526	0.584	5
Hunyuan-WorldPlay	0.692	0.424	0.316	0.477	6
WoW	0.708	0.250	0.345	0.434	7
Matrix-Game 2.0	0.325	0.255	0.113	0.231	8

Several findings stand out. First, LingBot-World—a 14B-activated model—leads on physical faithfulness (0.942) and interaction fidelity (0.734), but Cosmos-Predict-2.5 reaches a comparable overall score with only 2B parameters, indicating that targeted physical-world training can partly compensate for smaller scale. Second, physical regularity is emerging as a shared capability: five of eight models exceed 0.85 on the physical track, yet fine-grained breakdowns reveal thermodynamics as a systematic weakness—for example, vaporization pass rates are near zero across most models in the robotics domain, and several models score 0.000 on thermodynamics there entirely. Third, the geometry gap is unresolved: the best 3D consistency score is 0.424 (Hunyuan-WorldPlay), with most models between 0.25 and 0.40. The paper notes that Hunyuan-WorldPlay's reliance on camera/viewpoint control encourages spatial-layout preservation but constitutes a more constrained interaction mode than open-ended action conditioning, explaining its geometry strength alongside weak overall performance. Fourth, specialization does not uniformly generalize: WoW scores 0.502 on embodied videos but only 0.368 on gaming and 0.415 on general videos, whereas LingBot-World and Cosmos-Predict-2.5 transfer their domain training broadly.

Qualitative case studies identify three recurring failure modes: implausible dynamics (objects moving against gravity, deforming without contact, abrupt state changes); videos that look plausible in the original view but fail under 3D reconstruction, meta-view rendering, or trajectory comparison; and rollouts that follow isolated captions but reset state, lose objects, or drift across chunk boundaries.

Human preference alignment

A controlled study with five annotators compared all 28 unordered model pairs across 20 prompts (2,800 individual labels) using four criteria: visual quality, physical plausibility, temporal coherence, and interaction fidelity. The human ranking agrees with the automatic ranking at Spearman $\rho = 0.95$, with rank disagreements confined to two adjacent swaps (LongLive/Yume-1.5 and Matrix-Game 2.0/WoW). This supports the claim that the automatic evaluator captures model-level quality differences perceived by humans while remaining scalable and diagnostic.

Limitations and open questions

The paper concedes several constraints. Judge-based scoring depends on Gemini-3-Pro-Preview as the central evaluator, and although calibration rubrics and clamping are specified, judge reliability beyond the reported human-alignment study is not independently validated. The CLIP auxiliary thresholds are fixed heuristics rather than learned calibrations. Thermodynamics and material questions are frequently marked irrelevant in robotics videos, leaving some rule categories thinly covered in certain domains. The equal-weight aggregation of the three tracks is a design choice, not empirically derived. Most significantly, the authors acknowledge that many recent pipelines introduce memory modules whose gains cannot be disentangled from confounds in scale, data, and architecture; they propose, but do not deliver, a controlled environment comparing KV-cache reuse, explicit 3D scene memory, linear attention, and hybrid mechanisms under shared data and capacity. Whether memory design specifically drives long-horizon consistency therefore remains open.

Conclusion

WorldOlympiad provides a unified, interpretable protocol for diagnosing video world models across physical law adherence, 3D geometric consistency, and interactive long-horizon control, backed by a 1,000-video multi-domain test set and strong alignment with human preference ($\rho = 0.95$). Its evaluation of eight pipelines demonstrates that perceptually convincing generation does not imply reliable world simulation: thermodynamic reasoning, cross-view geometry, and cross-chunk state preservation remain substantial weaknesses even among top-ranked models.