Omni-WorldBench: 4D Interaction Benchmark

• Omni-WorldBench is an interaction-centric benchmark for evaluating video-based world models that capture action-driven temporal evolution in 4D settings.
• It consists of Omni-WorldSuite, a diverse prompt suite, and Omni-Metrics, an agent-based evaluation framework assessing video quality, controllability, and interaction effect fidelity.
• The benchmark emphasizes dynamic causal response by measuring intermediate state transitions and differentiating between final visual outcomes and temporal evolution.

Omni-WorldBench is an interaction-centric benchmark for video-based world models in 4D settings, introduced to evaluate interactive response: the ability to faithfully reflect how interaction actions drive state transitions across space and time. It is organized around two components—Omni-WorldSuite, a prompt suite spanning diverse interaction levels and scene types, and Omni-Metrics, an agent-based evaluation framework that scores generated videos in terms of generated video quality, camera-object controllability, and interaction effect fidelity. The benchmark is motivated by the claim that conventional video-generation evaluation emphasizes visual fidelity and text-video alignment, while static 3D reconstruction metrics neglect temporal dynamics; Omni-WorldBench instead treats causal, interaction-driven temporal evolution as the central capability of a world model (Wu et al., 23 Mar 2026).

1. Motivation and conceptual framing

Omni-WorldBench is positioned within two dominant paradigms of video-based world modeling: video generation-based world models and 3D reconstruction / geometry-aware world generation approaches. The benchmark argues that both prevailing evaluation styles are incomplete for world modeling. Metrics such as FID, IS, FVD, and suites such as VBench primarily assess visual fidelity, motion smoothness, text-video alignment, and perceptual quality. Benchmarks such as WorldScore move closer to world-model-specific evaluation through viewpoint changes and geometric consistency, but remain centered on static 3D scene coherence or camera-motion-conditioned rendering. In the benchmark’s framing, neither paradigm systematically measures whether a model produces the correct evolution of a world under interaction (Wu et al., 23 Mar 2026).

This formulation is tied to a broader claim about 4D generation. The benchmark treats the future of world modeling as joint modeling of spatial structure and temporal evolution, and therefore identifies interactive response as the core capability. In that usage, interactive response refers to whether a model faithfully reflects how actions drive state transitions across space and time, including correct effects on acted-upon objects, plausible propagation to other objects or the environment, and temporally coherent execution of those effects. The intended relevance is downstream: counterfactual reasoning, planning, decision making, autonomous driving, embodied robotics, and gaming all require more than visually plausible video; they require correct action-conditioned dynamics (Wu et al., 23 Mar 2026).

A central distinction in Omni-WorldBench is therefore between final appearance and temporal evolution. The benchmark evaluates not only final outcomes, but also intermediate state changes, event ordering, revisit consistency, and the separation between affected and unaffected entities. This emphasis makes it explicitly interaction-centric rather than purely appearance-centric or geometry-centric (Wu et al., 23 Mar 2026).

2. Omni-WorldSuite

Omni-WorldSuite is the prompt suite of Omni-WorldBench. Each benchmark instance consists of a textual prompt describing interaction-driven world-state evolution, an initial frame image specifying the starting state, and, for some prompts, an explicit camera trajectory. This design supports text-to-video, image-to-video, and camera-conditioned generation settings within a single suite (Wu et al., 23 Mar 2026).

The suite is built through two complementary pipelines. The dataset-grounded pipeline extracts the first video frame and camera motion trajectory from open-source datasets, uses Qwen-VL to generate an initial caption, manually verifies and refines the caption to correct spatial relations and object attributes, and then constructs the final prompt from the verified caption, the initial frame, and the optional original camera trajectory. The datasets named in this process are DriveLM for autonomous driving, InternData-A1 for embodied robotics and manipulation, and Sekai for gaming and simulation. The concept-driven pipeline samples prototype concepts spanning scene domains, target objects, actions, and interaction levels; uses ChatGPT-5.2 to generate a textual prompt and camera trajectory; cross-checks with Gemini and DeepSeek-R1; performs human verification and refinement; and generates first-frame images using FLUX.1-dev with 3 candidates per prompt, CFG scale 3.5, and 50 sampling steps. If needed, prompts are rewritten with ChatGPT-5.2 and refined with Qwen-Image, with only minor local in-painting permitted. The quality-control requirements include minimum resolution $1024 \times 1024$, consistency with the prompt, and clear visibility of target interactive objects (Wu et al., 23 Mar 2026).

A key organizing device is a three-level interaction hierarchy. Level 1 contains cases in which effects are confined to the acting object and do not alter other objects or the environment; examples include physical optics such as viewing fields through a crystal ball, and continuous spatial navigation such as moving along a riverside path. Level 2 contains direct one-object-to-one-object effects; examples include heating a metal rod in a campfire and ego-vehicle navigation alongside dynamic traffic. Level 3 contains interactions that influence multiple objects and produce broader environmental changes; examples include snapping spaghetti, tidying a room, and a robotic arm grasping a bottle and handing it to a person. This hierarchy operationalizes increasing dynamic and causal scope from local change to multi-object state transition (Wu et al., 23 Mar 2026).

Omni-WorldSuite also spans a broader taxonomy. The benchmark covers general daily-life scenes and task-oriented environments, including indoor scenes, outdoor scenes, autonomous driving, embodied robotics, and gaming. Its six major annotation dimensions are Physics Principles (PP), Commonsense (CS), Causality (Cau), Camera Motion (CM), Loop-Closure Consistency (LCC), and Spatial Constraints (SC). Within these, the paper lists subcategories such as Newtonian Mechanics, Fluid Mechanics, Material Properties, Waves and Optics, Momentum and Collision, Thermodynamics and Phase Transition, and Energy Conversion and Conservation under Physics Principles; Scene/Event Knowledge, Object Function Knowledge, and Human Action Knowledge under Commonsense; and Condition-to-Behavior, Action-to-Motion, and Collision-to-Outcome under Causality. The suite additionally annotates all objects in the prompt, splits them into affected and unaffected entities, records expected coarse motion direction and magnitude for affected entities, extracts temporally ordered key events, annotates expected camera motions for some prompts, and marks return-to-origin subsets that yield revisit frames (Wu et al., 23 Mar 2026).

The scale reported for Omni-WorldSuite is 1,068 evaluation prompts. Physics Principles is described as the most frequent major dimension; Causality and Commonsense are also common. By interaction level, Level 2 has the largest number of prompts, followed by Level 3, then Level 1. This distribution suggests an emphasis on direct but nontrivial interaction over both purely local effects and the most complex multi-object cascades (Wu et al., 23 Mar 2026).

3. Omni-Metrics

Omni-Metrics is the evaluation framework of Omni-WorldBench. It first extracts structured information from each generated video and then scores the result along three major dimensions: Generated Video Quality, Camera-Object Controllability, and Interaction Effect Fidelity. The paper states that the protocol encompasses 15 metrics, although the excerpt explicitly details the named metrics rather than enumerating all 15 individually (Wu et al., 23 Mar 2026).

The generated video is represented as

$V \in \mathbb{R}^{T \times H \times W},$

where $T$, $H$, and $W$ denote frame count, height, and width. To construct evaluation signals, the framework uses GroundingDINO and SAM to extract temporally consistent segmentation mask sequences for entities, RAFT to estimate optical flow, and relative camera motion approximations derived from optical-flow variations. These intermediate representations support both object-centric and motion-centric analysis (Wu et al., 23 Mar 2026).

The Generated Video Quality branch reuses existing metrics: Imaging Quality, Temporal Flickering, Motion Smoothness, and Dynamic Degree from VBench, plus Content Alignment from WorldScore. These quantify perceptual and semantic properties of the generated video, but the benchmark explicitly treats them as necessary rather than sufficient for world-model evaluation (Wu et al., 23 Mar 2026).

The Camera-Object Controllability branch comprises Camera Control, Object Control, and Transitions Detect. Camera Control adopts WorldScore’s camera-control formulation over rotational and translational discrepancies. Object Control is formulated as VQA over the prompt-specified object list $\mathcal{O}=\{o_i\}_{i=1}^K$, yielding

$\frac{1}{K}\sum_{i=1}^K \hat{y}_i,$

where $\hat{y}_i \in \{0,1\}$ indicates whether object $o_i$ is judged present. Transitions Detect uses PySceneDetect’s ContentDetector and produces a binary score that is 1 if the video has a single detected scene and 0 otherwise. This branch therefore measures whether a model obeys explicit camera instructions, preserves prompt-specified objects, and avoids scene-cut discontinuities (Wu et al., 23 Mar 2026).

The Interaction Effect Fidelity branch is the benchmark’s central innovation. It contains InterStab-L, InterStab-N, InterCov, and InterOrder. InterStab-L evaluates long-horizon temporal coherence at revisit points where the camera returns to the same or corresponding viewpoint. InterStab-N measures stability of non-target regions, penalizing gratuitous motion in regions that should remain unaffected. InterCov measures object-level causal faithfulness by checking whether affected entities respond appropriately and unaffected entities remain stable. InterOrder evaluates whether the temporal ordering of events matches the annotated event sequence, with the reported formula

$\mathrm{InterOrder} = \frac{2K_s}{K(K-1)},$

where $V \in \mathbb{R}^{T \times H \times W},$0 is the number of temporally consistent event pairs. This branch is summarized as

$V \in \mathbb{R}^{T \times H \times W},$1

The full benchmark score is then

$V \in \mathbb{R}^{T \times H \times W},$2

where $V \in \mathbb{R}^{T \times H \times W},$3 and $V \in \mathbb{R}^{T \times H \times W},$4 are the generated-video-quality and camera-object-controllability agents, respectively. The weights are assigned by an aggregation agent implemented with an MLLM conditioned on prompt semantics, so AgenticScore is prompt-adaptive rather than a fixed uniform average (Wu et al., 23 Mar 2026).

4. Experimental protocol and empirical findings

Omni-WorldBench evaluates 18 representative world models across three paradigms. The text-to-video (T2V) group contains Director3D, OpenSoraPlan, T2V-Turbo, and HunyuanVideo. The image-to-video (IT2V) group contains Matrix Game2.0, Wan2.1, Wan2.2, CogVideo, OpenSora, Cosmos, and LargeVideoPlanner. The camera-conditioned group contains HunyuanWorld, HunyuanGameCraft, ViewCrafter, Gen3C, Lingbot, FantasyWorld, and WonderWorld. The paper states that these models span diffusion-based, autoregressive, and hybrid paradigms (Wu et al., 23 Mar 2026).

The evaluation uses 410 prompts for T2V and IT2V models and 120 prompts with explicit camera trajectories for camera-conditioned models. Experiments run on NVIDIA H20 GPUs, and software environments follow each model’s official recommendations. This protocol deliberately preserves model-specific inference settings rather than imposing a single uniform generation configuration across all systems (Wu et al., 23 Mar 2026).

The headline empirical result is that IT2V models perform best overall. The top AgenticScore is Wan2.2: 75.92, followed by Cosmos: 75.42. The best T2V model is HunyuanVideo: 73.96. Among camera-conditioned models, HunyuanWorld: 74.36 leads, followed closely by WonderWorld: 74.02. The benchmark interprets this as evidence that image conditioning provides stronger grounding for interaction-centric video generation (Wu et al., 23 Mar 2026).

On Interaction Effect Fidelity, the strongest averages are Wan2.2: 67.34, Cosmos: 66.22, CogVideo: 65.58, HunyuanVideo: 64.88, and LargeVideoPlanner: 64.39. On Generated Video Quality, the strongest averages include WonderWorld: 85.18, OpenSora: 84.13, Gen3C: 83.07, Matrix Game2.0: 82.77, and Wan2.1: 82.64. On controllability-related scores, the paper highlights Cosmos: 94.90 and Wan2.2: 94.01 in the IT2V block, while WonderWorld achieves Camera Control: 96.12 and HunyuanWorld the best average controllability in the camera-conditioned group at 79.67 (Wu et al., 23 Mar 2026).

The benchmark’s main diagnostic claim is that high visual quality does not imply high interaction fidelity. Models frequently score strongly on temporal flickering and motion smoothness while lagging on InterCov, InterOrder, and InterStab-N. The excerpt gives illustrative low points: ViewCrafter has InterStab-N = 4.22, and Matrix Game2.0 has an interaction-fidelity average of 42.76. Qualitative examples reinforce the quantitative picture: for a prompt involving a baseball player performing a powerful throw, Wan2.2 produces a complete and anatomically reasonable pitching motion with strong temporal consistency, whereas Matrix-Game2.0 produces an incomplete action with severe temporal degradation and final collapse of the human figure. In a camera-controlled left-strafe example, HunyuanWorld remains relatively stable, while ViewCrafter introduces a spurious building and lowers visual consistency. These findings are presented as evidence that current world models remain limited in causally grounded, action-conditioned scene evolution even when appearance quality is high (Wu et al., 23 Mar 2026).

5. Position within the benchmark landscape

Omni-WorldBench is explicitly framed against VBench and WorldScore. VBench contributes perceptual video-quality metrics, and WorldScore contributes camera-control and content-alignment components, but Omni-WorldBench reorients evaluation toward interaction effect fidelity, intermediate state evolution, and affected-versus-unaffected entity dynamics (Wu et al., 23 Mar 2026).

In a broader benchmark ecology, Omni-WorldBench occupies a distinct position. OmniGenBench is a benchmark for instruction-following image generation across 57 diverse sub-tasks, spanning perception-centric and cognition-centric generation, but it remains image-generation-centric rather than a benchmark of interactive 4D world modeling (Wang et al., 24 May 2025). WorldBench is a 2,000-question, four-option multiple-choice multimodal reasoning benchmark built to prioritize visual diversity over task diversity for MLLMs, and therefore evaluates open-ended visual reasoning rather than generated state transitions under interaction (Yin et al., 4 Jun 2026). OmniWorld introduces a multi-domain, multi-modal dataset and benchmark for 3D geometric prediction and camera-control video generation, emphasizing 4D reconstruction and controllable generation rather than a unified interaction-effect fidelity framework (Zhou et al., 15 Sep 2025). A separate WorldBench proposes a disentangled diagnostic benchmark for intuitive physics and physical-parameter estimation in video continuation, isolating concepts such as object permanence, support relations, gravity, viscosity, and friction (Upadhyay et al., 29 Jan 2026). WorldRoamBench targets long-horizon stability of interactive world models across action, vision, physics, and memory in open-world roaming with discrete keyboard control (Xu et al., 30 Jun 2026).

This suggests that Omni-WorldBench is neither a general MLLM QA benchmark nor a purely physics-diagnostic suite nor a long-horizon closed-loop roaming benchmark. Instead, it functions as a comprehensive interaction-centric evaluation for video-based world models in 4D settings, centered on prompt-conditioned and optionally camera-conditioned world evolution under interaction (Wu et al., 23 Mar 2026).

6. Limitations and significance

The paper attributes several strengths to Omni-WorldBench. It presents the benchmark as the first benchmark dedicated to evaluating interactive response capabilities of world models; it spans general scenes and task-oriented domains; it organizes prompts through a hierarchical interaction design; it evaluates intermediate state transitions rather than only endpoints; and it aggregates multiple dimensions through a prompt-adaptive AgenticScore (Wu et al., 23 Mar 2026).

At the same time, the benchmark states several limitations. It does not fully capture the complexity of open-world interactive environments; its coverage of very long or highly dynamic interactions remains incomplete; and the authors plan additional human-aligned evaluation results beyond the claim that Omni-Metric aligns well with human preferences. The excerpt does not specify a public URL, train/test split, per-domain prompt counts beyond qualitative frequency statements, or benchmark-server details. It also relies operationally on an evaluation stack that includes GroundingDINO, SAM, RAFT, VLM or MLLM-based evaluators, and PySceneDetect, so practical reproduction depends on those components as well as on the released implementation (Wu et al., 23 Mar 2026).

The broader significance of Omni-WorldBench lies in its redefinition of what a world-model benchmark should measure. Rather than equating world modeling with visually plausible video or static geometric consistency, it treats the defining property of a world model as causally grounded, temporally coherent, interaction-aware 4D generation. In that sense, Omni-WorldBench is less a replacement for existing video, geometry, or multimodal reasoning benchmarks than a benchmark that inserts a missing axis into that landscape: whether a generated world actually responds correctly when something happens within it (Wu et al., 23 Mar 2026).