WBench: Multi-Turn Evaluation for Video World Models

• WBench is a unified evaluation benchmark for interactive video world models that combines multi-turn interactions with five assessment dimensions.
• It evaluates models on video quality, setting adherence, interaction adherence, consistency, and physics compliance using 289 test cases and 1,058 interaction turns.
• The toolkit integrates automated sub-metrics validated against human judgments to analyze rendering, control fidelity, and long-horizon state preservation.

WBench is a unified evaluation benchmark and toolkit for diagnosing open-domain interactive video world models. It was introduced to provide a standardized multi-turn, multi-dimension evaluation framework for systems that must jointly sustain high-fidelity rendering, correct world initialization, faithful execution of user controls, long-horizon state preservation, and physically plausible environment evolution. The benchmark evaluates models along five dimensions—video quality, setting adherence, interaction adherence, consistency, and physics compliance—over 289 test cases and 1,058 interaction turns, using 22 automatic sub-metrics that combine specialist vision models with large multimodal models and are validated against human judgments (Ying et al., 25 May 2026).

1. Motivation and problem setting

Interactive world models must combine five complementary capabilities identified in WBench as the “Renderer,” “Director,” “Controller,” “Memory,” and “Engine.” Prior benchmarks, as characterized in the benchmark description, either focus on single turns, single tasks such as navigation or driving, or non-interactive video quality or physics. WBench is designed as a response to that fragmentation: it is intended to span open-domain scenes, cover both first- and third-person views, support a full taxonomy of four interaction types, and evaluate multi-turn behavior across five distinct dimensions (Ying et al., 25 May 2026).

The benchmark’s design goal is therefore not merely to score perceptual video generation. It explicitly targets the interaction loop: a model is initialized with a world setting and then evaluated on whether it preserves that setting, executes subsequent user instructions, maintains temporal and geometric coherence over multiple turns, and respects causal and visual plausibility. This suggests an evaluation philosophy in which controllability and stateful behavior are treated as first-class properties rather than as secondary diagnostics.

2. Benchmark composition and task distribution

WBench contains 289 test cases and 1,058 interaction turns. Each case begins with an initial frame plus a textual world-setting prompt specifying scene, style, subject, and perspective, followed by a turn-by-turn interaction sequence. Turn depth ranges from 2 to 9 turns per case, with a mean of 3.7, so the benchmark stresses long-horizon consistency rather than single-step response quality alone (Ying et al., 25 May 2026).

The corpus is explicitly diverse across scene type, rendering style, perspective, subject identity, and interaction type.

Attribute	Distribution
Scenes	nature 31%, urban 21%, indoor 17%, workspace 13%, fantasy 10%, sports/game 8%
Styles	photorealistic 52% plus anime, cartoon, CG, oil painting, ink wash, pencil sketch, flat/abstract
Perspectives	first-person 62%, third-person 38%
Interaction types	navigation 57%, subject action 20%, event editing 17%, perspective switching 6%

Within first-person data, WBench further distinguishes “disembodied” views, where no avatar is visible, from “embodied” views with visible hands or tools. Subject statistics are reported for 194 third-person or embodied turns: human 64%, animal 9%, robot 9%, vehicle 7%, and other 10%. The result is a benchmark that mixes viewpoint control, semantic action, environmental intervention, and camera or subject reconfiguration within a single evaluation suite.

A plausible implication is that performance on WBench is shaped not only by model scale or rendering quality but also by the interaction semantics induced by perspective and subject type. That interpretation is consistent with later empirical findings on difficulty stratification.

3. Evaluation dimensions and metric system

WBench organizes evaluation into five high-level dimensions. All sub-metric scores are linearly rescaled to $[0,100]$, with higher values indicating better performance (Ying et al., 25 May 2026).

Dimension	Sub-metrics	Representative computation modules
Video Quality	6	CLIP, LAION aesthetic head, MUSIQ, RAFT, AMT-S, HPSv3
Setting Adherence	2	Doubao-Seed-2.0-lite
Interaction Adherence	4	MegaSaM, VLM question answering
Consistency	8	DreamSim, TransNetV2, SAM2, Depth Anything 3, DINOv2, CLIP
Physics Compliance	2	Doubao-Seed-2.0-lite, Qwen3-VL-30B-A3B

Video quality comprises six expert-model metrics. “Aesthetic Quality” embeds frames with CLIP and scores them with a LAION aesthetic head. “Imaging Quality” uses MUSIQ. “Temporal Flickering” is based on pixel MAE between consecutive frames. “Dynamic Degree” computes RAFT optical flow and thresholds the mean of the top 5% flow magnitudes. “Motion Smoothness” uses AMT-S interpolation error. “HPSv3-Norm” normalizes per-frame human-preference rewards by global percentiles.

Setting adherence asks whether generated content matches the world-setting prompt. Scene adherence decomposes the prompt into a visible part and an offscreen part, with a VLM scoring maintenance of visible elements and appearance of offscreen elements:

$S_{\rm scene} = \frac{m/5 + o}{2}\times100.$

Subject adherence, used for third-person and embodied first-person cases, similarly decomposes the subject into appearance and action priors.

Interaction adherence contains four sub-metrics. Navigation score estimates per-frame camera-to-world poses $\{T_k\in\mathrm{SE}(3)\}$ using MegaSaM, constructs a synthetic ground-truth trajectory from the discrete action sequence, rigidly aligns predicted and ground-truth trajectories, and computes normalized Absolute Trajectory Error for translation and rotation. It then combines accuracy and cross-turn consistency:

$$S_{\rm nav} = \frac{\mathrm{Acc}+\mathrm{Cons}}{2}\times100.$$

Event editing adherence and subject action adherence each use a five-question Yes/No VLM protocol over sampled frames, while perspective switching adherence uses three Yes/No checks—transition visibility, target-type consistency, and framing compliance—and scores a turn as successful only if all three are satisfied.

Consistency is divided into eight expert-model diagnostics. “Spatial Consistency” operates on round-trip navigation turns using DreamSim similarity between the first frame and the return frame, gated by the minimum similarity of intermediate frames:

$$S_{\rm spatial} = s_{\rm ret}\,\min\!\bigl(1,\,(1-s_{\min})/\tau\bigr)\times100,\quad \tau=0.15.$$

“Gated Spatial Consistency” applies the same gate even to large motions and is used as a diagnostic. “Segment Continuity” uses TransNetV2 to detect hard cuts. “Perspective Consistency” tracks subject masks with SAM2 and penalizes centroid variance. “Geometric Consistency” and “Photometric Consistency” rely on Depth Anything 3, pose estimation, reprojection, and either normalized displacement or PSNR. “Subject Consistency” uses DINOv2 and CLIP features from masked subject crops. “Background Consistency” averages CLIP feature cosine similarity across consecutive frames.

Physics compliance contains two metrics. “Causal Fidelity” uses Doubao-Seed-2.0-lite with a two-track rubric: Track 1 evaluates global rendering physics and causal consistency, while Track 2 scores a case-specific subset of fluid or smoke behavior, collision, surface tracks, deformation, wind forces, reflection, and human motion. “Visual Plausibility” fine-tunes Qwen3-VL-30B-A3B on 6,000 expert-rated videos with scores 1–5 and normalizes the predicted score to $[0,100]$.

4. Interaction taxonomy and interface unification

WBench defines four interaction types: navigation, subject action, event editing, and perspective switching (Ying et al., 25 May 2026).

Navigation is the most structurally elaborate. Under first-person control, W/S/A/D translate the camera forward, back, left, and right, while arrow keys tilt or turn the camera. Under third-person control, W/S/A/D move the subject, while arrow keys orbit the camera around that subject. WBench unifies text, 6-DoF pose, and discrete-action control so that models with different native input interfaces can be evaluated within the same task family. For text-driven models, an instruction such as “move forward” is converted into text; for camera-controlled models, the system supplies a $4\times4$ pose delta; for action-conditioned models, it sends a key code.

Subject action turns require the primary subject to perform an instructed action, including manipulation, tool use, locomotion, combat, or gestural behavior. Event editing turns change the environment through interventions such as weather shifts, time-of-day transitions, appearance changes, or mechanical, physical, and natural phenomena. Perspective switching turns move between first- and third-person views, including same-subject or multi-subject transitions and scope changes such as zoom.

This interface unification is central to the benchmark’s comparative logic. It permits text-driven, camera-controlled, and action-conditioned models to be compared on matched tasks despite heterogeneous native control schemes. A plausible implication is that WBench evaluates control semantics independently of interface modality as far as the translation layer allows.

5. Human validation and empirical findings

To validate automated scoring, WBench follows the VBench protocol with 400 trained annotators. For ten evaluation aspects—four interaction-adherence metrics, two setting metrics, aesthetic, spatial consistency, causal fidelity, and visual plausibility—annotators perform blind pairwise comparisons among models with answers of A wins, B wins, or Tie. These judgments are aggregated into per-model win rates, and Spearman’s $\rho$ is computed between human win rates and automated scores. All ten aspects achieve $\rho\ge0.94$, and four reach 1.00, indicating strong alignment between the automatic metric stack and human comparative judgments (Ying et al., 25 May 2026).

The empirical study evaluates 20 state-of-the-art text-driven, camera-controlled, and action-conditioned models under a dual-track protocol: all models are tested on a 158-case navigation split, and text-driven models are also evaluated on the full 289 cases. No single model performs strongly across all five dimensions. Video quality is reported as largely saturated across paradigms, with all averages near 80/100. By contrast, control-related and state-related dimensions remain differentiating.

Text-driven models lead in setting adherence, with Wan 2.7 and Kling 3.0 at approximately 91/100, but lag in navigation at approximately 67/100 compared with camera-controlled models at approximately 76 and action-conditioned models at approximately 78. Navigation is almost uncorrelated with video quality ($r\sim-0.12$), consistency ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$0), or physics ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$1). Camera control does not guarantee perspective consistency ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$2). Semantic interactions—event editing and subject action—correlate strongly with setting adherence ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$3) but weakly with navigation ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$4).

Several diagnostic conclusions follow directly from these results. First, consistency is multi-faceted rather than unitary: geometric consistency is high for camera models, above 93, while perspective consistency remains low, around 67. Second, high spatial-consistency scores can be misleading in near-static scenes, which is why gated spatial consistency is included as a diagnostic; the benchmark explicitly notes that some high “spatial” scores stem from near-static scenes rather than genuine revisits. Third, physical compliance correlates strongly with video quality ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$5) and consistency ($S_{\rm scene} = \frac{m/5 + o}{2}\times100.$6), indicating that broad generative priors help but do not eliminate control deficits.

Difficulty is structured by perspective, scene, and subject. Third-person cases are harder by $S_{\rm scene} = \frac{m/5 + o}{2}\times100.$7; sports/game scenes are the hardest scene type; and non-rigid motion is harder than more constrained subject behavior. Over multiple turns, navigation degrades the fastest, dropping 33 points from turn 1 to turn 4+, while subject action and event editing degrade more moderately, by 9 and 13 points respectively; perspective switching remains low and flat.

6. Implementation, availability, and research role

WBench is released with code, data, and an evaluation pipeline. The benchmark provides a GitHub repository at https://github.com/meituan-longcat/WBench and a HuggingFace dataset at https://huggingface.co/datasets/meituan-longcat/wbench. Installation consists of cloning the repository and installing dependencies with pip install -r requirements.txt, followed by downloading initial frames, prompts, and case metadata via the provided script (Ying et al., 25 May 2026).

Evaluation is organized around two scripts. Generation is invoked through: $S_{\rm scene} = \frac{m/5 + o}{2}\times100.$8 and scoring through: $S_{\rm scene} = \frac{m/5 + o}{2}\times100.$9 The configuration distinguishes the navigation split from the full split by argument. Sub-metric activation, including Track 2 physics selection, is pre-computed and supplied. Human–auto alignment scripts are available for recalibration. The implementation recommends a single GPU such as an A100 for local models, while API and web models are supported via browser automation for Genie 3 and Happy Oyster.

Within the broader evaluation landscape, WBench functions as a benchmark for open-domain interactive video world models rather than for single-turn video generation or narrowly defined control tasks. Its principal contribution is the coupling of a multi-turn test suite, interface-normalized interaction design, and a 22-metric diagnostic stack validated against human comparisons. This suggests that future progress in interactive world modeling will need to be tracked along separable axes—rendering, control fidelity, long-horizon memory, and physics—rather than by aggregate perceptual quality alone.