WorldMark: Benchmark for Interactive Video Models

• WorldMark is a unified benchmark suite for interactive video world models that standardizes scenes, action sequences, and control interfaces.
• It employs a unified action-mapping layer and hierarchical test suites to translate shared commands into model-specific controls and assess performance across varying difficulty tiers.
• Its evaluation workflow integrates automated metrics—covering visual quality, control alignment, and world consistency—with human preference studies for reproducible, extensible model comparisons.

WorldMark is a unified benchmark suite for interactive video world models, introduced to make fair cross-model comparison possible in a setting where models such as Genie, YUME, HY-World, and Matrix-Game had previously been evaluated on private scenes, private trajectories, and heterogeneous control interfaces. It defines a common playing field for interactive Image-to-Video world models by standardizing the reference images shown to the model, the action sequences the model must follow, the translation from a shared action language to each model’s native control format, and the evaluation workflow itself (Xu et al., 23 Apr 2026).

1. Purpose and problem formulation

WorldMark addresses a basic evaluation failure mode in interactive world modeling: reported metrics are not directly comparable when different systems are tested on different scenes, different trajectories, and different control semantics. Existing public benchmarks provide useful signals such as trajectory error, aesthetic scores, and VLM-based judgments, but they do not provide the standardized test conditions—identical scenes, identical action sequences, and a unified control interface—required for apples-to-apples comparison across models with heterogeneous inputs (Xu et al., 23 Apr 2026).

The benchmark is positioned as the first benchmark to provide such a common playing field for interactive Image-to-Video world models. In the paper’s framing, the problem is not merely metric selection; it is the absence of a standardized task definition. A model asked to follow one trajectory in one scene under one action vocabulary cannot be fairly compared with a model evaluated under a different scene distribution or a different control regime. WorldMark therefore treats standardization of inputs and controls as a prerequisite for meaningful evaluation.

A common misconception is that a shared metric suite alone is sufficient for fair benchmarking. WorldMark rejects that premise by insisting that the underlying control problem must also be identical across systems. This includes the scene, the trajectory, and the semantic meaning of actions, not merely the scoring function.

2. Unified control interface and action mapping

The benchmark’s central systems contribution is a unified action-mapping layer built around a shared vocabulary of six discrete actions: W for forward, S for backward, A for strafe left, D for strafe right, L for yaw left, and R for yaw right. These actions are parameterized by duration and then translated into each model’s native control format through per-model adapters (Xu et al., 23 Apr 2026).

This action layer is necessary because the evaluated models expose markedly different interfaces. The paper lists the following mapping strategies:

• YUME 1.5: caption prompts with directional keywords
• HY-World 1.5: 6-DoF pose parameters via latent timescale matching
• HY-GameCraft: pose-to-Plücker ray embeddings
• Genie 3: gamepad button presses
• Matrix-Game: corresponding action API calls
• Open-Oasis: set movement dimensions in 25-D action vectors

The significance of this design lies in semantic normalization. “Move forward” or “turn right” is not directly comparable when one model expects caption tokens, another expects pose parameters, and another expects continuous vectors. WorldMark’s adapter layer is intended to ensure that all systems receive semantically identical instructions, rather than loosely analogous model-specific prompts. This suggests that benchmark fairness in interactive video generation is as much an interface-design problem as a metric-design problem.

3. Hierarchical test suite and standardized scenarios

WorldMark provides a hierarchical test suite of about 500 evaluation cases. The suite is constructed from 50 reference images, paired first-person and third-person views, realistic and stylized scene variants, 15 standardized action sequences, and 3 difficulty tiers (Xu et al., 23 Apr 2026).

The paper describes the tiers as follows:

• Easy: single-segment trajectories, about 20 s
• Medium: two-segment compositions, about 40 s
• Hard: three-segment sequences, about 60 s

The image suite spans Nature, City, and Indoor scenes, and it includes both realistic and stylized aesthetics. The stylized subset includes oil painting, Ukiyo-e, cyberpunk, and Minecraft-style scenes. Original images are treated as first-person views, while corresponding third-person views are synthesized with an image-generation model, yielding a total of 100 test images when both viewpoints are included. The reference images are described as curated from WorldScore-derived sources, with duplicates and near-duplicates removed.

The action suite consists of 15 trajectories, ranging from simple forward motion and lateral translation to rotation-only motion, combined translation-plus-rotation, and cyclic or repetitive motion. A VLM-based filtering step selects physically plausible actions for each scene. This matters because it prevents the benchmark from conflating model failure with scene-action incompatibility. The benchmark is therefore standardized without being entirely scene-agnostic: it remains constrained by contextual plausibility.

The scenario design induces four evaluation splits: First-Person Real, First-Person Stylized, Third-Person Real, and Third-Person Stylized. The paper stresses that third-person generation is substantially harder, especially for control precision and geometric consistency, because the camera must move around a visible character rather than merely simulate egocentric motion.

4. Evaluation workflow and metric design

WorldMark uses a modular evaluation workflow with four stages: image selection, action mapping, video generation, and metric evaluation. Researchers may choose images by viewpoint, scene category, or style, or provide custom images; translate shared WASD plus L/R commands into model-specific controls; run the target model under standardized conditions; and then apply the eight-metric suite, optionally plugging in custom metrics (Xu et al., 23 Apr 2026).

The metric design spans three axes: Visual Quality, Control Alignment, and World Consistency.

For Visual Quality, the benchmark uses:

• Aesthetic Quality via the LAION aesthetic predictor
• Imaging Quality via MUSIQ

For Control Alignment, it uses pose-based geometric metrics reconstructed using DROID-SLAM:

• Translation Error
• Rotation Error

The paper specifies translation error as $\|\mathbf{t}_{\text{gt}} - s\mathbf{t}\|_2$, where $s$ is a least-squares scale factor, and rotation error as an $\arccos$ function of $\mathrm{tr}(\mathbf{R}_{\text{gt}}\mathbf{R}^T)$. These metrics test whether the generated video follows the intended camera motion.

For World Consistency, the benchmark combines geometric and VLM-based checks:

• Reprojection Error
• State Consistency
• Content Consistency
• Style Consistency

Reprojection error is computed with DROID-SLAM and dense bundle adjustment by averaging $\|\mathbf{p}_{ij}^* - \Pi(\mathbf{P}_{ij})\|_2$ over co-visible pixel pairs. The VLM-based metrics evaluate whether object identity and motion remain stable, whether objects appear or disappear unexpectedly, and whether style, lighting, and color palette remain stable.

A notable design feature is modularity. The benchmark does not insist on a closed metric ontology; rather, it standardizes the inputs and workflow while allowing the metric layer to evolve. This suggests a separation between benchmark protocol and metric implementation, which is useful in a rapidly moving area where evaluators may later replace or supplement current SLAM- and VLM-based scorers.

5. Evaluated models and reported empirical patterns

WorldMark evaluates six models: YUME 1.5, Matrix-Game 2.0, HY-World 1.5, HY-GameCraft (abbreviated HY-Game in tables), Open-Oasis, and Genie 3. All six are tested in the first-person setting, while only Matrix-Game 2.0, HY-World 1.5, and Genie 3 are tested in the third-person setting because only these support third-person viewpoints (Xu et al., 23 Apr 2026).

The benchmark reports several salient per-split outcomes.

Split	Model(s)	Reported standout metrics
First-Person Real	YUME 1.5	Aesthetic Quality 56.94; Imaging Quality 74.36
First-Person Real	HY-Game	Translation Error 0.159; Reprojection Error 0.447
First-Person Real	Matrix-Game 2.0	Rotation Error 1.324
First-Person Real	Genie 3	State Consistency 6.416; Content Consistency 6.914
First-Person Stylized	HY-World 1.5	Aesthetic Quality 58.50
First-Person Stylized	HY-Game	Translation Error 0.116; Rotation Error 0.932
First-Person Stylized	Genie 3	Reprojection Error 0.256; State Consistency 6.835; Content Consistency 7.306; Style Consistency 7.523
Third-Person Real	HY-World 1.5	Aesthetic Quality 57.69; Imaging Quality 70.76; Translation Error 0.206; Rotation Error 2.137
Third-Person Real	Genie 3	Content Consistency 7.424; Style Consistency 8.247
Third-Person Real	Matrix-Game 2.0	Rotation Error 27.606
Third-Person Stylized	HY-World 1.5	Aesthetic Quality 60.57; Imaging Quality 66.45; Rotation Error 5.285
Third-Person Stylized	Genie 3	Translation Error 0.129; Content Consistency 7.109; Style Consistency 8.541
Third-Person Stylized	Matrix-Game 2.0	Reprojection Error 0.744

From these results, the paper draws four main conclusions. First, visual quality and world consistency are largely uncorrelated: YUME produces attractive frames but weaker long-horizon coherence, whereas Genie 3 is the most consistent world simulator without being the most aesthetically impressive. Second, good control alignment does not guarantee high overall quality: HY-Game follows actions precisely, but its visual quality is weaker than the best models. Third, third-person generation is a major failure mode, with some systems showing dramatic degradation in rotation control and consistency. Fourth, domain-specific training does not generalize well: Open-Oasis, despite strength in Minecraft-like settings, performs poorly across the real-world and stylized scenes in WorldMark.

The paper further notes that switching to third-person can increase rotation error dramatically, with some models experiencing roughly an order-of-magnitude degradation. This is one of the benchmark’s most consequential empirical observations because it exposes a failure mode that private, model-specific evaluations could easily conceal.

6. Validation, platformization, and limitations

WorldMark includes a human preference alignment study involving 20 volunteers who manually ranked 50 sets of first-person videos from the six models. The reported Spearman rank correlation $\rho > 0.9$ indicates strong agreement between automated evaluation and human judgments (Xu et al., 23 Apr 2026). This does not eliminate dependence on proxy metrics, but it provides evidence that the benchmark’s automated scores track perceived comparative performance.

Beyond offline benchmarking, the paper launches World Model Arena at warena.ai, an online platform for side-by-side battles between leading world models and a live leaderboard. This extends the benchmark from a static evaluation suite into an ongoing comparative infrastructure. A plausible implication is that WorldMark is intended not only as a paper benchmark but also as a community protocol for continued model comparison.

Several limitations are evident from the benchmark design. The suite uses 50 reference images and 15 action templates, which is practical but still limited relative to the diversity of interactive world modeling tasks. Coverage is restricted to first-person and third-person viewpoints, without explicit treatment of other camera regimes or multimodal control settings. Fairness remains partly dependent on carefully hand-crafted adapter mappings from the canonical action space into each model’s native interface. The control and consistency metrics rely on external estimators such as DROID-SLAM and Gemini-3.1-Pro, so the benchmark inherits their biases and failure modes. Third-person support is also incomplete because not all models participate in all scenario splits.

These constraints do not negate the benchmark’s contribution. Rather, they clarify its scope. WorldMark standardizes evaluation conditions for a concrete class of interactive Image-to-Video world models and demonstrates that, under those conditions, visual quality, action obedience, and world coherence diverge in systematic ways. In that sense, its main significance lies less in proposing a single dominant metric than in formalizing a reproducible, extensible evaluation regime for a field that had previously lacked a shared testing ground.