PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

Abstract: Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence required to achieve the same objective may vary substantially between models, making fixed action-conditioned evaluation unsuitable for cross-model comparison. To address this, we employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives. Building on this paradigm, we introduce PlayWorld, a benchmark providing 171 scenarios, each with a specified objective. To evaluate performance thoroughly, we assess models along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. In addition, we incorporate basic ability metrics for video quality and controllability. Experiments across nine state-of-the-art world models reveal that current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. Code and data are available at https://github.com/kxding/PlayWorld.

• The paper introduces PlayWorld, a benchmark of 171 human-annotated cases and 50 action patterns that uses an adaptive Agent Player to evaluate nine world models over 10–60-second objectives.
• The benchmark shows that leading systems perform better on visible geometry than on persistent state evolution, with Genie 3 achieving the highest overall score at 2.12 but only 1.51 for insight evolution.
• The paper validates its automated VQA scoring against human preferences, reporting an overall Spearman correlation of 0.933 and showing that basic video quality and trajectory control do not reliably predict long-horizon world-model capability.

Motivation and problem statement

Interactive video world models—systems that generate environments continuously in response to user actions—are proliferating rapidly, spanning open-source systems such as Matrix-Game 3.0, LingBot-World2, and Hunyuan-GameCraft-2, and closed-source web products such as Genie 3 and HappyOyster. Evaluation, however, has not kept pace with this development. Existing benchmarks (WorldScore, WorldMark, MemoBench, WBench, WorldRoamBench, Omni-WorldBench) drive models with predefined low-level action trajectories. This paradigm is fundamentally mismatched to cross-model comparison: because models differ in action granularity and response dynamics, the same three commands may complete a full $360^\circ$ rotation in one rollout but only a partial turn in another, so geometry-consistency scores become incomparable. Fixed schedules also cannot handle complex patterns such as orbiting a landmark while holding a fixed viewing direction, and closed-source web-only systems resist scripted evaluation entirely.

PlayWorld (2608.13552) addresses this by replacing predefined trajectories with long-horizon objectives pursued by a multi-modal Agent Player, mirroring how a human player actually judges a world model: turn around fully and check whether the scene remains consistent; walk into water and inspect whether ripples appear.

The Agent Player: closed-loop control with a shared reference

The Agent Player couples a replaceable multi-modal agent model with an execution interface that translates decisions into each world model's native controls (W/A/S/D, arrow keys, WAIT), including browser automation for web-based models. A key design choice is the hybrid control strategy: rather than planning actions from scratch online (which incurs decision latency and produces divergent trajectories) or executing a fixed preset (which overshoots or undershoots across models), each case provides a human-annotated basic action sequence as a shared initial reference. At each step, the agent observes the generated frame and issues one of five decisions—Keep, Stop, Extend, Correct, or End—adapting action count and duration to the model's actual response within a 40-step budget.

Ablations on Genie 3 and HappyOyster support this design. Preset + Agent achieves the highest Trajectory Score (1.08 vs. 0.92/0.88 on Genie 3; 1.12 vs. 1.00/0.68 on HappyOyster) and Human Preference (~65–67% vs. 24–41%), while modifying only 12–15% of actions—confirming that targeted online adaptation, not wholesale re-planning, is what improves trajectory reliability. Notably, fully online "Agent Only" control performs worst, an instructive negative result for agentic-evaluation design. Agent-model selection is comparatively insensitive: Claude Haiku, Claude Sonnet, and Gemini 3.1 Pro yield similar trajectory quality, and Haiku is adopted for its latency advantage (3.83 s/call vs. 6.21 s for Sonnet).

Benchmark composition and VQA rubric verifier

The benchmark comprises 171 human-annotated cases covering 50 action patterns (one to five composed actions) over 10–60-second rollouts, evaluated across nine world models to produce over 1,400 interactive videos scored by more than 820 task-conditioned VQA questions. Initial worlds are curated from Pexels and Google Images across natural, urban, and fantasy environments; Gemini 3.1 Pro drafts captions that annotators verify, and annotators define objectives with visually verifiable completion conditions plus sample-specific rubrics.

Scoring is performed by Gemini 3.1 Pro answering weighted Yes/No rubric questions, aggregated into a 1–5 scale per dimension. Critically, each dimension except insight evolution applies a validation gate before scoring: Trajectory Validity (geometry, out-of-sight) or Subject-and-Reachability (interaction). Failed rollouts receive the minimum score of 1, preventing instruction-following failures from contaminating capability measurements—a concern previously raised by MemoBench. The four dimensions are:

• Geometry consistency: scene identity (object identity, count, texture) and spatial consistency under camera movement and revisitation.
• Interaction fidelity: contact/collision, motion causality, and visual response (e.g., ripples when entering water); reaching an obstacle deliberately does not end the rollout, so penetration failures are exposed.
• Out-of-sight evolution: reappearance consistency, hidden-state progression, and physical causality across disappear–reappear intervals.
• Insight evolution: identity/state progression, physical plausibility, and temporal scene consistency under stationary 60-second observation.

Complementary basic-ability metrics cover seven video-quality measures (VBench-style aesthetics/imaging/motion/flickering, Temporal Consistency, Depth Stability via Depth Anything V2, Subject Consistency adapted from HyDRA) and Action Controllability pass rates computed from VGGT-estimated poses against the executed (agent-adjusted) trajectories.

Principal findings

The headline results expose substantial weaknesses even among frontier systems:

Model	Geometry	Interaction	Insight	Out-of-sight	Overall
Genie 3	2.74	2.40	1.51	1.81	2.12
HappyOyster	2.54	2.15	1.47	1.54	1.92
LingBot-World2	2.04	2.13	1.95	1.16	1.82
Matrix-Game-3.0	1.30	1.25	1.00	1.00	1.14

Three observations stand out. First, persistent state evolution is the dominant bottleneck: out-of-sight and insight evolution scores are uniformly low (many near or at the floor of 1.0), indicating that models preserve visible structure better than they sustain semantic state change over time or through occlusion. Second, long-horizon revisitation reveals global spatial inconsistency: orbiting the Taj Mahal can regenerate the monument at multiple inconsistent locations—each frame plausible, but the sequence violating spatial uniqueness—consistent with reliance on local appearance continuity rather than a persistent global 3D representation. Third, trajectory control and world-model capability diverge: SANA-WM attains the second-highest validation pass rate (80.4%) yet modest rubric scores, showing it can follow trajectories without preserving memory or spatial consistency. Similarly, HappyOyster's top Basic Ability Score (76.4%) does not translate into the best long-horizon capability, underscoring that automatic quality metrics test only basic competence.

Interaction results are mixed: leading third-person models stop characters at obstacle boundaries, but first-person movement often penetrates solid geometry, and water/fire interactions frequently lack expected physical feedback.

Human validation

A study with 600 pairwise judgments (five raters, 120 video pairs balanced across dimensions) shows strong rank agreement between VQA scores and human preference: Spearman's $\rho = 0.933$ overall, up to $\rho = 0.983$ for geometry consistency, all significant at $p<0.05$. Inter-rater majority agreement reaches 95.8% with Fleiss' $\kappa = 0.434$, supporting the reliability of both the human rankings and the automated verifier's alignment with them.

Limitations and open questions

The paper concedes several constraints. VQA verification relies on a single Gemini 3.1 Pro scoring pass per rollout; a two-pass robustness check reports mean per-model sample variance of only 0.0112, but the authors acknowledge this is not a precise uncertainty estimate and recommend multi-pass averaging when budgets allow. Model-specific interface requirements introduce coverage gaps: HY-World2 skips cases whose initial images do not support global scene modeling, and Hunyuan-GameCraft-2 cannot always be evaluated under the intended protocol for insight-evolution cases requiring passive observation, since it demands non-empty actions per chunk. Decision-latency measurements depend on provider infrastructure and may not generalize. Finally, the benchmark's objectives are restricted to visually verifiable outcomes with continuous keyboard-style controls; whether the Agent Player paradigm extends to richer action spaces (gamepad, text instruction) or objectives requiring quantitative geometric ground truth remains untested.

Conclusion

PlayWorld reframes world-model evaluation around long-horizon objectives pursued by an adaptive agent player, replacing fixed action trajectories with closed-loop, observation-conditioned execution that preserves comparable evaluation intent across heterogeneous models. Its central empirical finding—that current state-of-the-art systems, including closed-source leaders, remain unreliable at spatial persistence and sustained state evolution over 10–60-second horizons—provides a concrete, human-aligned measurement basis against which progress on memory-augmented and globally consistent world modeling can be assessed.