- The paper introduces a hierarchical agentic evaluation harness that routes cases to specialized skills, decomposes judgments into measurable sub-questions, and produces auditable evidence trees instead of opaque scores.
- The benchmark evaluates 18 world models across 330 cases and achieves strong human alignment, with Spearman correlations of 0.93 for Intentional Transition and 0.87 for Physical Transition.
- The paper shows that HarnessEval-W improves Physical Transition pairwise accuracy to 71.7% versus 31.9% for WBench and reveals that world-model fine-tuning can trade intervention correctness for persistence capabilities.
Overview
HarnessEval-W is an agentified evaluation harness for interactive world models, introduced by a large multi-institution collaboration spanning MirroS, Tsinghua, Peking University, NVIDIA, UC Berkeley, CMU, MIT, and others. The paper's central claim is that world model benchmarking should not reduce to a scalar score computed by a fixed rubric; instead, each evaluation should produce an inspectable reasoning chain that justifies the verdict. The authors argue that existing benchmarks compute metrics "brute-force," leaving no verifiable evidence trail, and that no prior benchmark automates the human ability to spot physical, causal, and state-consistency violations in generated rollouts. HarnessEval-W addresses this by importing the evaluation-harness paradigm from the LLM ecosystem: a parent agent interprets each case, routes it to applicable skills from an extensible skill library, decomposes each skill into measurable sub-questions answered by specialized sub-agents and diagnostic tools, validates the returned evidence, and aggregates it into a transparent evidence tree.
The system is instantiated as a live benchmark of 330 cases over 18 representative world models. Its reported headline results are strong: Spearman rank correlation of ρ=0.93 with human preference on Intentional Transition (ρ=0.87 on Physical Transition), pairwise accuracy of 71.7% versus 31.9% for the closest WBench protocol on Physical Transition, and run-to-run stability roughly 4.9× tighter than WBench under repeated evaluation.
The paper formalizes an interactive world model as predicting future observations conditioned on historical observations and user-specified actions, factorizing the observation distribution through hidden states into three components: initial state inference P(s0∣o−T,…,o0), observation likelihood S(oi∣si), and action-conditioned transition T(si∣si−1,ai−1). This factorization directly motivates three evaluation axes:
- Observation Quality: whether rendered observations are visually reliable, covering Render Quality (Obs-R) and Physical Observation (Obs-P).
- Transition Correctness: whether actions produce intended state changes at the right time, split into Exploratory (viewpoint/navigation), Intentional (targeted entity/relation/event edits), and Physical (intervention-induced dynamics) transitions.
- World Persistence: whether state sequences remain coherent over time, covering Drift Resistance, Revisit Consistency, and Offscreen Evolution (whether endogenous processes continue while unobserved rather than freezing or resetting).
Notably, persistence does not require a frozen world: invariant properties must hold while dynamic properties evolve consistently. The eight settings map one-to-one onto eight reported metrics, normalized to a 1–100 scale.
Hierarchical Agentic Workflow
The workflow is hierarchical and context-dependent. Case-specific skill routing selects applicable skills based only on the case context and evaluation intention—explicitly independent of the world model under test—so identical routing applies to all models on a given case, preserving fairness. Skills record evidence-grounded reasons for both activation and skipping (e.g., the Offscreen Evolution Verifier is skipped when all requested actions remain visible). Each activated skill spawns sub-agents, each answering one measurable sub-question with a discrete score and diagnostics; for example, the Intentional Change Verifier decomposes into eight sub-agents checking target visibility, transition occurrence, target correctness, final-state validity, anchor preservation, extra events, and judgeability, preceded by a sub-agent that predicts the expected outcome from case context. The parent agent then validates and aggregates evidence into the final score, and the process can recurse as sub-agents spawn their own investigations.
A concrete failure trace illustrates interpretability: when evaluating a cube-manipulation case, the Intentional Change Verifier detects an unrelated human intervention picking up the cube and zeroes the no-extra-event score—a diagnosis a holistic VLM judge would likely blur into an opaque low score.
Benchmark Construction
Cases are authored by an agentic pipeline rather than manually. A structured scene taxonomy spans six axes—environment (indoor/outdoor/transitional), foreground entities, midground spatial structure, scene density, appearance style, and perspective—and a sampler selects semantically compatible combinations. Six probe families correspond to the six non-observation settings; Observation Quality is evaluated in every case and thus has no separate family. Probe assignment is constrained by scene affordances (e.g., exploratory transitions require traversable routes; revisit consistency requires a valid departure-and-return path). Three agents then author each case: an Image Generator produces the initial observation, an image-grounded Planner specifies instructions, camera trajectories, control sequences, rollout plans, and physical parameters without altering the probe family or introducing absent entities, and a separate Case Validator audits visibility, feasibility, specificity of expected outcomes, and evidential adequacy, returning failed candidates for resampling. The released set comprises 330 validated cases across diverse environments, appearances, densities, perspectives, and interaction horizons.
Main Results
Eighteen models are evaluated on the same frozen 330-case set, grouped by conditioning interface: prompt-driven I2V generators, native-action autoregressive models, and camera-pose models. Seedance 2.0 ranks first overall at 75.5, followed by Wan 2.7 (75.0), Kling 3.0 (74.4), and MiniMax H3 (74.3)—all four text-driven general-purpose video generators, which the authors attribute to large-scale text-conditioned training providing strong grounding for interpreting action consequences. Strengths are distributed unevenly: Wan 2.7 leads Intentional (83.6) and Physical (71.1) Transition, while Seedance 2.0 leads Drift Resistance (79.8), HY-WorldPlay 1.5 leads Revisit Consistency (81.9) and Physical Observation (64.6), and SANA-WM leads Offscreen Evolution (72.3). Native-action and camera-pose models dominate Exploratory Transition (ABot-World at 83.5) but fall sharply on Intentional and Physical Transition (mostly below 51), indicating that interactive fine-tuning has not yet conferred semantic intervention grounding comparable to text-trained generators.
Evaluating the Evaluator
The paper devotes substantial effort to validating the evaluator itself, focusing on Intentional and Physical Transition as the hardest-to-automate settings. Human alignment is established via 5000 A/B judgments across nine models aggregated with a Bradley–Terry model, yielding Spearman correlations of 0.93 (Intentional) and 0.87 (Physical).
Against WBench protocols re-run on identical videos with the same GPT-5.5 backend, temperature, and frame sampling—so only the protocol differs—the results are stark:
| Setting |
Metric |
WBench |
HarnessEval-W |
| Physical |
Pairwise accuracy |
31.9% |
71.7% |
| Physical |
Draw rate |
52.2% |
1.8% |
| Intentional |
Pairwise accuracy |
60.2% |
77.8% |
| Intentional |
Draw rate |
36.1% |
11.1% |
WBench's Causal Fidelity compresses physics and causal consistency into a single 0–3 score, which explains its high draw rate; HarnessEval-W's decomposition into separately grounded sub-questions is both more discriminative and better aligned. Robustness testing over repeated runs shows HarnessEval-W's fitted score-to-human slopes within 9.6–10.8, correlations within 0.928–0.964, and a fit envelope of only 0.33 Bradley–Terry units, versus WBench's slope nearly doubling (11.2 to 21.0), correlation ranging 0.646–0.780, and a 1.61-unit envelope—4.9× wider. These are among the strongest claims in the paper and are supported by controlled comparisons.
Analysis of Capability Structure
Pairwise Pearson correlations across the eight axes over all 18 models reveal a nontrivial capability geometry. Render Quality is essentially uncorrelated with Physical Observation (r=−0.04), so visual polish does not imply physical plausibility. Intentional and Physical Transition are almost perfectly coupled (r=0.98), consistent with both demanding semantic understanding of commanded interventions. Exploratory Transition is nearly unrelated to either (r=−0.15 and r=−0.18): coherent open-loop continuation does not predict intervention grounding.
Fine-tuning analysis compares two lineage pairs—Wan 2.2 → DreamX-World and HunyuanVideo 1.5 → HY-WorldPlay 1.5—computing per-axis deltas. Both pairs show the same redistribution: DreamX-World gains 4.8 points in Exploratory, 7.8 in Revisit, and 3.5 in Offscreen while losing 11.9 in Intentional and 7.2 in Physical; HY-WorldPlay gains 8.4 in Revisit while losing 24.2 in Intentional and 11.2 in Physical. The authors' conclusion—that adapting a video generator into a world model redistributes capability rather than uniformly improving it—is well supported here, though their explanation (fine-tuning data emphasizing exploration-style trajectories over commanded interventions) remains a hypothesis rather than a demonstrated cause.
Limitations and Open Questions
Several limitations are acknowledged or implicit. The evaluator depends on a single VLM backend (GPT-5.5) for all sub-agents; robustness is measured across repeated runs of that backend, not across backend diversity, so systematic biases of the underlying judge are not characterized. Human-alignment validation covers only two of the eight settings (Intentional and Physical Transition); alignment on Observation Quality and Persistence axes is asserted but not quantified against human reference orderings. The 330-case scale is modest relative to the breadth of the taxonomy, and per-metric scores are averaged over small probe-family subsets, which may widen variance for less common families. The claimed causal account of fine-tuning capability loss is explicitly hypothetical. Finally, the proposed self-evolving directions—test-time scaling of the evaluator, skill-library growth, and recursively self-improving benchmarking—are outlined as future work rather than demonstrated, leaving open how skill gaps are detected reliably and whether self-authored skills preserve the validation rigor of the current library.
Conclusion
HarnessEval-W contributes an agentified, hierarchical evaluation harness for interactive world models that replaces fixed rubrics with case-adaptive skill routing, sub-agent decomposition, tool-grounded evidence gathering, and validator-audited aggregation, producing auditable evidence trees instead of opaque scores. Empirically, it achieves close human alignment (ρ up to 0.93), substantially outperforms the closest existing protocols on discriminative power and stability, and yields a finding of practical consequence: converting video generators into interactive world models trades intervention-correctness capabilities for persistence capabilities. Released as an open-source living benchmark, its long-term value will depend on whether community-contributed skills can maintain the validation standards that underpin its current reliability.
