HarnessEval-W: Agentic World Model Evaluation

• HarnessEval-W is an agentic evaluation pipeline that decomposes world-model tests into validated subproblems, using specialized agents and diagnostic tools to assess observation quality, transitions, and world persistence.
• The benchmark evaluates 330 cases across eight metrics and 18 world models, producing auditable evidence trees that distinguish genuine model failures from inapplicable tests, insufficient evidence, and evaluator limitations.
• HarnessEval-W closely matches human judgments, reaching Spearman correlations up to 0.93 and outperforming WBench in pairwise accuracy for physical and intentional transition evaluation.

HarnessEval-W is an agentified evaluation pipeline for world models that replaces fixed, monolithic scoring procedures with hierarchical, evidence-grounded investigation. It interprets each evaluation case, decomposes its requirements into measurable subproblems, invokes specialized sub-agents and diagnostic tools, validates the resulting evidence, and aggregates the validated findings into a final verdict. The pipeline is designed to evaluate whether generated rollouts preserve visual structure, physical plausibility, causal relationships, and coherent world state over time, while retaining an auditable evidence tree for every score (Chen et al., 17 Aug 2026).

1. Purpose and conceptual foundations

Conventional world-model benchmarks commonly reduce a generated rollout to a scalar obtained from fixed metrics. Such metrics can measure image quality, prompt adherence, temporal stability, or selected physical properties, but do not necessarily identify which frames, entities, relations, or events support the result. They may also fail to distinguish genuine model errors from insufficient evidence, inapplicable measurements, or evaluator failures.

HarnessEval-W treats evaluation as an agentic workflow. The evaluator interprets the case context, identifies the intended intervention and relevant world-state constraints, selects applicable evaluation skills, gathers targeted evidence, validates intermediate conclusions, and constructs a final verdict. The central output is therefore not only a score but an evidence tree linking the score to subquestions, observations, tools, validation decisions, and failure diagnoses.

The paper models an interactive world model using initial observations $\{o_i\}_{i=-T}^{0}$ and future actions $\{a_i\}_{i=0}^{t-1}$. Future observations are represented through a latent-state transition and rendering process:

$$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$

Here, $s_i$ denotes latent world state, $T$ the action-conditioned transition model, and $S$ the observation or rendering model. This formulation motivates three evaluation capabilities:

1. Observation Quality: whether the current state is rendered coherently and plausibly.
2. Transition Correctness: whether an intended action produces the appropriate change.
3. World Persistence: whether latent state remains coherent across time, revisitation, and periods of unobserved evolution.

The benchmark applies these capabilities to prompt-driven, camera-conditioned, and native-action world models while retaining their native interfaces.

2. Evaluation axes and metrics

HarnessEval-W uses three top-level axes and eight detailed metrics. Observation metrics are evaluated for every case; transition and persistence metrics are evaluated for the applicable probe families.

Axis	Metrics	Evaluation focus
Observation Quality	Render Quality (Obs-R); Physical Observation Quality (Obs-P)	Visual coherence, readability, structural and physical plausibility
Transition Correctness	Exploratory Transition (Trans-E); Intentional Transition (Trans-I); Physical Transition (Trans-P)	Viewpoint changes, specified edits, causal and physical consequences
World Persistence	Drift Resistance (Pers-D); Revisit Consistency (Pers-R); Offscreen Evolution (Pers-O)	Identity, layout, appearance, revisitation, and hidden dynamics

Render Quality (Obs-R) evaluates video coherence, stability, readability, and general visual reliability. Physical Observation Quality (Obs-P) evaluates whether individual frames are structurally and physically plausible.

Exploratory Transition (Trans-E) examines whether a requested viewpoint or positional change occurs while the world remains compatible. Intentional Transition (Trans-I) evaluates whether a specified entity, relation, or event changes as requested while protected state remains stable. Physical Transition (Trans-P) examines whether a physical intervention produces the expected dynamical response.

Drift Resistance (Pers-D) measures stability of layout, style, object identity, and appearance over long rollouts. Revisit Consistency (Pers-R) examines whether a location or object remains compatible after the observer leaves and returns. Offscreen Evolution (Pers-O) tests whether endogenous processes continue while temporarily unobserved rather than freezing or resetting.

The original metric values range from $0$ to $1$, while reported scores use a $1$–$100$ scale. The overall score is the arithmetic average of 330 case-level scores:

$\{a_i\}_{i=0}^{t-1}$0

For metric $\{a_i\}_{i=0}^{t-1}$1, the reported value is averaged over its applicable case subset. Observation metrics use all cases, whereas transition and persistence metrics use cases belonging to their corresponding probe families. The supplied paper does not specify a more detailed weighting rule.

3. Hierarchical harness architecture

The HarnessEval-W harness is scaffolding for an agentic evaluation process rather than a wrapper around a single metric. It contains reusable evaluation skills specifying when a procedure applies, what evidence it requires, which subquestions should be asked, which tools can provide grounding, and how validated evidence contributes to a score.

Routing is case-dependent but model-independent. The evaluator selects skills according to the evaluation intention and case context, not according to the identity or apparent quality of the evaluated model. This design is intended to prevent the evaluator from applying different standards to different models.

A typical workflow has the following stages:

1. Case interpretation: identify entities, intended actions, expected changes, protected invariants, relevant temporal horizons, and required evidence.
2. Skill routing: select primary and supporting skills, while recording reasons for activated and skipped skills.
3. Subproblem decomposition: convert a broad evaluation question into specialized, measurable checks.
4. Evidence gathering: assign sub-agents and diagnostic tools to inspect relevant frames, regions, temporal windows, trajectories, or state transitions.
5. Parent-agent validation: assess applicability, evidence sufficiency, temporal correctness, entity identity, contradiction, and judgeability.
6. Aggregation: combine validated sub-agent findings into skill-level, case-level, metric-level, and overall results.
7. Trace recording: preserve the complete reasoning chain and evidence provenance.

For an intentional-change case, the Intentional Change Verifier is described as using eight branches that examine expected-outcome specification, target visibility, transition occurrence, target correctness, final-state validity, anchor preservation, unrelated events, and overall judgeability. For a physical collision, representative checks include object localization and tracking, spatial extent estimation, temporal contact, motion estimation, and post-contact response.

Sub-agents receive tailored context rather than indiscriminately receiving the entire evaluation problem. Their inputs can include the case specification, initial image, action instruction, expected outcome, relevant frames, target object or region, subquestion, and upstream findings. Their outputs include a discrete score, diagnostic rationale, supporting evidence, and potentially a failure label or indication that the question is not judgeable.

The parent agent validates whether the selected skill was applicable, whether the target was visible and identifiable, whether evidence addressed the subquestion, whether temporal ordering was correct, whether tool output was reliable, and whether contradictions existed. Insufficient evidence is not automatically converted into a model failure, while genuine violations remain negative evidence.

4. Evidence trees and failure handling

The defining output of HarnessEval-W is an evidence tree:

$\{a_i\}_{i=0}^{t-1}$2

Each node can record the tested question, relevant context, responsible agent or tool, input evidence, observation or score, rationale, validation status, aggregation contribution, and failure or skip status. Sub-agents may recursively spawn further investigations when additional evidence is required.

The pipeline distinguishes several classes of outcome:

• Invalid case: returned to the sampler for resampling or regeneration.
• Inapplicable skill: skipped with a recorded reason.
• Insufficient evidence: treated as a judgeability or evidence-validity issue, not automatically as a model failure.
• Detected violation: retained as negative evidence.
• Contradictory or unreliable evidence: subjected to parent-agent validation before aggregation.
• Unsupported scenario: treated as a limitation of the evaluator and a possible trigger for future skill expansion.

Skipping is a substantive part of the protocol. For example, the Offscreen Evolution Verifier can be skipped when all relevant actions remain visible. Unsupported measurements should not be interpreted as failures merely because the corresponding skill was not applicable.

The evidence-tree design localizes failure. In a described cube-change case, the evaluator detects an unrelated human intervention that picks up the cube. The “no extra event” branch therefore receives zero even if target-change checks succeed. This exposes a causal and intentionality violation rather than hiding it within an undifferentiated holistic judgment.

The paper does not specify a complete error taxonomy, retry budget, abstention equation, or exact mathematical aggregation rule for combining subquestions. It is therefore not possible to determine from the supplied material whether individual branches are averaged, multiplied, minimized, or combined through rule-based logic.

5. Benchmark construction and evaluation protocol

HarnessEval-W contains 330 evaluation cases generated through an agentic construction pipeline. Initial scenes are sampled along six axes:

1. Environment: indoor, outdoor, or transitional.
2. Foreground: people, vehicles, articulated objects, apparatuses, and other manipulable entities.
3. Midground: corridors, sidewalks, workbenches, roads, shorelines, and related structures.
4. Scene density: clutter level while preserving readability.
5. Appearance: photorealistic, cinematic, game-engine, stylized 3D, or anime-like styles.
6. Perspective: first-person or third-person viewpoint.

The sampler rejects semantically incompatible combinations. Six probe families cover transition and persistence settings: Exploratory Transition, Intentional Transition, Physical Transition, Drift Resistance, Revisit Consistency, and Offscreen Evolution. Observation Quality is evaluated across all cases rather than treated as a separate family.

Case authoring includes four components:

• Image Generator: converts sampled metadata into an initial-world prompt and produces the initial observation.
• Image-grounded Planner: specifies a feasible action, camera path, control sequence, rollout plan, and physical conditions from the generated image.
• Case Validator: verifies target visibility, action feasibility, specificity of the expected result, and adequacy of evidence.
• Resampling/regeneration loop: rejects invalid candidates instead of silently retaining them.

Cases provide an initial observation, an action or interaction instruction, an evaluation setting or world-state question, and a generated rollout. The interaction is translated into the native interface of each model: text instructions for prompt-driven video models, camera trajectories or poses for camera-conditioned models, and native control sequences for action-conditioned models.

The benchmark evaluates 18 world models using released checkpoints or official APIs. Each model is evaluated through its native interface. Reported overall scores are:

Model	Interface	Overall
Seedance 2.0	Prompt I2V	75.5
Wan 2.7	Prompt I2V	75.0
Kling 3.0	Prompt I2V	74.4
MiniMax H3	Prompt I2V	74.3
Grok Imagine 1.5	Prompt I2V	73.4
FLUX 3	Prompt I2V	72.2
Cosmos3-Super	Prompt I2V	71.9
HunyuanVideo 1.5	Prompt I2V	70.3
LingBot World v2	Camera pose	68.8
SANA-WM	Native action	68.7
Wan 2.2	Prompt I2V	67.7
HY-WorldPlay 1.5	Camera pose	67.1
DreamX-World	Native action	66.8
ABot-World	Native action	66.1
Lyra 2	Camera pose	65.5
LTX-2.3	Prompt I2V	64.6
Fantasy-World	Camera pose	62.1
InSpatio-World	Camera pose	61.4

Metric-level leaders differ by capability. Grok Imagine 1.5 leads Obs-R at $\{a_i\}_{i=0}^{t-1}$3, HY-WorldPlay 1.5 leads Obs-P at $\{a_i\}_{i=0}^{t-1}$4 and Pers-R at $\{a_i\}_{i=0}^{t-1}$5, ABot-World leads Trans-E at $\{a_i\}_{i=0}^{t-1}$6, Wan 2.7 leads Trans-I at $\{a_i\}_{i=0}^{t-1}$7 and Trans-P at $\{a_i\}_{i=0}^{t-1}$8, Seedance 2.0 leads Pers-D at $\{a_i\}_{i=0}^{t-1}$9, and SANA-WM leads Pers-O at $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$0. The distribution indicates that overall ranking does not capture all dimensions of world-model competence.

6. Human alignment, robustness, and limitations

The human comparison collected 5,000 pairwise A/B judgments over nine representative models. Annotators compared two rollouts generated for the same case and selected the preferred one. The judgments were aggregated into per-model strengths using a Bradley–Terry model.

HarnessEval-W’s ordering aligned with human preferences with Spearman $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$1 and Kendall $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$2 for Intentional Transition, and Spearman $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$3 and Kendall $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$4 for Physical Transition.

The paper compares HarnessEval-W with WBench on the same videos, using the same GPT-5.5 backend, temperature, and frame sampling:

Setting	Evaluator	Pairwise accuracy	Draw rate
Physical	WBench	31.9%	52.2%
Physical	HarnessEval-W	71.7%	1.8%
Intentional	WBench	60.2%	36.1%
Intentional	HarnessEval-W	77.8%	11.1%

HarnessEval-W also achieved the lower Brier score in both settings. The supplied paper does not provide the exact Brier values or its fitted Bradley–Terry equations.

Repeated GPT-5.5 evaluation at temperature $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$5 produced HarnessEval-W score-to-human slopes from $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$6 to $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$7 and correlations with human strength from $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$8 to $$P(o_1,\ldots,o_t\mid o_{-T},\ldots,o_0;a_0,\ldots,a_{t-1}) \propto P(s_0\mid o_{-T},\ldots,o_0) \prod_{i=1}^{t}S(o_i\mid s_i)T(s_i\mid s_{i-1},a_{i-1}).$$9. Its fitted-curve envelope spanned $s_i$0 Bradley–Terry units. WBench’s slopes ranged from $s_i$1 to $s_i$2, correlations from $s_i$3 to $s_i$4, and its envelope spanned $s_i$5 units. The HarnessEval-W envelope was therefore $s_i$6 narrower under the reported analysis.

The benchmark targets violations involving physics, causality, temporal consistency, and world-state consistency. Examples include incorrect collision behavior, objects passing through one another, implausible support relations, incorrect post-intervention motion, object teleportation, altered protected anchors, layout drift, inconsistent attributes, reset after revisitation, frozen offscreen dynamics, unrelated interventions, and failure to execute the requested transition.

The supplied paper does not report conventional component ablations isolating parent-agent reasoning, sub-agent decomposition, diagnostic tools, context selection, evidence validation, skill routing, or aggregation rules. It also does not provide computational cost, latency, token usage, or dollar-cost measurements. Consequently, the comparative results support claims about the complete HarnessEval-W protocol rather than causal claims about individual architectural components.

Other limitations include evaluator dependence on the VLM and diagnostic tools, possible bias from hand-designed skills and case construction, incomplete formal specification of normalization and aggregation, limited human comparisons focused on selected dimensions, and the absence of a full variance decomposition across temperatures and VLM backends. The benchmark’s future direction includes test-time scaling, a larger reusable skill library, explicit detection of missing skills, external or self-driven skill acquisition, and recursive addition of evaluation procedures.

HarnessEval-W’s principal contribution is an evidence-grounded, hierarchical paradigm for evaluating visual worlds. It reframes benchmarking from brute-force metric computation toward structured investigation: identifying what should happen, gathering the relevant evidence, validating whether that evidence is sufficient, and exposing how each final score follows from observable world-state and causal judgments.