World Consistency Score in Video Evaluation

• World Consistency Score is a unified metric that quantifies video generation coherence via object permanence, relation stability, causal compliance, and flicker penalty.
• It employs learned weighted sums and standardized sub-components to provide an interpretable score that aligns with human judgments.
• The metric informs both benchmark evaluations and reference-free assessments by diagnosing temporal, spatial, and physical consistency in generative videos.

Searching arXiv for papers on World Consistency Score, WBench, and related video/world-model evaluation. arxiv_search(query="WBench interactive video world model evaluation consistency 2026", max_results=5) World Consistency Score (WCS) denotes a family of metrics for evaluating whether a generated video sustains a coherent internal world over time. In its explicit formulation, WCS is a unified, interpretable, no-reference metric for generative video models that combines object permanence, relation stability, causal compliance, and flicker penalty into a single score aligned with human judgments (Rakheja et al., 31 Jul 2025). In adjacent literature, the term is used less formally: WBench does not define a metric named “World Consistency Score,” but its Consistency dimension is the closest construct and can be treated as a WCS-like aggregate over eight consistency sub-metrics (Ying et al., 25 May 2026). Related work on world-model video generation introduces reference-free physical-consistency measures without coining the WCS name, focusing instead on scalar anomaly ranking and spatio-temporal artifact localization (Oh et al., 21 Jun 2026).

1. Terminological status and scope

The term “World Consistency Score” is not yet a single universally standardized quantity. One paper explicitly introduces WCS as a unified metric for video generation quality (Rakheja et al., 31 Jul 2025). By contrast, WBench introduces a benchmark for interactive world model evaluation and reports a Consistency dimension rather than a metric named WCS (Ying et al., 25 May 2026). A third line of work proposes reference-free physical-consistency measures motivated by WorldScore and by robotic evaluation environments such as WorldGym and WorldEval, but does not coin the term WCS (Oh et al., 21 Jun 2026).

Source	Status of the term	Core structure
(Rakheja et al., 31 Jul 2025)	Explicitly defined	OP, RS, CC, FP with learned weights
(Ying et al., 25 May 2026)	Not defined by that name	Eight consistency sub-metrics averaged after rescaling
(Oh et al., 21 Jun 2026)	Not coined as WCS	Relative anomaly score and absolute artifact heatmap

This terminological divergence matters because the underlying evaluation targets differ. The explicit WCS paper frames world consistency as the internal, temporally coherent depiction of a plausible “mini-world” within a single generated video, where objects persist unless they plausibly exit or are occluded, inter-object relations evolve smoothly, and events obey basic physical causality (Rakheja et al., 31 Jul 2025). WBench operationalizes a related but broader notion for interactive video world models, emphasizing object identity and attributes across time, spatial layout stability under camera or agent motion, perspective anchoring, and temporal continuity (Ying et al., 25 May 2026). The reference-free physical-consistency work narrows the scope further to whether a generated video is physically consistent with a single coherent 3D scene undergoing plausible camera or scene motion (Oh et al., 21 Jun 2026).

2. Unified-metric formulation in generative video evaluation

In “World Consistency Score: A Unified Metric for Video Generation Quality” (Rakheja et al., 31 Jul 2025), WCS is defined over four interpretable sub-components: Object Permanence (OP), Relation Stability (RS), Causal Compliance (CC), and Flicker Penalty (FP). The metric is a learned weighted sum with a negative penalty for flicker:

$$\mathrm{WCS}(V)=w_{\mathrm{OP}}\mathrm{OP}(V)+w_{\mathrm{RS}}\mathrm{RS}(V)+w_{\mathrm{CC}}\mathrm{CC}(V)-w_{\mathrm{FP}}\mathrm{FP}(V).$$

The weights satisfy non-negativity constraints, and an optional bias term $b$ can be used during training or calibration, with post-hoc linear scaling to $[0,100]$ if desired (Rakheja et al., 31 Jul 2025). The paper also allows standardized features,

$s_k'=\frac{s_k-\mu_k}{\sigma_k},$

when training on z-scored submetrics.

Object Permanence measures whether tracked entities remain present after first appearance. The construction begins from detector-plus-tracker tracklets over frames $I_1,\ldots,I_T$, defines a binary visibility indicator $v_i(t)$ and first-appearance time $t_i^{\mathrm{start}}$, and averages persistence ratios across unique tracked objects. To avoid penalizing plausible disappearances, the formulation discounts full occlusion through an adjusted visibility $v_i^\star(t)=\max(v_i(t),o_i(t))$, and it also notes a border-exit heuristic when mask-based occlusion is unavailable. Identity robustness is improved by augmenting tracking with per-object appearance embeddings such as CLIP embeddings of cropped regions and merging fragmented tracklets with matching embeddings (Rakheja et al., 31 Jul 2025).

Relation Stability evaluates whether pairwise spatial relations change abruptly or implausibly. For co-visible pairs, it tracks centroid positions, 2D distances, left-right and above-below predicates, and contact state defined through bounding-box IoU. Instability events include ordering flips without plausible crossing and contact toggles without sufficient distance change. The aggregate is expressed as

$\mathrm{RS}(V)=1-\frac{1}{M}\sum_{(i,j)} r_{ij},$

where $r_{ij}$ is a per-pair instability rate. The formulation can down-weight relation events around explicit interactions such as collisions (Rakheja et al., 31 Jul 2025).

Causal Compliance measures whether significant motion changes and interactions have plausible causes and expected effects. It derives velocities and accelerations from tracked centroids, defines motion-change events using an acceleration threshold, and marks violations when sharp motion onset occurs without nearby collision or recognized agent action, or when a collision occurs without any kinematic response. The aggregate is

$b$0

where the numerator counts unexplained or ineffective causal events and the denominator counts significant motion-change and collision events (Rakheja et al., 31 Jul 2025).

Flicker Penalty measures frame-to-frame instability unexplained by motion. Using RAFT or an alternative flow estimator, the method warps frame $b$1 to $b$2 and computes a normalized residual,

$b$3

optionally restricted to low-flow regions. The aggregate $b$4 is the average over frame transitions (Rakheja et al., 31 Jul 2025).

The intended pipeline is fully open-source in orientation: object detection and segmentation may use YOLOv8, DETR or DINO with SAM; tracking may use ByteTrack, SORT, or DeepSORT; re-identification may use CLIP; action recognition may use SlowFast or MoViNet; optical flow may use RAFT or PWC-Net; and optional depth estimation may use MiDaS or DPT (Rakheja et al., 31 Jul 2025). Weight learning is proposed through either constrained MOS regression or a Bradley–Terry pairwise logistic preference model on human preference data.

3. WBench consistency dimension as a WCS-like aggregate

WBench is a comprehensive multi-turn benchmark for interactive world model evaluation along five dimensions: video quality, setting adherence, interaction adherence, consistency, and physics compliance. It contains 289 test cases and 1,058 interaction turns, covering diverse scenes, styles, subjects, first- and third-person perspectives, and four interaction types: navigation, subject action, event editing, and perspective switching (Ying et al., 25 May 2026). The paper does not define a metric named “World Consistency Score,” but its Consistency dimension is the closest operational construct.

Within WBench, world consistency targets four properties: object identity and attributes across time, spatial layout stability under camera or agent motion, perspective anchoring, and temporal continuity. These checks are meaningful precisely because the benchmark is multi-turn: navigation tests cross-view coherence and revisitation, subject action tests identity persistence, event editing stresses continuity under localized modifications, and perspective switching tests whether subject framing and scene layout remain anchored across viewpoint changes (Ying et al., 25 May 2026).

The WBench Consistency dimension comprises eight sub-metrics. Spatial Consistency measures revisitation similarity on roundtrip trajectories by matching the return frame to the first frame using MegaSaM trajectories and computing DreamSim similarity. Gated Spatial Consistency adds a motion gate with $b$5 to penalize near-static videos that would otherwise score highly on revisitation alone. Segment Continuity uses TransNetV2 shot-boundary detection, scoring a video as cut-free only when no frame exceeds the cut threshold of $b$6 under a minimum scene length of 10 frames. Perspective Consistency tracks the subject with SAM2 and evaluates normalized centroid stability, weighted by the presence rate over valid frames with mask area at least 10 px. Geometric Consistency uses Depth Anything 3 depth and pose estimates to compute reprojection displacement normalized by the image diagonal, with score $b$7. Photometric Consistency uses the same reprojection to warp frames and evaluates PSNR. Subject Consistency uses SAM2 masks together with DINOv2 and CLIP cosine similarities to measure adjacent-frame continuity and anchoring to the first frame. Background Consistency uses consecutive-frame CLIP similarity over full frames. All eight sub-metrics rely on specialist vision models; no LMM scoring is used in consistency (Ying et al., 25 May 2026).

Scores are computed per video, aggregated per model over the applicable cases, and linearly rescaled to $b$8, with higher being better. If the Consistency dimension is interpreted as a WCS-like score, the aggregation is the arithmetic mean of the eight model-level sub-metric means:

$b$9

Here, $[0,100]$0 denotes the per-model average of the rescaled per-video scores for sub-metric $[0,100]$1, computed over the set of videos to which that sub-metric applies; the roundtrip-conditioned spatial metrics, for example, apply only on roundtrip navigation cases (Ying et al., 25 May 2026).

This formulation makes WBench’s WCS-like score fundamentally diagnostic rather than monolithic. High scores indicate strong temporal coherence, robust identity persistence, stable subject framing, coherent geometry and appearance under motion, and absence of hard cuts. Low scores indicate drift, identity swaps, anchoring instability, layout breakage, and reprojection inconsistencies (Ying et al., 25 May 2026).

4. Reference-free physical consistency assessment

“Reference-Free Assessment of Physical Consistency in World Model-based Video Generation” introduces two reference-free measures for generated videos: a relative assessment, used as a scalar anomaly score for ranking or filtering videos, and an absolute assessment, used as a spatio-temporal heatmap for localizing artifacts (Oh et al., 21 Jun 2026). Although the paper does not coin the name WCS, a plausible interpretation is that these measures form scalar and spatial analogues of world-consistency scoring focused specifically on physical integrity.

The formulation assumes a pinhole camera with intrinsics $[0,100]$2, camera poses $[0,100]$3, and depth $[0,100]$4. A pixel $[0,100]$5 is back-projected into 3D, transformed by the relative motion from $[0,100]$6 to $[0,100]$7, and reprojected to obtain a geometry-induced motion field $[0,100]$8. SEA-RAFT supplies optical flow, while DROID-SLAM supplies poses, depth, and factor-graph residuals. From these components, the method defines three residuals: the flow-versus-geometry residual

$[0,100]$9

a SLAM reprojection residual $s_k'=\frac{s_k-\mu_k}{\sigma_k},$0, and a forward–backward cycle-consistency residual $s_k'=\frac{s_k-\mu_k}{\sigma_k},$1 (Oh et al., 21 Jun 2026).

Occlusions and dynamic regions are handled through weighting. The occlusion mask is derived from forward–backward flow consistency; dynamic masking excludes or downweights pixels with large flow divergence, low SLAM inlier weight, or low RAFT confidence; and the final weight is

$s_k'=\frac{s_k-\mu_k}{\sigma_k},$2

Frame-level aggregation then computes a robust mean of weighted residuals,

$s_k'=\frac{s_k-\mu_k}{\sigma_k},$3

with suggested starting values $s_k'=\frac{s_k-\mu_k}{\sigma_k},$4 and $s_k'=\frac{s_k-\mu_k}{\sigma_k},$5 (Oh et al., 21 Jun 2026).

At the video level, two base metrics are retained: $s_k'=\frac{s_k-\mu_k}{\sigma_k},$6, a 3D consistency error instantiated from reprojection or factor-graph residuals, and $s_k'=\frac{s_k-\mu_k}{\sigma_k},$7, a photometric or flow consistency error instantiated by the average end-point discrepancy between SEA-RAFT flow and SLAM-implied geometry. To weight these components, the paper computes Shannon entropies over calibration-set histograms with equal-width bins $s_k'=\frac{s_k-\mu_k}{\sigma_k},$8, then applies a softmax over negative entropy:

$s_k'=\frac{s_k-\mu_k}{\sigma_k},$9

After z-score normalization,

$I_1,\ldots,I_T$0

where lower values indicate lower anomaly and thus higher physical consistency. Optional higher-is-better mappings include $I_1,\ldots,I_T$1 or $I_1,\ldots,I_T$2 (Oh et al., 21 Jun 2026).

The absolute assessment localizes failures in time and space. After excluding the initial warm-up frames $I_1,\ldots,I_T$3, the method finds the frame $I_1,\ldots,I_T$4 with the maximal jump in SLAM frame error,

$I_1,\ldots,I_T$5

It then selects the factor-graph edge with the largest mean distortion and visualizes the normalized magnitude of its dense residual vector field as an artifact heatmap. An optional generalized map combines $I_1,\ldots,I_T$6, $I_1,\ldots,I_T$7, and $I_1,\ldots,I_T$8 over time (Oh et al., 21 Jun 2026).

5. Validation, diagnostics, and comparative findings

WBench validates its automatic metrics against human judgments through a large-scale human preference study with 400 annotators and triple redundancy. Spearman rank correlations between per-model human win rates and automated scores across ten aspects are all at least $I_1,\ldots,I_T$9, and four aspects reach $v_i(t)$0; spatial consistency specifically achieves $v_i(t)$1 (Ying et al., 25 May 2026). This is evidence that the consistency suite used in WBench tracks human ranking closely, at least at the model-comparison level defined by the benchmark.

WBench’s diagnostics also show that consistency is not reducible to raw navigation accuracy. Motion magnitude is negatively correlated with consistency, with Pearson $v_i(t)$2, which motivates the gated spatial variant that penalizes stationary videos. Accurate navigation does not imply stable subject anchoring: several strong navigation models are weak on perspective consistency. The benchmark identifies characteristic failure modes, including identity drift or swaps, near-static videos inflating revisitation similarity, layout breakage or geometric distortion, temporal shot cuts, appearance flicker, and long-horizon degradation under multi-turn interaction (Ying et al., 25 May 2026).

Across 20 state-of-the-art models in WBench, no single model performs strongly across all five benchmark dimensions (Ying et al., 25 May 2026). On the Consistency dimension as interpreted here, LingBot-World achieves the highest overall score, reported as approximately $v_i(t)$3 on the navigation split. Camera-controlled models lead geometric consistency but can underperform on perspective consistency. Text-driven models show a smaller drop from spatial to gated spatial consistency, approximately $v_i(t)$4 points versus approximately $v_i(t)$5 for camera-controlled models, suggesting that some camera-controlled runs are near-static under the gated criterion. The benchmark analysis also highlights Kling 3.0, Wan 2.7, LongCat-Video, HY-World 1.5, and Matrix-Game 2.0 for specific strengths or failure patterns (Ying et al., 25 May 2026).

The explicit WCS paper presents a validation blueprint rather than finalized benchmark numbers. It proposes evaluation on VBench-2.0, EvalCrafter, and LOVE, comparing WCS with FVD, CLIPScore, VBench composite, and FVMD, and it recommends sensitivity analyses through synthetic artifact injection, component ablations, and robustness tests under changes in video length, object count, motion magnitude, and tool noise (Rakheja et al., 31 Jul 2025). This makes the paper a metric proposal and technical specification rather than a benchmark report.

The reference-free physical-consistency work provides downstream evidence in robotic evaluation. Across five OpenVLA tasks in WorldGym, with 30 generated rollouts per task for 150 total videos, filtering by the relative assessment improves downstream VLA evaluation accuracy: the low-anomaly group achieves $v_i(t)$6 success, compared with $v_i(t)$7 for random selection, and approaches the real-world reference of $v_i(t)$8 (Oh et al., 21 Jun 2026). Entropy-derived weights consistently favor 3D consistency, with $v_i(t)$9–$t_i^{\mathrm{start}}$0 and $t_i^{\mathrm{start}}$1–$t_i^{\mathrm{start}}$2, and the absolute heatmaps localize morphology artifacts such as deformed robot arms (Oh et al., 21 Jun 2026).

6. Limitations and prospective directions

The explicit WCS formulation inherits the limitations of its tools. Detector, tracker, optical-flow, and action-recognition errors directly affect score reliability, especially for off-distribution content such as cartoons or abstract art. Causal compliance remains difficult because subtle causes may be missed and benign motions may be mislabeled; the paper characterizes its physics checks as heuristic and suggests that better learned “physics critics” may be needed. WCS is also intentionally narrow: it focuses on consistency rather than visual fidelity, so a consistently blurry or low-detail video may score well; slow attribute drift is not fully captured by object permanence alone; and stylized or intentionally non-physical videos may be penalized despite legitimate artistic intent (Rakheja et al., 31 Jul 2025).

WBench exposes a different set of limitations specific to interactive world models. The benchmark analysis suggests that reducing near-static responses under navigation is important for fair spatial evaluation; that explicit viewpoint anchoring losses or controllers may improve perspective consistency; that cross-view geometric regularizers can improve reprojection-based metrics; that identity-aware temporal modules such as memory or persistent tokens may suppress drift and swaps; and that better chunk stitching may reduce implicit hard cuts (Ying et al., 25 May 2026). These are not universal prescriptions, but they indicate the implementation levers most directly tied to WBench’s observed failure modes.

The reference-free physical-consistency approach is also bounded by SLAM and flow reliability. Tracking can fail under severe structural inconsistencies, extreme non-rigid motion, de-lighting conditions, or inserted corrupted frames; absolute localization can be imprecise for broad non-rigid deformations; and low anomaly is necessary but not sufficient for downstream task success, as shown by cases where a robot never contacts the object despite a physically plausible video (Oh et al., 21 Jun 2026). The paper therefore suggests extensions through scene flow, explicit rigidity masks, multi-view or multi-seed consistency checks, stronger self-calibration of intrinsics, and use of generator-provided depth or normal maps (Oh et al., 21 Jun 2026).

This suggests that “World Consistency Score” is best understood not as a settled single metric but as an evaluation axis centered on temporal, geometric, causal, and photometric coherence. The current literature spans a learned four-component scalar metric for single-video assessment, an eight-component benchmark dimension for multi-turn interactive world models, and a reference-free physical-consistency pipeline for world-model video generation. Their common premise is that world quality cannot be reduced to visual fidelity or prompt alignment alone, because coherent video generation requires that a depicted world continue to make sense from frame to frame and turn to turn (Rakheja et al., 31 Jul 2025).