WorldJen: Multi-Dimensional Video Model Benchmark

• WorldJen is a comprehensive benchmark for generative video models that assesses 16 quality dimensions including semantics, motion, physics, and aesthetics.
• It uses adversarially curated prompts from a large corpus and prompt-specific Likert questionnaires scored by a VLM at native video resolution.
• The benchmark reveals a clear three-tier model ranking and underscores challenges in physical realism, notably inertial consistency and physical mechanics.

WorldJen is an end-to-end benchmark for evaluating generative video models through a human-anchored, multi-dimensional pipeline rather than through pixel fidelity, distributional similarity, or binary video question answering alone. It is designed around two linked claims: first, that modern video quality is irreducibly multi-factorial, spanning semantics, motion, physics, temporal coherence, and aesthetics; second, that existing evaluation protocols frequently collapse these factors into weak signals that are either semantically blind or insufficiently discriminative. In response, WorldJen replaces binary VQA with prompt-specific Likert questionnaires graded by a VLM at native video resolution, and replaces one-dimension-at-a-time prompting with adversarially curated prompts intended to exercise up to 16 quality dimensions simultaneously (Inbasekar et al., 5 May 2026).

1. Problem setting and benchmark rationale

WorldJen is motivated by a critique of two families of prior evaluation. The first comprises reference-based metrics such as SSIM, PSNR, LPIPS, and FVD. In the WorldJen framing, SSIM and PSNR reward low-level similarity, LPIPS remains largely framewise, and FVD favors distributional resemblance and texture realism over physical plausibility, logical consistency, or faithful prompt execution. The second family comprises benchmark suites such as VBench, VBench++, and VBench 2.0, which decompose video quality into dimensions but either rely on specialist detectors and shallow features or use binary VQA, a setup that WorldJen identifies as prone to yes-bias and score compression (Inbasekar et al., 5 May 2026).

A central methodological claim is that prompts targeting one dimension at a time are both unrealistic and costly. WorldJen therefore uses adversarially curated prompts drawn from VidProM, a corpus of roughly 1.7 million human-authored video prompts. After filtering for safety, duplicates, length, complexity, and spam, around 5,000 prompts are retained; further review and remediation produce a final curated pool of 3,754 prompts. Each prompt is scored by Gemini 3.1 Flash-Lite on two 1–10 scales for every applicable evaluation dimension: suitability and difficulty. Prompts are then LLM-enhanced to improve both suitability and difficulty without changing the core theme. The appendix reports that before enhancement, prompts had a median of 10 dimensions above the suitability threshold $>6$, and after enhancement the median shifts to 15, which means most prompts test nearly the whole dimension set simultaneously (Inbasekar et al., 5 May 2026).

This design is also an explicit cost intervention. The benchmark contrasts itself with VBench, which reportedly requires 6,230 videos across 946 prompts, and VBench 2.0, which requires 3,209 videos across 1,013 prompts. WorldJen instead evaluates one video per model–prompt pair and reuses that same video across all applicable dimensions. A plausible implication is that discriminative benchmarking can be improved without proportionally increasing generation volume, provided prompts are deliberately adversarial and multi-dimensional.

2. Evaluation ontology: dimensions, prompts, and questionnaires

WorldJen organizes video quality into 16 dimensions grouped into four categories (Inbasekar et al., 5 May 2026).

Group	Dimensions
Motion & Stability	Subject Consistency; Scene Consistency; Motion Smoothness; Temporal Flickering; Inertial Consistency
Logic & Physics	Physical Mechanics; Object Permanence; Human Fidelity; Dynamic Degree
Instruction Adherence	Semantic Adherence; Spatial Relationship; Semantic Drift
Aesthetic Quality	Composition & Framing; Lighting & Volumetric; Color Harmony; Structural Gestalt

The prompt design is meant to make these dimensions interact rather than remain isolated. One enhanced prompt asks for “a full-body shot of a hulking humanoid dragon lord of terror” whose armor reflects blue and red inferno, who lunges forward, makes a sudden heavy stop, slashes a glowing sword into a crumbling temple floor, produces sparks, moves behind pillars, and casts dynamic shadows. Another example involves a female assassin riding a lavender horse through a fairytale forest and then abruptly decelerating. These prompts are not merely descriptive; they are constructed to expose inertial consistency, physical mechanics, object permanence, lighting, semantic adherence, and structural gestalt within a single clip (Inbasekar et al., 5 May 2026).

Prompt semantics are not passed directly to the judge at evaluation time. Instead, WorldJen generates prompt-specific, dimension-specific questionnaires offline. For each prompt and each applicable dimension, the system generates 10 evaluation questions. The generator is instructed to cover expected events and details, failure modes, success modes, and adversarial probes, and to avoid generic or near-duplicate questions. Each question is scored on a 1–5 Likert scale anchored as 1 = major failure, 3 = mediocre/passable, and 5 = flawless execution. This substitution of Likert grading for binary VQA is one of the benchmark’s central design choices, because it is intended to mitigate yes-bias and recover gradations of failure severity (Inbasekar et al., 5 May 2026).

3. Human ground truth and Bradley–Terry structure

WorldJen anchors its automated evaluation in a blind human preference study. The human protocol uses 50 prompts sampled from the curated pool, specifically prompts with suitability greater than 8 on at least 5 dimensions. Six state-of-the-art video generation models are evaluated on these prompts: Veo 3.1 Fast, Kling v2.6 Pro, LTX-2, Wan v2.2 A14B, Hunyuan v1.5, and Wan 2.1 1.3B as a local open-source baseline. One video is generated for each $(\text{model}, \text{prompt})$ pair, producing 300 videos total. Because all $\binom{6}{2}=15$ model pairs are compared for each prompt, the study contains 750 distinct comparison pairs (Inbasekar et al., 5 May 2026).

Seven annotators participated: four domain experts and three external annotators from mathematics, social sciences, and visual art backgrounds. Together they produced 2,696 pairwise annotations, with 100% coverage of all 750 comparison slots. Model identities were hidden as “Video A” and “Video B,” left/right positions were randomized independently for each pair, the response was forced-choice with three confidence levels (“Much better,” “Clearly better,” or “Slightly better”), and there was no tie option. Voting was disabled until both videos had loaded and played for at least 10 seconds, and a break was inserted every 50 pairs (Inbasekar et al., 5 May 2026).

Agreement is reported as a mean inter-annotator agreement of 66.9% and Krippendorff’s $\alpha = 0.273$, described by the authors as “Fair” for binary preference data. Human ground truth is then established by fitting a Bradley–Terry model with latent strengths $p_i>0$ such that

$P(i \succ j) = \frac{p_i}{p_i + p_j}.$

The resulting ratings reveal a clear three-tier structure (Inbasekar et al., 5 May 2026).

Model	Human BT rating	VLM BT rating
Veo 3.1 Fast	1614 $[1595,1635]$	1652 $[1590,1728]$
Kling v2.6 Pro	1572 $[1552,1592]$	1628 $[1571,1697]$
Wan v2.2 A14B	1518 $(\text{model}, \text{prompt})$0	1509 $(\text{model}, \text{prompt})$1
LTX-2	1479 $(\text{model}, \text{prompt})$2	1504 $(\text{model}, \text{prompt})$3
Hunyuan v1.5	1462 $(\text{model}, \text{prompt})$4	1433 $(\text{model}, \text{prompt})$5
Wan 2.1 1.3B	1355 $(\text{model}, \text{prompt})$6	1274 $(\text{model}, \text{prompt})$7

The top tier consists of Veo 3.1 Fast and Kling v2.6 Pro; the middle tier consists of Wan v2.2 A14B, LTX-2, and Hunyuan v1.5; the bottom tier consists of Wan 2.1 1.3B. The paper emphasizes that the top-to-mid gap and mid-to-bottom gap are larger than the relevant bootstrap half-widths, whereas within-tier ordering is not statistically resolved (Inbasekar et al., 5 May 2026).

4. VLM-as-a-judge engine

The automated component of WorldJen uses Gemini 3 Flash, specifically gemini-3-flash-preview, as the primary judge. Gemini 3.1 Flash-Lite is used in the prompt-scoring and prompt-enhancement stage, while Gemini 3 Flash is used both to generate questionnaires and to evaluate the videos. The judge receives prompt-specific questionnaires rather than the original prompt text directly. For each question it returns an integer Likert score and a short justification; chain-of-thought is deliberately avoided to reduce length bias, while the justifications are retained as an audit trail (Inbasekar et al., 5 May 2026).

Frame delivery is dimension-aware and performed at native video resolution. Three sampling modes are defined. Holistic mode samples 32 uniformly spaced frames across the clip for dimensions such as Semantic Adherence, Composition & Framing, Lighting & Volumetric, Color Harmony, Structural Gestalt, Dynamic Degree, and Semantic Drift. Sampled mode uses 16 uniformly spaced frames for Scene Consistency, Object Permanence, and Subject Consistency. Micro mode uses a dense prefix of about 12 frames extracted every fifth frame from the first $(\text{model}, \text{prompt})$8 seconds for Motion Smoothness, Temporal Flickering, Inertial Consistency, Physical Mechanics, and Human Fidelity. The stated rationale is that motion and physics failures are often most visible at action onset (Inbasekar et al., 5 May 2026).

The full 50-prompt, 6-model evaluation contains 47,160 actual scored responses. The theoretical maximum is $(\text{model}, \text{prompt})$9, but null dimensions are excluded: 78 Human Fidelity nulls on prompts with no humans and 6 Object Permanence nulls on prompts without trackable objects. Scores are aggregated in two directions. Dimension analysis averages the 10 question scores to obtain a $\binom{6}{2}=15$0 score and then averages across prompts. Leaderboard construction instead averages dimension scores into a per-prompt model score and converts the resulting pairwise wins into Bradley–Terry ratings (Inbasekar et al., 5 May 2026).

WorldJen also defines PHAS, the Predicted Human Alignment Score, calibrated against human pairwise preferences by learning non-negative ridge logistic regression weights on a 30-prompt calibration split and applying them on a held-out 20-prompt validation split. The PHAS formulation includes a variance-based penalty term $\binom{6}{2}=15$1 intended to downweight uncertain or internally inconsistent VLM judgments. This suggests that WorldJen treats evaluator variance not merely as noise but as a useful confidence signal (Inbasekar et al., 5 May 2026).

5. Empirical behavior, ablations, and comparison to earlier benchmarks

The canonical VLM leaderboard reproduces the human three-tier structure exactly. Compared against human Bradley–Terry ratings, the VLM rankings achieve Spearman $\binom{6}{2}=15$2, $\binom{6}{2}=15$3, and Kendall $\binom{6}{2}=15$4, $\binom{6}{2}=15$5. The paper explicitly cautions that, because there are only six models and the ranking naturally clusters into three well-separated tiers, $\binom{6}{2}=15$6 should be read as exact tier-level concordance rather than proof of fine within-tier discrimination. Even so, the concordance is supported by 47,160 scored responses rather than by a sparse pairwise signal (Inbasekar et al., 5 May 2026).

The dimension-wise analysis identifies Inertial Consistency and Physical Mechanics as the weakest dimensions across models, with score ranges of 2.72–3.31 and 2.73–3.45 respectively. By contrast, Color Harmony and Semantic Drift are high and tightly clustered. Human Fidelity is especially discriminative, with Veo scoring 4.15 and Wan 1.3B scoring 2.94. A cross-method validation of Semantic Adherence using Gemini Embedding 2 yields a Spearman correlation of $\binom{6}{2}=15$7, $\binom{6}{2}=15$8, differing only in a swap of Veo and Kling at the top (Inbasekar et al., 5 May 2026). This supports the paper’s substantive conclusion that physical plausibility remains the dominant unresolved weakness of current generative video systems.

The ablation suite is unusually dense. Prompt enhancement leaves global rank order unchanged, with Spearman $\binom{6}{2}=15$9, $\alpha = 0.273$0, but lowers the overall average Likert score from 4.164 to 4.053, which the authors interpret as difficulty amplification rather than rank manipulation. Subsampling the number of questions per dimension over $\alpha = 0.273$1 also leaves rank order unchanged, though score variance drops about 23-fold from $\alpha = 0.273$2 to $\alpha = 0.273$3. Prompt-count ablation shows mean Spearman correlation with the full 50-prompt ranking rising from 0.875 at $\alpha = 0.273$4 to 1.000 at $\alpha = 0.273$5, with perfect rank recovery in only 13.2% of 10-prompt draws but 100.0% at 50 prompts. Claude Sonnet 4.6 reproduces the same ranking and tier partition on 20 validation prompts; it is systematically stricter by about 1 point on average, but that offset largely cancels in pairwise Bradley–Terry aggregation (Inbasekar et al., 5 May 2026).

Uncertainty analysis also bears on benchmark robustness. Within-run variance above $\alpha = 0.273$6 is most common for Semantic Adherence (54.2%), Physical Mechanics (52.8%), and Dynamic Degree (47.2%), and least common for Color Harmony (6.0%), Temporal Flickering (9.8%), and Scene Consistency (11.5%). Extreme oscillation, defined as $\alpha = 0.273$7, appears in only 43 of 4,716 instances, or 0.9%. Re-running the full Gemini pipeline twice yields Pearson $\alpha = 0.273$8, Spearman $\alpha = 0.273$9, MAE $p_i>0$0, RMSE $p_i>0$1, and ICC(3,1) $p_i>0$2, while preserving the same three-tier structure; the only rank change is a swap between LTX-2 and Wan A14B within the middle tier (Inbasekar et al., 5 May 2026).

The direct comparison to VBench is one of WorldJen’s sharpest empirical claims. On the same 50 prompts, WorldJen recovers all 15 of 15 pairwise model orderings relative to human BT, with Spearman $p_i>0$3, $p_i>0$4, and Kendall $p_i>0$5, $p_i>0$6. VBench recovers 11 of 15 pairwise orderings, with Spearman $p_i>0$7, $p_i>0$8, and Kendall $p_i>0$9, $P(i \succ j) = \frac{p_i}{p_i + p_j}.$0. The paper attributes this gap to score compression in VBench, especially on shared conceptual dimensions such as subject consistency, scene/background consistency, motion smoothness, dynamic degree, and human quality (Inbasekar et al., 5 May 2026).

6. Position in the benchmark landscape and stated limitations

WorldJen belongs to a broader shift from generic video-quality evaluation toward world- and task-oriented benchmarking, but its emphasis is distinctive. WorldModelBench scores video generators as world models along Instruction Following, Physics Adherence, and Commonsense, using 350 text/image condition pairs and 67K human labels (Li et al., 28 Feb 2025). WorldMark standardizes interactive image-to-video evaluation through a shared WASD-style action vocabulary, a unified action-mapping layer, and a 500-case suite for first- and third-person control (Xu et al., 23 Apr 2026). 4DWorldBench evaluates 3D/4D world generation across Perceptual Quality, Condition-4D Alignment, Physical Realism, and 4D Consistency, using a hybrid of LLM-as-judge, MLLM-as-judge, and network-based methods (Lu et al., 25 Nov 2025). WorldJen’s contribution within this landscape is not low-level control or embodied rollout evaluation, but human-aligned ranking of text-to-video models under adversarial, high-coverage prompting (Inbasekar et al., 5 May 2026).

The paper is explicit about several limitations. Human evaluation collects only pairwise preferences, not dimension-specific human labels, so individual dimensions are not directly validated against human ratings. Only one seed is generated per model–prompt pair, which can inflate variance. Video duration is heterogeneous: Veo produces about 8-second videos while the others produce about 5 seconds, which may advantage or disadvantage different dimensions. Audio is ignored. The benchmark uses 50 prompts and 6 models, which is sufficient for stable tier-level conclusions but limited for finer-grained statistical power. Open-source auditors can recover the coarse three-tier structure, but Gemma 4 31B shows weaker correlation with human BT than Gemini, particularly on temporal, physics, and spatial reasoning dimensions. The paper also notes that open problems remain for text-image-to-video, audio-conditioned generation, video editing, and interactive world-model evaluation (Inbasekar et al., 5 May 2026).

The release strategy reflects the benchmark’s reproducibility orientation but also reveals a minor accounting tension. The authors release source code, prompts, questionnaires, VLM scoring results, 420 generated videos according to the abstract and release note, and the anonymized human dataset; the main body, however, describes 300 videos for the six-model shared dataset, suggesting that the larger release may include additional ablation assets. This discrepancy does not alter the benchmark’s core methodology, but it is relevant when interpreting the released corpus size (Inbasekar et al., 5 May 2026).

WorldJen’s central significance lies in treating evaluation as a structured measurement problem rather than as a single proxy metric. Its strongest empirical message is that modern generative video models form a clear three-tier hierarchy under dense human-anchored evaluation, and that physical plausibility—especially inertial consistency and physical mechanics—remains the field’s most persistent weakness.