- The paper introduces MME-CoF-Pro, a 303-sample benchmark with 16 reasoning categories and a process-level Reasoning Score that measures whether generated videos complete annotated cause-and-effect steps.
- The evaluation finds weak reasoning coherence and a sharp separation from visual quality: Veo-3.1 leads no-hint performance at 55.9 RS, while Kling reaches 65.1 generation quality but only 13.8 RS.
- Text hints generally increase step completion but reduce consistency, while visual hints help spatial and GUI tasks yet often cause models to render arrows or markers as scene content; neither hint type produces reliable gains as guidance increases.
Motivation and problem definition
Video generative models increasingly exhibit behaviors that resemble reasoning—planning, causal inference, and prediction of physical outcomes—but existing benchmarks do not directly measure whether generated events remain causally consistent across frames. The paper formalizes this property as reasoning coherence: the degree to which a generated video maintains consistent and plausible cause–effect relationships over time, including long-horizon dependencies, interaction reasoning, and the fidelity of intermediate transitions. Prior evaluation efforts fall into two camps: outcome-level benchmarks such as V-ReasonBench and Gen-ViRe verify only last-frame correctness, while process-aware efforts such as VIPER evaluate intermediate steps without targeting causal-chain consistency. MME-CoF-Pro is positioned as the first benchmark that both targets reasoning coherence explicitly and treats reasoning guidance (hints) as an independently controllable input variable.
Benchmark design
MME-CoF-Pro comprises 303 samples spanning 16 reasoning categories organized into four groups of increasing abstraction: perceptual reasoning (visual detail, rotation, object counting), spatial and structural reasoning (visual trace/maze navigation, real-world spatial, 2D/3D geometry), physical and causal reasoning (physics-based, 4D dynamics, natural science), and task-oriented reasoning (embodied manipulation, GUI, medical imaging, table/chart, text-based/mathematical, visual logical). Data are curated from 27 public real-world and synthetic benchmarks under their original licenses. Each sample is annotated with an average of 4.6 key reasoning steps (maximum 10), drafted by Gemini-2.5-Pro and refined through three rounds of manual verification by human annotators to ensure each step is necessary and unambiguously verifiable.
Two design elements distinguish the benchmark:
- Reasoning Score (RS): a process-level metric computed as the fraction of annotated reasoning steps correctly completed in the generated video, judged automatically by Gemini-2.5-Flash over uniformly sampled frames. Unlike last-frame correctness, RS captures partial correctness and localizes where along the reasoning chain a model fails.
- Controlled hint settings: every sample is evaluated under (a) no hint, (b) text hint (explicit textual descriptions of reasoning steps, averaging 17.4 extra words), and (c) visual hint (bounding boxes or directional arrows drawn on the input image) for the 8 perceptually demanding categories forming MME-CoF-Pro-mini. All other instruction components remain identical across settings, so performance differences can be attributed to the guidance itself.
Evaluation setup
Seven open- and closed-source models are evaluated with default configurations: Veo-3.1 and Veo-3.1-fast, Sora-2, Seedance-1.0-pro and -fast, Kling-v2.1, and Cosmos-Predict2-14B. Generation quality (GQ) is assessed along five dimensions—consistency score (CS), temporal consistency, visual stability, hallucination, and physics grounding—with CS reported separately. A human study with 10 participants scoring 10 videos validates the metric: RS achieves the highest Spearman correlation with human judgment (0.61), compared to 0.17 for Instruction Alignment and −0.41 for Pass@5 last-frame correctness. This negative correlation for outcome-level metrics is a notable result in itself, indicating that final-frame verification can actively misrepresent reasoning quality relative to human assessment.
Main findings
Weak reasoning coherence, decoupled from generation quality
Model-level results show clear stratification: Veo-3.1 leads with an average no-hint RS of 55.9, Sora-2 reaches 49.9, while Seedance-1.0-pro (35.7), Cosmos-Predict2 (28.6), and Kling-v2.1 (13.8) lag substantially. Even the strongest model barely exceeds chance-level step completion on many categories—for instance, Kling scores 0.0 RS on 2D Geometry Reasoning and 2.0 on Visual Detail Reasoning. More strikingly, RS and GQ are decoupled: Kling attains the highest average generation-quality score (65.1) alongside the lowest RS (13.8), producing visually coherent videos that ignore the required reasoning trajectory (e.g., rendering wind dynamics faithfully while failing to execute an instructed zoom-in search). This decoupling implies that visual fidelity metrics cannot serve as proxies for reasoning ability and motivates dedicated process-level evaluation.
Text hints inflate apparent correctness at the cost of consistency
Text hints improve average RS for most models (Veo-3.1 +4.2, Sora-2 +7.4, Cosmos +7.9), but all seven models suffer CS degradation under text hints, with drops as large as −15.6 in 4D Dynamics Reasoning, where every model degrades. Qualitative analysis reveals instruction-induced hallucination: models render textual descriptions literally rather than grounding them in the scene—for example, duplicating an elf to satisfy a motion directive instead of moving the existing one. The authors interpret this as overfitting to textual cues, where models prioritize instruction completion over visual faithfulness, inflating RS without genuine understanding.
Visual hints are task-dependent and frequently rendered as scene content
On MME-CoF-Pro-mini, visual hints help structured, spatially guided tasks—Embodied Reasoning gains up to +22.9 RS (Veo-3.1) and GUI Reasoning up to +20.0 (Seedance-1.0-pro)—but hurt fine-grained perception: Veo-3.1 loses 13.0 RS and 14.4 CS on Visual Detail Reasoning, and Seedance-1.0-fast shows comparable degradation. A recurring failure mode is that the hint itself is hallucinated into the video: arrows indicating motion direction are rendered or transformed into scene objects (e.g., an arrow becoming a curved trajectory). The authors attribute this to training-data distribution bias, where annotated markers co-occur with edited or synthetic content—an assumption stated as a hypothesis rather than an established cause.
Non-monotonic returns from additional hints
A scaling case study on Frozen Lake using Sora-2 progressively increases the number of text and visual hints across stages, rolling out each setting ten times. Both modalities raise RS above the no-hint baseline of 0.23, but neither exhibits monotonic improvement; curves fluctuate substantially with no clear upward trend. This indicates current models cannot cumulatively integrate increasingly detailed guidance, and that simply adding hint information does not guarantee improved reasoning coherence.
Limitations and open questions
Several constraints qualify these findings. The benchmark contains only 303 samples (67 with visual hints), so per-category results rest on small sample sizes and category-level fluctuations may not be statistically robust. Evaluation relies on Gemini-2.5-Flash as judge; although the human study supports RS's validity (Spearman 0.61), this correlation is moderate, leaving room for judge-model bias, particularly for fine-grained perceptual judgments. Videos are limited to 4–8 seconds at 24 fps, so long-horizon coherence beyond this window remains untested. The hint-scaling study covers a single environment (Frozen Lake) and a single model, making its conclusions preliminary. Finally, the mechanistic explanations for hint-induced hallucination—literal rendering of text cues and reproduction of visual markers—are hypotheses grounded in qualitative observation, not causal analyses; whether these behaviors stem from training-data distribution or from architectural factors is left unresolved.
Conclusion
MME-CoF-Pro contributes a controlled evaluation framework for reasoning coherence in video generative models, combining a human-validated process-level Reasoning Score with explicit manipulation of text and visual hint guidance. Its central empirical claims—that reasoning coherence is weak even in frontier models, that it is decoupled from generation quality, and that explicit hints improve apparent step completion while degrading consistency and inducing hallucinations—are supported by consistent cross-model patterns. The results collectively suggest that current models follow hints superficially rather than grounding them, identifying visual grounding, instruction understanding, and anti-hallucination mechanisms as concrete deficiencies that future work must address.