Video-MME-v2: Robust Video Understanding Benchmark

• The paper introduces a tri-level hierarchy that assesses visual aggregation, temporal dynamics, and complex multimodal reasoning in videos.
• It employs a group-based non-linear evaluation strategy that penalizes inconsistent answers and rewards coherent multi-step reasoning.
• Constructed with rigorous human annotation and quality control, Video-MME-v2 exposes real-world limitations in model performance.

Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding at a point when existing benchmarks are becoming increasingly saturated and a discrepancy has emerged between inflated leaderboard scores and real-world model capabilities. Its design combines a progressive tri-level hierarchy of video-comprehension skills with a group-based non-linear evaluation strategy that rewards consistency across related queries and coherence in multi-step reasoning rather than isolated per-question correctness. Constructed through a rigorously controlled human annotation pipeline involving 12 annotators, 50 independent reviewers, 3,300 human-hours, and up to 5 rounds of quality assurance, Video-MME-v2 is positioned as a demanding testbed for next-generation video MLLMs (Fu et al., 6 Apr 2026).

1. Conceptual basis and evaluation target

Video-MME-v2 was introduced to address a specific benchmark pathology: existing video-understanding evaluations were becoming increasingly saturated, while their headline scores no longer tracked robust real-world competence. The benchmark therefore targets not only accuracy, but also robustness and faithfulness in video understanding. In the benchmark’s framing, robustness concerns whether model competence persists across increasingly difficult forms of video comprehension, while faithfulness concerns whether correct answers are supported by valid reasoning rather than fragmented or guess-based correctness (Fu et al., 6 Apr 2026).

Two design choices organize this objective. First, the benchmark uses a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension, moving from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Second, it replaces conventional per-question scoring with a group-based non-linear evaluation that penalizes inconsistent or logically incoherent response patterns. This combination is meant to expose failure modes that conventional average accuracy can obscure, particularly in settings where a model answers some easy sub-questions correctly without demonstrating stable underlying competence (Fu et al., 6 Apr 2026).

A central implication of this design is that Video-MME-v2 is not merely a larger multiple-choice benchmark. It is structured to diagnose the dependence of high-level reasoning on lower-level perception and temporal grounding. The benchmark’s reported analyses explicitly identify a hierarchical bottleneck in which errors in visual information aggregation and temporal modeling propagate upward and limit complex reasoning performance (Fu et al., 6 Apr 2026).

2. Progressive tri-level hierarchy

The benchmark’s task ontology is organized as a three-level “difficulty ladder,” with each level targeting a distinct core skill in video comprehension (Fu et al., 6 Apr 2026).

At Level 1: Visual Information Aggregation, tasks require identification and aggregation of visual, and simple audio, cues at one or more timestamps, without involving temporal ordering. This level contains three sub-domains and 12 leaf categories in total: Visual Recognition, which includes objects, scenes, attributes, and OCR; Cross-Modal Consistency, which concerns aligning tone or mood and semantic consistency between vision and audio; and Basic Counting & Numerical Calculation, which includes static counting across frames, reading numbers, and simple arithmetic (Fu et al., 6 Apr 2026).

At Level 2: Temporal Dynamics, the benchmark adds the requirement to understand how events change over time. This level comprises three sub-levels and 15 leaf categories: Action & Motion Analysis, including fine-grained action recognition, motion counting, and trajectory inference; Sequential Ordering, including object-appearance or event-sequence ordering; and Attribute/Scene Change & Causal Reasoning, including appearance and disappearance, attribute changes, scene transformations, cause–effect, and future-prediction questions (Fu et al., 6 Apr 2026).

At Level 3: Complex Multimodal Reasoning, the benchmark simulates real-world, multi-hop inference and professional-level cognition. This level also contains 15 leaf categories, grouped under Narrative Understanding, Social Dynamics, and Physical-World Reasoning. The examples listed include plot twists, metaphors, cinematic devices, individual intent and emotion, dyadic interactions, group behavior, object persistence under occlusion, 3D spatial relations, and counterfactual or counterintuitive physics (Fu et al., 6 Apr 2026).

Tasks are evenly distributed across levels, and the paper reports a complete taxonomy with 12+ subcategories and 30+ concrete task types in its appendix. This structure makes the benchmark diagnostically useful: low-level perception, temporal grounding, and higher-order reasoning are evaluated as related but distinct strata rather than as a single undifferentiated notion of “video QA” (Fu et al., 6 Apr 2026).

3. Construction pipeline and quality assurance

Video-MME-v2 was constructed through a human-in-the-loop pipeline designed to maximize dataset quality and reduce contamination, ambiguity, and spurious solvability. The reported human effort involved 12 full-time annotators who drafted videos, questions, distractors, and answer keys, and 50 independent reviewers who were never involved in annotation and instead performed quality-control checks. The total annotation effort is reported as approximately 3,300 hours (Fu et al., 6 Apr 2026).

The pipeline applies up to five layers of scrutiny. The first is manual video curation, which includes recency-oriented selection, view-count filtering, and manual decontamination. The second is frontier-model stress testing, exemplified by a Gemini-3-Pro text-only pass used to filter language-solvable questions. The third consists of three rounds of cross-review among annotators to eliminate ambiguities and strengthen distractors. The fourth is independent blind testing by the 50 reviewers, who answer every video-QA pair to compute a human upper bound. The fifth is a closed-loop Correction→Re-verification process in which any revised question is re-tested by a text-only baseline and independent reviewers (Fu et al., 6 Apr 2026).

The resulting benchmark contains the following core inventory:

Item	Value
Videos	800
Questions	3,200 multiple-choice questions
Per-video question count	4
Answer options per question	8
Option composition	1 correct, 1 adversarial distractor, 6 strong distractors

This construction protocol is central to the benchmark’s intended role. Rather than relying on scale alone, it attempts to ensure that questions genuinely require video-grounded understanding and that the distractor structure prevents shallow elimination strategies. The benchmark therefore operationalizes “faithfulness” not only through scoring, but also through data curation (Fu et al., 6 Apr 2026).

4. Group-based non-linear evaluation

A defining feature of Video-MME-v2 is its evaluation strategy. Instead of scoring each question independently, the benchmark groups related questions into 4-question bundles and computes a non-linear group-level score. This is intended to reward either breadth of competence across related aspects of a video or depth of coherent multi-step reasoning, depending on the group type (Fu et al., 6 Apr 2026).

Per-question linear accuracy is defined as

$$\mathrm{AvgAcc} \;=\; \frac{1}{|\mathcal{Q}|}\sum_{q\in \mathcal{Q}} \mathbf{1}[\hat y_q = y_q].$$

The benchmark’s overall non-linear score is

$$\mathrm{NonLin}\;=\;\frac{1}{|\mathcal{G}|}\sum_{g\in \mathcal{G}} S(g).$$

For Consistency-Based Groups (“Capability Consistency”), if a model answers $N$ out of 4 correctly in a group $g$, the score is

$S_{\text{cons}}(g)\;=\;\bigl(\tfrac{N}{4}\bigr)^2.$

This quadratic suppression heavily penalizes getting only 1 or 2 of 4 correct, thereby reducing the credit assigned to partial or lucky success (Fu et al., 6 Apr 2026).

For Coherence-Based Groups (“Reasoning Coherence”), the questions are ordered as sequential reasoning steps $1 \to 2 \to 3 \to 4$. Let $c_i=1$ if step $i$ is correct, and define $k$ as the length of the initial correct prefix. The score is then

$S_{\text{coh}}(g)\;=\;\frac{k}{4}.$

Under this rule, once a model fails an early step, later correct answers do not restore credit. The benchmark describes this as assigning no “back-door” points for fragmentary guesses (Fu et al., 6 Apr 2026).

The rationale is explicit: per-question AvgAcc can be inflated by isolated correct guesses on easy sub-questions, while non-linear scoring demands either consistency across related aspects or a valid sequential reasoning chain. In this sense, the metric is designed as a more faithful measure of robust, grounded comprehension than standard multiple-choice accuracy (Fu et al., 6 Apr 2026).

5. Dataset composition, modalities, and coverage

The benchmark contains 800 videos and 3,200 questions, with approximately 1,067 questions assigned to each of the three levels. Its temporal and topical coverage are intended to reduce memorization and stale prior effects. More than 80% of videos were published in 2025 or later, and 40% were published after October 2025 (Fu et al., 6 Apr 2026).

Domain coverage is organized into four top-level domains: Sports/Competition, Lifestyle/Entertainment, Art/Literature, and Knowledge/Education. Within these, the benchmark reports 31 fine-grained subcategories, including cooking, travel, physics, and computer-science. The videos have an average length of 10.4 minutes; 99% are shorter than 20 minutes, and 53% are shorter than 10 minutes (Fu et al., 6 Apr 2026).

Video-MME-v2 is also explicitly multi-setting in its treatment of accompanying language and audio. It defines a “wo sub” condition using visual frames only, and a “w. sub” condition using visual frames plus aligned subtitles or ASR transcripts, or raw audio. This split is not a minor implementation detail. It is part of the benchmark’s diagnostic strategy for distinguishing video-grounded reasoning from reasoning that is anchored primarily by textual cues (Fu et al., 6 Apr 2026).

This modality design matters because several of the benchmark’s Level 1 tasks already involve audio-visual interaction, especially under Cross-Modal Consistency. A plausible implication is that Video-MME-v2 is structured not simply as a visual benchmark with optional text, but as a benchmark for controlled studies of multimodal dependence, including cases where subtitles help and cases where they can distort purely visual reasoning (Fu et al., 6 Apr 2026).

6. Reported findings, exposed failure modes, and use as a testbed

The benchmark’s headline empirical result is a substantial gap between frontier systems and humans. Under the “w. sub” setting and the Non-Lin Score, Human Expert attains 90.7 overall, with 94.8 on Level 1, 91.1 on Level 2, and 87.9 on Level 3. Gemini-3-Pro attains 49.4 overall, with 64.0 on Level 1, 50.0 on Level 2, and 40.6 on Level 3. For reference, the paper reports AvgAcc of 94.9% for humans and 66.1% for Gemini-3-Pro (Fu et al., 6 Apr 2026).

The benchmark reports a clear monotonic degradation Level 1→Level 3 for all models. Crucially, it does not attribute weak Level 3 results solely to inadequate reasoning algorithms. Instead, it emphasizes a hierarchical bottleneck: errors in Level 1 mis-aggregated visual cues propagate to Level 2 faulty temporal modeling, which in turn cripple multi-hop inference at Level 3. The paper summarizes this dependency in the claim that improving complex reasoning requires strengthening foundational perception and temporal grounding (Fu et al., 6 Apr 2026).

Video-MME-v2 also exposes a misconception about “thinking-mode” reasoning. In the w. subtitle setting, enabling Thinking can yield substantial gains; one reported example is Qwen3.5-122B improves by +5.8 points. In the wo. subtitle setting, however, some models regress when Thinking is enabled; one reported example is KimiVL-16B drops –3.3 overall and –4.0 at Level 3. The benchmark’s conclusion is that current MLLMs over-rely on language priors and have not yet learned to build robust, solely vision-based reasoning chains (Fu et al., 6 Apr 2026).

The benchmark has already been used as an evaluation bed for long-video inference methods. In GridProbe, a training-free posterior-probing framework for adaptive test-time compute in long-video VLMs, the reported Video-MME-v2 setup uses $$\mathrm{NonLin}\;=\;\frac{1}{|\mathcal{G}|}\sum_{g\in \mathcal{G}} S(g).$$0 with pool size $$\mathrm{NonLin}\;=\;\frac{1}{|\mathcal{G}|}\sum_{g\in \mathcal{G}} S(g).$$1 and Qwen3-VL-Instruct-2B as both probe and QA model. The paper reports Monolithic 2B (M=144) at 23.16 Avg Acc and 820 TFLOPs, versus GridProbe 2B ($$\mathrm{NonLin}\;=\;\frac{1}{|\mathcal{G}|}\sum_{g\in \mathcal{G}} S(g).$$2 auto) at 21.53 Avg Acc and 245 TFLOPs, a –1.63 pp change in accuracy and a 3.36× TFLOP reduction (Eltahir et al., 11 May 2026). This suggests that Video-MME-v2 is not only a benchmark for end-task accuracy, but also a stress test for whether long-video VLM methods preserve reasoning quality under aggressive compute constraints.

Taken together, these findings define the benchmark’s significance. Video-MME-v2 serves simultaneously as a benchmark for comprehensive video understanding, a diagnostic instrument for hierarchical failure analysis, and a controlled testbed for studying textual-cue dependence, reasoning coherence, and compute-efficient long-video inference (Fu et al., 6 Apr 2026).