CVBench: Cross-Video Reasoning Benchmark
- CVBench is a comprehensive benchmark for evaluating cross-video relational reasoning in multimodal language models using groups of independent videos.
- It organizes 1,315 videos into 250+ groups with a three-tier QA structure assessing object association, event sequencing, and complex reasoning.
- Results highlight challenges in inter-video context retention and entity disambiguation, guiding future improvements in model architecture.
CVBench is a benchmark for evaluating whether multimodal LLMs can reason across multiple, independent video streams rather than within a single clip. Introduced as “the first comprehensive benchmark designed to assess cross-video relational reasoning rigorously,” it targets settings such as multi-camera surveillance and cross-video procedural learning, where relevant evidence is distributed over distinct videos and must be synthesized at the level of entities, events, and higher-order narratives (Zhu et al., 27 Aug 2025).
1. Problem setting and benchmark identity
CVBench is explicitly designed to probe cross-video reasoning in MLLMs. Its central premise is that strong performance on single-video tasks, including video question answering, does not establish competence when evidence is split across multiple videos. The benchmark therefore focuses on multi-video relational reasoning: identifying shared entities, linking temporal or causal event chains, and integrating commonsense or domain knowledge across clips (Zhu et al., 27 Aug 2025).
The benchmark operates over groups of two to four videos that share entities, events, or high-level narratives. This grouping structure is fundamental: the task is not to answer questions about isolated clips, but to resolve relations that emerge only when several dynamic visual contexts are considered jointly. The benchmark’s construction reflects the claim that robust multimodal reasoning in real applications requires spatio-temporal integration across stream boundaries rather than clip-local recognition alone (Zhu et al., 27 Aug 2025).
A common source of confusion is the benchmark name itself. “CVBench” has also been used in other papers for an image-based, vision-centric benchmark associated with Cambrian-1, covering counting, relation, depth, and distance reasoning. That benchmark is distinct from the cross-video CVBench discussed here (Bigverdi et al., 2024).
2. Dataset composition and hierarchical task taxonomy
CVBench is built from five domain-diverse video clusters: artistic performances, sports competitions, films and television, life records, and knowledge. More concretely, these include dance recitals and live stage shows; matches and training drills; expository clips and scene montages; daily vlogs and home tours; and tutorials and documentary excerpts. Across these domains, CVBench assembles 1,315 videos into 250+ groups, each containing two to four videos (Zhu et al., 27 Aug 2025).
From these groups, expert annotators derive 1,000 multiple-choice question–answer pairs organized into three hierarchical tiers. The tiers escalate from cross-video identity and attribute matching, to temporal and causal chaining, to reasoning that requires commonsense integration or procedural transfer.
| Tier | QA pairs | Focus |
|---|---|---|
| Cross-video object association | 249 | Shared entities, identity, attributes, counting |
| Cross-video event association | 369 | Temporal links, causal links, anomaly detection, ordering |
| Cross-video complex reasoning | 382 | Commonsense integration, counterfactuals, procedural transfer, joint narratives |
The benchmark description provides representative examples. Tier 1 includes questions such as whether “the phone held in Video 1 is the same device charging in Video 3,” or how many times a recurring logo appears across all videos. Tier 2 includes tasks such as identifying which video shows a safety violation absent in the others, ordering events across videos, or retrieving scenes based on cross-video patterns. Tier 3 includes counterfactual questions, joint narrative summarization, navigation in a fused 3D space constructed from all views, and procedural knowledge transfer from one tutorial video to another (Zhu et al., 27 Aug 2025).
The choice of video clusters was intended to span a spectrum of visual complexity, from controlled studio recordings in knowledge tutorials to highly dynamic sports footage. This suggests that success on CVBench is meant to indicate domain-agnostic cross-video reasoning rather than adaptation to a narrow visual regime.
3. Formal task definition and evaluation methodology
Formally, each instance consists of a set of videos with , a question , and four candidate answers . A model produces a predicted index . Over total QA pairs, the primary metric is accuracy:
In addition to overall accuracy, the benchmark reports per-category accuracies, including categories such as cross-video anomaly detection and joint-video summarization, in order to isolate strengths and weaknesses (Zhu et al., 27 Aug 2025).
The evaluation compares two prompting protocols. In zero-shot prompting, the model is shown only the question and videos, with no additional reasoning guidance. In chain-of-thought prompting, models are encouraged via a few illustrative examples to produce intermediate reasoning steps before answering. In the reported setup, chain-of-thought runs use three exemplar QA pairs drawn uniformly from the training pool (Zhu et al., 27 Aug 2025).
The benchmarked model set includes more than ten state-of-the-art MLLMs. Closed-source systems include GPT-4o-mini, GPT-4o, Gemini-1.5-flash, and Gemini-2.0-flash. Open-source models include Qwen2.5-Omni-7B, Qwen2.5-VL-7B, LLaVA-OneVision-7B, VideoLLaMA3-7B, InternVL2.5-8B, Phi-4-Multimodal-5B, Internvideo2.5-8B, and Video-R1-7B (Zhu et al., 27 Aug 2025).
For open-source models, the evaluation uses eight uniformly sampled frames per video at resolution; for example, a four-video group yields 32 total frames. To signal video ordering, each video’s visual tokens are prepended with “The video [i].” Closed-source APIs use their native frame-sampling schemes (Zhu et al., 27 Aug 2025).
4. Empirical results and diagnosed bottlenecks
CVBench reports substantial gaps between human annotators and current MLLMs across all three tiers. Human annotators achieve 88.9% on Tier 1, 92.7% on Tier 2, and 91.3% on Tier 3. The model-level results reported in the benchmark’s main comparison are summarized below (Zhu et al., 27 Aug 2025).
| Tier | Human (%) | Gemini-2.0-flash (%) | GPT-4o (%) |
|---|---|---|---|
| Object association (Tier 1) | 88.9 | 64.5 | 66.9 |
| Event association (Tier 2) | 92.7 | 67.0 | 70.8 |
| Complex reasoning (Tier 3) | 91.3 | 69.4 | 69.1 |
Within Tier 3, the reported failure modes are especially pronounced. GPT-4o attains only approximately 60% accuracy on cross-video counterfactual reasoning, versus 100% human performance, and approximately 47% on joint-video spatial navigation. These results expose weaknesses in tasks that require hypothetical inference and synthesis of multi-view geometry across clips (Zhu et al., 27 Aug 2025).
The qualitative error analysis identifies two fundamental bottlenecks. The first is inter-video context retention: models frequently forget entities introduced in earlier videos when answering questions about later ones. In ablations where video order is shuffled, accuracy drops by approximately 4.5%; when only a single video is provided, it drops by approximately 9.3%. The second is entity disambiguation: visually similar objects or people appearing across videos are often conflated, producing errors in counting and matching tasks (Zhu et al., 27 Aug 2025).
These findings sharpen the benchmark’s intended diagnostic role. Performance deficits do not appear limited to raw perception. Rather, the benchmark isolates failures in maintaining persistent cross-video representations, aligning co-referent entities across streams, and preserving causal structure over longer horizons.
5. Architectural implications and subsequent benchmark use
The benchmark authors argue that existing MLLMs lack explicit mechanisms to persist and index visual entities across stream boundaries, maintain causal chains over extended time horizons, and ground hypothetical or procedural knowledge into multi-video contexts. To address these issues, they recommend three architectural directions: a dedicated long-term memory module that incrementally encodes and retrieves entity representations across videos; cross-video attention sparsity patterns that explicitly link co-referent objects or events; and a hybrid symbolic–neural reasoning layer for counterfactual and procedural inference (Zhu et al., 27 Aug 2025).
Subsequent work has treated CVBench as a stringent testbed for transferable multimodal reasoning rather than as a benchmark to be trained on directly. “Structured Over Scale: Learning Spatial Reasoning from Educational Video” reports that a Qwen3-VL-8B model fine-tuned with Group Relative Policy Optimization on the DoraVQA dataset reaches 86.16% on CVBench, compared with a 45.8% baseline for the same backbone, while preserving the benchmark’s zero-shot evaluation protocol (Galoaa et al., 30 Jan 2026). That work attributes the gain to the pedagogically structured “context–question–pause–answer” format of educational video, together with GRPO and unchanged Qwen backbones.
The later result is notable for two reasons. First, it indicates that CVBench is sensitive to training interventions aimed at reasoning rather than merely scaling model size. Second, it suggests that cross-video competence may benefit from supervision that emphasizes structured, temporally grounded reasoning traces. A plausible implication is that benchmark performance may depend strongly on whether a model has acquired reusable mechanisms for cross-segment tracking and compositional inference, even if those mechanisms are learned from unrelated video domains.
6. Naming ambiguity and relation to adjacent benchmarks
The designation “CVBench” is not unique in the recent literature. Several papers use “CVBench” or “CVB” to refer to a separate image-based benchmark, often linked to Cambrian-1, that evaluates counting, relation, depth, and distance reasoning on still images rather than cross-video understanding. In those works, CVBench is described as a vision-centric benchmark with 2D and 3D splits or task families such as counting and relative depth, and results are reported in exact-match or short-answer accuracy rather than multi-video multiple-choice accuracy (Bigverdi et al., 2024, Zhou et al., 7 Mar 2025, Zhang et al., 2024).
This naming overlap has methodological consequences. Reported numbers such as 56.0% for object counting with perception tokens, 59.47% for reinforcement learning on a non-SFT 2B model, or gains on “CVB-2D” and “CVB-3D” do not refer to the cross-video benchmark introduced in 2025; they refer to the earlier image-based benchmark family (Bigverdi et al., 2024, Zhou et al., 7 Mar 2025, Zhang et al., 2024). A plausible implication is that any citation of “CVBench” must specify the underlying task family—cross-video versus image-based vision-centric reasoning—to preserve comparability.
CVBench is also distinct from CVSBench, a 2026 benchmark for cross-view spatial reasoning through satellite-street pairs. CVSBench evaluates cross-view VQA, grounding, and viewpoint identification under extreme viewpoint shifts, whereas CVBench evaluates cross-video relational reasoning across multiple independent videos (Liu et al., 21 Jun 2026). The two benchmarks share an emphasis on compositional visual reasoning across distributed evidence, but they differ in modality structure, annotation scheme, and target failure modes.
The cross-video CVBench data, QA pairs, annotation scripts, and evaluation code are publicly available at the project repository listed by the benchmark authors (Zhu et al., 27 Aug 2025).
