A Very Big Video Reasoning Suite

Abstract: Rapid progress in video models has largely focused on visual quality, leaving their reasoning capabilities underexplored. Video reasoning grounds intelligence in spatiotemporally consistent visual environments that go beyond what text can naturally capture, enabling intuitive reasoning over spatiotemporal structure such as continuity, interaction, and causality. However, systematically studying video reasoning and its scaling behavior is hindered by the lack of large-scale training data. To address this gap, we introduce the Very Big Video Reasoning (VBVR) Dataset, an unprecedentedly large-scale resource spanning 200 curated reasoning tasks following a principled taxonomy and over one million video clips, approximately three orders of magnitude larger than existing datasets. We further present VBVR-Bench, a verifiable evaluation framework that moves beyond model-based judging by incorporating rule-based, human-aligned scorers, enabling reproducible and interpretable diagnosis of video reasoning capabilities. Leveraging the VBVR suite, we conduct one of the first large-scale scaling studies of video reasoning and observe early signs of emergent generalization to unseen reasoning tasks. Together, VBVR lays a foundation for the next stage of research in generalizable video reasoning. The data, benchmark toolkit, and models are publicly available at https://video-reason.com/ .

• The paper introduces a systematic video reasoning benchmark with 200 diverse, parameterized tasks spanning five cognitive faculties.
• The methodology employs a cloud-based, deterministic pipeline to generate over 1M videos and 2M images for reproducible evaluation.
• Empirical results reveal consistent in-domain and out-of-domain gains while highlighting a persistent gap to human performance.

A Very Big Video Reasoning Suite: Systematic Scaling and Evaluation of Video-Based Cognitive Intelligence

Introduction and Motivation

The paper "A Very Big Video Reasoning Suite" (2602.20159) represents a substantive contribution to the rigorous and systematic study of video-based reasoning in AI models. While progress in video generation has predominantly focused on realism and fidelity, the authors explicitly delineate the study of cognitive reasoning capabilities within video models, operationalized in the context of spatiotemporally consistent environments. By grounding their evaluation suite in a comprehensive cognitive architecture and creating a benchmark that is three orders of magnitude larger than prior work, they provide the empirical resources and structured methodology necessary to push the boundaries of generalizable video reasoning.

Cognitive Architecture and Task Taxonomy

The suite’s cognitive framework is based on an explicit taxonomy synthesized from philosophical, cognitive, and neuroscientific literature, partitioning the space of video reasoning into five core faculties: Abstraction, Knowledge, Perception, Spatiality, and Transformation. This organization facilitates systematic diagnosis of reasoning competencies and their developmental interrelations.

To instantiate this taxonomy operationally, the suite comprises 200 diverse, parameterized video reasoning tasks. These tasks are implemented as generator programs that, given a random seed and configuration parameters, deterministically synthesize input sequences, prompts, and ground-truth video solutions, ensuring massive diversity while enabling full reproducibility. The taxonomy is concretized by tasks that range from multi-step symbolic manipulation, spatial navigation, and constraint satisfaction to hierarchical planning and visual perception, each mapped to a specific cognitive faculty.

Figure 1: The VBVR framework spans a large space of reasoning task families, compared at scale with prior benchmarks.

Figure 2: Representative parameterized task instances, systematically targeting distinct cognitive faculties.

Dataset Construction and Infrastructure

The VBVR dataset includes over 2 million images and one million videos distributed across 200 systematically reviewed tasks. Task design is community-driven but standardized through iterative review against six quality criteria: information sufficiency, deterministic solvability, video dependency, visual clarity, parametric diversity, and technical feasibility. Generators are implemented with an abstracted interface, ensuring modular expansion and scalable synthesis to arbitrary corpus size.

Large-scale data generation employs a cloud-based, distributed pipeline leveraging serverless infrastructures (e.g., AWS Lambda + S3), which enables reproducible, validated sample production with robust monitoring and fault tolerance.

Figure 3: System for distributed, scalable generation of parameterized video reasoning tasks.

Benchmark, Evaluation, and Human Alignment

The VBVR-Bench evaluation framework provides a deterministic, rule-based metric for each task, eschewing subjective or LLM-based assessment in favor of transparent, programmatic scoring aligned with task semantics. Metrics jointly assess stepwise spatial/temporal correctness, logical validity, constraint adherence, and task goal fulfillment. Dual evaluation splits enable measurement of both in-distribution and true out-of-distribution generalization, addressing concerns of overfitting and memorization.

A comprehensive human preference alignment study demonstrates that the rule-based evaluation correlates very strongly with human judgments ($\rho > 0.9$), establishing benchmark validity.

Scaling, Transfer, and Capability Analysis

Empirical scaling studies leverage VBVR to train and fine-tune Wan2.2-based models at different orders of data scale, culminating in the VBVR-Wan2.2 model. Results show:

• Monotonic improvements in both in-domain (ID) and out-of-domain (OOD) reasoning scores with dataset scaling, but eventual saturation and a persistent performance gap to humans even as model capacity increases.
• Fine-tuned models (VBVR-Wan2.2) achieve state-of-the-art across all considered reasoning categories, with overall scores of 0.685 (ID) and 0.610 (OOD), representing substantial improvements over baselines.
• Proprietary closed models (e.g., Sora 2) generally outperform open-source baselines on reasoning tasks, but still trail human benchmarks by significant margins.

Correlational analyses between different faculties identify statistically significant structural dependencies: For example, strong positive residualized correlation between Knowledge and Spatiality ($\rho=0.461$), and strong negative correlations between Knowledge–Perception and Abstraction–Transformation, supporting cognitive science findings regarding neural and representational modularity.

Figure 4: Residualized correlation structure between cognitive faculties, revealing nontrivial developmental coupling and trade-offs.

Qualitative Behavioral Diagnostics

Qualitative study reveals that controllability and verifiable, stable scene manipulation (as opposed to mere generative realism) are critical for effective video reasoning. Post-VBVR training, Wan2.2-derived models attain precise, instruction-following edit behaviors and coherent multi-step strategies on held-out, OOD tasks, sometimes surpassing closed commercial models on constraint-heavy scenarios. Nevertheless, failures remain for long-horizon procedural fidelity and identity preservation over extended trajectories.

Figure 5: Qualitative comparison of controllable reasoning behaviors across challenging OOD task families.

Dataset Coverage and Task Examples

VBVR's task suite is sampled to uniformly cover all aspects of its cognitive taxonomy. This diversity is critical for both systematic ablation and longitudinal benchmarking.

Figure 6: Distribution of tasks across the five core cognitive faculties in the VBVR dataset.

Typical data instances include a prompt, start frame, required final state, and a stepwise ground-truth solution, enabling supervision for both task completion and detailed trajectory reasoning.

Figure 7: Standardized data format for each video reasoning sample, showing prompt, initial state, and target solution.

Model and Domain Variability

Domain-wise performance distributions highlight significant model-dependent heterogeneity and variation in proficiency across reasoning domains.

Figure 8: Domain-wise score variation across models, highlighting reasoning specialization and overall spectrum.

Implications and Future Directions

This work establishes new infrastructure for reproducible, systematic study of video-based reasoning and transformation. The explicit cognitive taxonomy and scale of VBVR enable precise measurement of cross-domain generalization, compositionality, and representational constraints in current generative video models. The persistent model–human gap and observed generalization limitations suggest that architectural advances—e.g., explicit memory, causal abstraction mechanisms, or integrated symbolic-connectionist modules—will be required to close the gap for physical and logical video reasoning. Furthermore, the methodologically rigorous approach undertaken here provides a template for the principled development (and community extension) of foundational AI benchmarks.

Conclusion

The VBVR suite represents a critical leap in the systematic evaluation of video-based cognitive reasoning for AI. By integrating a principled cognitive architecture with large-scale, diverse, and reproducible task generation and deterministic, human-aligned evaluation, this work enables robust measurement, training, and scaling studies of generative models in spatiotemporal reasoning domains. The results and methodology are poised to inform both practical benchmarking and deeper theoretical development of future multimodal and world-model AGI systems.