VBVR-Bench: Video Reasoning Benchmark

• VBVR-Bench is a comprehensive video reasoning benchmark that uses rule-based, human-aligned scorers to assess spatiotemporal tasks.
• It employs a scalable, automated data generation pipeline producing over 1 million video clips across 200 parameterized reasoning tasks.
• The platform reveals model strengths in controllability and exposes limitations in long-horizon, complex reasoning, guiding future architectural innovations.

VBVR-Bench is a comprehensive, large-scale benchmark for video reasoning that offers a fully verifiable and interpretable framework for evaluating the spatiotemporal reasoning capabilities of modern video generation and understanding models. Developed as part of the Very Big Video Reasoning (VBVR) suite, VBVR-Bench is designed to move beyond visual fidelity metrics and instead assess the complex reasoning skills of models across hundreds of algorithmically generated tasks. It introduces principled, rule-based, human-aligned scorers and supports reproducible large-scale experimentation, laying the groundwork for systematic studies of generalization, scaling behavior, and the architectural limitations of current models (Wang et al., 23 Feb 2026).

1. Scope and Cognitive Taxonomy

VBVR-Bench is constructed atop the VBVR-Dataset, which comprises over 1 million video clips spanning 200 parameterized reasoning tasks. These tasks are structured according to a taxonomy reflecting core cognitive faculties, synthesizing classical philosophical divisions, core knowledge theory, and contemporary cognitive neuroscience. The five primary faculties underlying the task design are:

• Abstraction: Pattern completion, Raven’s matrices, sequence prediction (33 tasks).
• Knowledge: Physical intuition, chart reading, semantic identification (23 tasks).
• Perception: Object discrimination, color mixing, counting, shape identification (45 tasks).
• Spatiality: Grid navigation, maze solving, key-door puzzles (20 tasks).
• Transformation: Mental rotation, dynamic occlusion, stacking (29 tasks).

Each faculty encompasses a diverse set of tasks, enabling systematic probes into distinct reasoning mechanisms and providing fine-grained diagnostic capability regarding model performance across cognitive dimensions.

2. Dataset Structure and Generation Pipeline

The dataset curation pipeline for VBVR-Bench is a multi-stage, cloud-based process that supports continuous, large-scale, and diverse data generation, while ensuring correctness and reproducibility.

• Design and Approval: Tasks are proposed by the community and reviewed against criteria including information sufficiency, deterministic solvability, true video dependency, clarity, parametric diversity, and feasibility.
• Programmatic Generators: Each task is implemented as a subclass of a BaseGenerator interface, specifying sample generation (generate_sample(seed, params)) and validation. Parameter spaces typically span 10⁵–10⁹ unique instances per task.
• Distributed Generation and Quality Control: Automated cloud infrastructure (AWS Lambda, SQS, S3; up to 990 parallel workers) supports rapid generation and validation. Built-in retrial logic and dead-letter queues keep overall sample failure rates below 1%.

Each data sample is packaged with an initial frame, free-text prompt, and solution path video (ground-truth), supporting a spectrum of both in-domain and out-of-domain splits for robust generalization assessment.

3. Evaluation Framework and Rule-Based Scoring

VBVR-Bench’s distinguishing feature is its reliance on deterministic, rule-based scorers that are validated for human alignment. Each test task comes with automated code that inspects a model-generated video, producing a set of dimension-wise scores. The principal scoring dimensions include:

• Spatial Accuracy: End-point correctness, object positioning.
• Trajectory/Path Validity: Logical movement, obstacle adherence.
• Temporal Consistency: Frame-to-frame continuity, avoidance of visual artifacts (e.g., flicker).
• Logical/Sequential Validity: Task-dependent criteria such as order of actions or correct execution of compound operations.

A large-scale annotation campaign establishes strong agreement between these rule-based metrics and human judgments (Spearman’s ρ > 0.9). This ensures that the automated framework not only provides reproducibility and transparency but also tracks human-intuitive correctness.

The formal aggregation is as follows: For task $t$ with $N_t$ samples and scoring dimensions $d$ with weights $w_d$,

$S(t,i) = \sum_{d} w_d S_d(t, i)$

$S_M(t) = \frac{1}{N_t} \sum_{i=1}^{N_t} S(t, i)$

$$\mathrm{Score}_{\mathrm{VBVR}}(M) = \frac{1}{|T|} \sum_{t \in T} S_M(t)$$

4. Large-Scale Scaling Experiments and Emergent Generalization

The VBVR-Bench platform enables standardized scaling experiments, allowing researchers to systematically vary data and task exposure and to benchmark generalization and learning curves. For example, using a fixed-architecture 14B latent diffusion model (Wan-2.2-I2V-A14B), the scaling study reveals:

• Rapid improvements in both in-domain (ID) and out-of-domain (OOD) task performance up to 200K training samples.
• Plateauing of further gains beyond this threshold, indicating diminishing returns from increased data alone.
• Emergent transfer: OOD performance improves from 0.329 (no data) to 0.611 (200K examples), demonstrating non-trivial generalization of reasoning primitives.
• Persistent gaps: Even at maximal scale, a 15–20% performance gap remains between ID and OOD splits, and the overall model performance ($\approx 0.685$) lags substantially behind human performance ($\approx 0.97$).

Data Scale	Overall Score	ID Score	OOD Score
0 K	0.371	0.412	0.329
50 K	0.549	0.576	0.522
100 K	0.623	0.701	0.545
200 K	0.689	0.767	0.611
300 K	0.682	0.763	0.601
400 K	0.682	0.771	0.593
500 K	0.685	0.760	0.610

This pattern suggests that while scale is necessary for emergent generalization in video reasoning, it is insufficient for closing the gap to human-level competence, implying a need for architectural and procedural innovations.

5. Qualitative Analysis and Observed Limitations

Model analysis on VBVR-Bench uncovers both promising behaviors and persistent limitations:

• Controllability: VBVR-trained models maintain fidelity to input layouts and entity identities, supporting precise symbolic manipulations not matched by comparably sized or larger commercial models.
• Multi-step strategies: On compositional or pattern-completion tasks, models can exhibit internally consistent, multi-step completion policies.
• Process unfaithfulness: In procedural settings, models sometimes achieve correct results through incorrect intermediates ("shortcut" solutions), highlighting a need for better process supervision.
• Long-horizon fragility: Errors manifest in complex, extended tasks, such as duplicated entities or instability in navigation trajectories.

These observations underline that, while large data and carefully designed benchmarks foster transfer and generalization, current architectures are limited in their ability to coordinate reasoning over long spatiotemporal horizons and complex dependencies.

6. Infrastructure, Reproducibility, and Community Extension

VBVR-Bench employs a modular, version-controlled generator architecture in which each task resides in a distinct GitHub repository, tracked by commit hash. The serverless production pipeline—based on AWS Lambda, SQS, and S3—enables the generation and quality-assurance of one million samples in several hours at moderate computational cost. Automated Pydantic-based validation schemas and dead-letter queues enforce dataset integrity.

All scoring pipelines are fully deterministic, facilitating exact reproducibility across runs. The evaluation framework’s design prioritizes extensibility: new tasks, generator variants, and scoring mechanisms can be contributed by the broader community. This open structure supports expanding the diversity of video reasoning challenges and enables continual evolution of the benchmark (Wang et al., 23 Feb 2026).

7. Impact, Lessons, and Future Research Directions

VBVR-Bench establishes a rigorous standard for evaluating video reasoning by (a) supporting large-scale, multi-faculty, parameterized tasks, (b) using human-aligned rule-based evaluation, and (c) enabling systematic scaling and ablation studies. Major insights include:

• Controllability is essential for verifiable reasoning evaluation.
• Data scale alone saturates; closing the human–model performance gap requires architectural changes, such as explicit state tracking or structured reasoning modules.
• Task diversity and out-of-domain (OOD) evaluation are critical for rigorous assessment of generalization.
• Community-driven generator extension is foundational for keeping benchmarks current and comprehensive.

A plausible implication is that continued progress in video reasoning will hinge on both innovations in architecture—enabling richer stateful and hierarchical reasoning—and advances in procedural curriculum design and supervision. VBVR-Bench provides an extensible platform for such research, with all data and infrastructure made publicly available at https://video-reason.com/ (Wang et al., 23 Feb 2026).