VidNum-1.4K: A Comprehensive Benchmark for Video-based Numerical Reasoning

Abstract: Video-based numerical reasoning provides a premier arena for testing whether Vision-LLMs (VLMs) truly "understand" real-world dynamics, as accurate numerical deduction necessitates a profound grasp of temporal events, object permanence, and compositional logic beyond superficial pattern matching. However, existing benchmarks are often confined to narrow domains, such as repetitive athletic motions, or treat simple counting merely as a superficial regression task, failing to assess multi-step numerical logic within the inherent complexity of real-world multimedia content. We introduce VidNum-1.4K, a comprehensive VideoQA benchmark comprising 1,379 strictly human-annotated video-question pairs designed to evaluate genuine numerical reasoning across highly diverse environments, encompassing object, action, and event quantification. The VidNum-1.4K is uniquely structured into a three-level hierarchy that evolves from direct visual perception to video-based compositional numerical reasoning, requiring models to perform arithmetic operations, comparisons, and logical deductions grounded in temporal evidence. Our evaluations across a diverse suite of state-of-the-art VLMs reveal a striking reasoning gap: while the Gemini-3.1-pro barely reaches a 60% accuracy threshold, representative open-source families struggle heavily in the 25%--45% range. These findings demonstrate that current VLMs still lack a stable "internal world model", positioning VidNum-1.4K as a demanding diagnostic testbed for the next generation of numerical video intelligence.

• The paper introduces VidNum-1.4K, a benchmark specifically designed to assess video-based numerical reasoning in vision-language models.
• The paper employs a three-level hierarchical structure with rigorous human annotation to diagnose failures in perceptual, constrained, and compositional reasoning tasks.
• The paper reveals a significant reasoning gap in current models, highlighting performance variations between direct answer and zero-shot chain-of-thought protocols.

VidNum-1.4K: A Diagnostic Benchmark for Video-based Numerical Reasoning in Vision-LLMs

Introduction

VidNum-1.4K introduces a targeted, multi-domain benchmark for video-based numerical reasoning, addressing an explicit gap in existing evaluation protocols for Vision-LLMs (VLMs). Existing benchmarks are either narrow in scope—focusing primarily on repetitive action counting—or subsume numeric reasoning into broader semantic tasks, thus failing to rigorously probe the compositional and temporal logic required for plausible “world models.” VidNum-1.4K disrupts this status quo with a meticulously human-annotated corpus of 1,379 video-question pairs, spanning highly diverse scenarios, with a hierarchical structure that analytically dissects the sources of VLM failure.

Figure 1: Distribution analysis of VidNum-1.4K illustrating topic diversity, temporal span of videos, and partitioning by question level and reasoning category.

Benchmark Architecture and Annotation Rigour

VidNum-1.4K is constructed to enforce diagnostic clarity and annotation fidelity. Each question is tied to a distinct video segment sampled from real, educational, and synthetic domains. The design follows a three-level hierarchy:

• Level 1 (Homogeneous Counting): Perceptual aggregation over a single entity class under minimal constraints; e.g., counting objects within tracking-demanding temporal windows.
• Level 2 (Constrained/Heterogenous Counting): Instance-level differentiation involving multi-attribute discrimination and re-identification across possibly discontinuous shots.
• Level 3 (Compositional Reasoning): High-order numeracy involving arithmetic, comparison, and multi-shot event isolation.

The annotation protocol leverages isolated, role-specific human teams to prevent biases and guarantee robust filtering, double-auditing, and verification. Questions are strictly constructed with “visual-description-only” policies to eliminate answerability via textual priors.

Figure 2: Top—the separated, multi-stage annotation and verification pipeline; Bottom—depicts the standardized model evaluation pipeline and prompt formatting.

Evaluation Protocol and Model Suite

VidNum-1.4K proposes a dual evaluation protocol:

• Direct Answer: Models provide immediate outputs for each MCQ.
• Zero-shot Chain-of-Thought (CoT): Models must articulate intermediate reasoning steps prior to selecting an answer.

SOTA open- and closed-source VLMs are benchmarked, encompassing InternVL, Qwen-VL, LLaVA-NeXT, and Google Gemini-3 models, with standardized frame sampling for temporal consistency.

Quantitative Findings: Reasoning Gap and Limiting Factors

Overall performance manifests a persistent and substantial reasoning gap. Gemini-3.1-pro reaches approximately 60% overall accuracy under CoT prompting, but leading open-source competitors (including large InternVL3 variants) remain clustered in the 25%–45% range. This deficit is robust across question types and particularly acute for compositional, temporally grounded reasoning.

The impact of the CoT protocol is bifurcated. For high-level reasoning (especially Level 3), CoT significantly boosts accuracy; however, for low-level perceptual questions, it introduces both gains and systematic regressions, exposing an overreliance on unstable or non-causal prediction heuristics.

Figure 3: Effects of zero-shot CoT prompting—accuracy gains at higher abstraction levels/event-centric tasks, but negative transfer for perceptual-level counting categories.

Model Scaling Dynamics

Scaling model size (considering InternVL3 from 8B to 78B parameters) results in widely divergent gains across the reasoning hierarchy. Level 3 performance substantially improves with scale, particularly under CoT, whereas Level 1 (and notably Level 2) plateaus, signaling that perceptual instance tracking and cross-shot association remain intractable for extant VLM architectures, even as parameterization increases.

Figure 4: Model scaling trends—substantial improvements for high-level compositional reasoning; minor or stagnant gains for perceptual and constrained counting as model size grows.

Discussion and Implications

The consistent inability of all surveyed VLM architectures to exhibit robust, cross-modal numerical reasoning—even when equipped with prompting strategies and large capacities—implies the absence of an internal, temporally consistent “world model.” Performance plateaus for cross-shot instance re-identification and action tracking signal that such abilities are not emergent from further scale or generic pretraining. Instead, these challenges likely demand architectural innovations, novel inductive biases (e.g., object permanence and event-level memory), or new training regimens specifically targeting temporal compositionality.

The practical implications for video-based analytics, robotics, and autonomous systems are immediate: With existing VLMs, numerically precise, temporally grounded understanding remains unreliable—posing a barrier for deployment in domains requiring quantifiable scene interpretation (e.g., sports analytics, surveillance, procedural monitoring). Theoretically, the results presented by VidNum-1.4K delineate a clear wall for current foundation models, motivating orthogonal research in multimodal episodic memory and causal, physical scene modeling.

Conclusion

VidNum-1.4K constitutes a stringent, hierarchical testbed for video-based numerical reasoning, systematically exposing the limitations of contemporary VLMs. The dataset demonstrates that scale and incremental prompting do not close the compositional reasoning gap in video understanding. This benchmark sets a new empirical standard for diagnosing and driving advances toward robust, temporally grounded video intelligence in VLMs.

(2604.03701)