EC-Bench: Enumeration and Counting Benchmark for Ultra-Long Videos

Abstract: Counting in long videos remains a fundamental yet underexplored challenge in computer vision. Real-world recordings often span tens of minutes or longer and contain sparse, diverse events, making long-range temporal reasoning particularly difficult. However, most existing video counting benchmarks focus on short clips and evaluate only the final numerical answer, providing little insight into what should be counted or whether models consistently identify relevant instances across time. We introduce EC-Bench, a benchmark that jointly evaluates enumeration, counting, and temporal evidence grounding in long-form videos. EC-Bench contains 152 videos longer than 30 minutes and 1,699 queries paired with explicit evidence spans. Across 22 multimodal LLMs (MLLMs), the best model achieves only 29.98% accuracy on Enumeration and 23.74% on Counting, while human performance reaches 78.57% and 82.97%, respectively. Our analysis reveals strong relationships between enumeration accuracy, temporal grounding, and counting performance. These results highlight fundamental limitations of current MLLMs and establish EC-Bench as a challenging benchmark for long-form quantitative video reasoning.

• The paper introduces EC-Bench, a novel benchmark for enumerating and counting events in ultra-long videos with explicit temporal evidence.
• It features 152 untrimmed videos with 1,699 queries across six reasoning categories to rigorously test multi-hop and conditional inference.
• Experimental results reveal significant performance gaps between MLLM architectures and human-level performance in evidence-supported video analysis.

EC-Bench: A Benchmark for Long-Form Video Enumeration and Counting

Motivation and Problem Statement

Quantitative reasoning in long-form videos—specifically, enumerating and counting events or instances that are sparsely distributed over tens of minutes—represents a critical yet underexplored capability in contemporary multimodal LLMs (MLLMs). Existing benchmarks predominantly focus on short clips (usually under a few minutes) and typically restrict evaluation to final count outputs, failing to probe the quality of instance identification, the fidelity of event enumeration, and the reliability of temporal evidence grounding.

The EC-Bench framework directly addresses these deficits by proposing rigorous, multi-faceted evaluation protocols for enumeration (list all relevant instances), counting (aggregate enumerated events), and temporal support localization over ultra-long, untrimmed videos ($\geq$30min). This joint evaluation paradigm more accurately reflects the demands of real-world video understanding, where reliable aggregation requires both comprehensive instance identification and robust evidence grounding.

Benchmark Composition and Task Design

EC-Bench comprises 152 untrimmed videos (median duration: 47 minutes, all exceeding 30 minutes; mean duration ≈57 minutes) curated from diverse domains, including sports, documentaries, TV programs, news, cartoons, and academic recordings. It provides 1,699 queries meticulously annotated with temporal evidence spans, covering six orthogonal reasoning categories: Parallel, Causal, Speech/Audio, Appearance, Spatial, and Conditional tasks.

Figure 1: Task exemplars from EC-Bench, demonstrating enumeration of brands and flavors and counting their occurrences, with explicit evidence spans required for answer verification.

The benchmark uniquely supports open-ended (non-multiple-choice) enumeration and counting, penalizes omission and double-counting, and provides reference annotations for human-equivalent evaluation. The task protocols require models to output not only itemized instance lists or counts, but also video intervals as evidence for each answer.

Figure 2: Six representative reasoning categories identified and sampled for EC-Bench.

The query and evidence annotation protocols emphasize compositionality, diagnostic coverage of multi-hop and conditional tasks, and preclude answerability via world knowledge, thereby enforcing fidelity to video-grounded evidence.

Dataset Statistics and Structural Properties

Task balancing ensures uniform coverage of all inferential categories, with most enumeration tasks requiring 2–6 items and counting answers distributed between 1 and 8 but with outliers reflecting greater complexity.

Figure 3: Category and answer-type distributions for enumeration and counting queries in EC-Bench.

The video set reflects domain and genre diversity, designed to stress-test both instance-level and causal/conditional aggregation mechanisms across vastly varied contexts.

Figure 4: Left: Duration distribution showing all videos above 30 minutes; Right: genre diversity overview with representative academic, sports, documentary, TV, cartoon, and other sources.

Quantitative comparison with prior counting and video understanding benchmarks demonstrates that EC-Bench is unique in supporting multi-hop reasoning over ultra-long contexts, open-ended enumeration, explicit evidence span annotation, and systematic evaluation of temporal reasoning error modes.

Evaluation Protocols

Enumeration accuracy is measured via an LLM-as-Judge procedure, outputting precision, recall, and F1 over matched instance lists, supporting partial credit for synonym or paraphrase matches. Temporal localization is evaluated using mean temporal IoU (tIoU), computed as the intersection/union of predicted and ground-truth evidence intervals. Counting task performance is reported using both exact-match accuracy and mean absolute error (MAE), measuring not only correctness but error magnitude.

Experimental Results

Twenty-two MLLMs, spanning both open-source (LongLLaVA, VideoLLaMA3, mPLUG-Owl3, LLaVA-OneVision, LongVILA-R1, Qwen3-VL, etc.) and proprietary (GPT-4o, GPT-4.1, GPT-5, Gemini 2.0/2.5 Pro/Flash) architectures, are evaluated.

Key results:

• Best model (GPT-5): 29.98% Enumeration accuracy, 22.70% Counting accuracy.
• Human reference: 78.57% Enumeration, 82.97% Counting.
• Top open-source (Qwen3-VL 32B): 16% Enumeration, 14.27% Counting.

A clear performance gap exists between proprietary and open-source models, with significant degradation observed in long-context, multi-hop, conditional, and causal queries relative to straightforward perception or parallel-event tasks.

Figure 5: Performance evaluated across query lengths shows model robustness to question length, but persistent open-source/proprietary gaps.

Figure 6: Video length analysis indicates mild degradation as video duration increases, especially for open-source models.

Figure 7: Performance decays as number of required evidence clues increases, highlighting combinatorial aggregation failures in current models.

Further, strong positive correlation is observed between enumeration F1 and counting accuracy, affirming that reliable event enumeration is a prerequisite for robust video-level aggregation. Models with higher tIoU for temporal evidence localization also achieve higher counting and enumeration accuracy, indicating that instance identification and evidence grounding, rather than pure numerical reasoning, are the critical failure loci.

Error Analyses and Ablations

Systematic failure modes (documented with GPT-4o) include:

• Severe undercounting in dense/crowded scenes due to lack of instance individuation.
• Abstracted or hallucinated enumerations, with models outputting broad categories or world-knowledge artifacts instead of evidence-grounded lists.
• Audio-event recognition failures, with models reproducing topical rather than quantitative outputs.
• Degraded performance on counting tasks with high evidence-clue cardinality, reflecting memory and aggregation capacity limits.

Enumeration-first prompting—requiring models to explicitly list instances before aggregating—significantly boosts counting accuracy and reduces MAE. However, naive frame sampling augmentation results in only marginal improvements, with saturation and eventual noise/redundancy impact as context windows expand. Multimodal ablations (audio- or video-only) confirm that optimal performance requires both modalities.

Figure 8: Explicit enumeration-first prompting robustly improves counting accuracy for both open and proprietary models.

Practical and Theoretical Implications

EC-Bench reveals foundational limitations in contemporary MLLM architectures with respect to ultra-long temporal reasoning, evidence aggregation, and faithful instance tracking—capabilities that are not adequately trained by current datasets or architectural paradigms optimized for short video or image-level tasks.

The low model–human performance ratio, coupled with sharp declines on combinatorially complex or evidence-sparse queries, underscores the need for fundamentally new representations or architectures capable of maintaining persistent memory, supporting robust entity and event tracking over hour-level contexts, and grounding numerical inferences in explicit evidence retrieval.

Practically, EC-Bench provides an indispensable diagnostic for future system and benchmark development targeting real-world video analytics, compliance, and multi-modal quantification scenarios (e.g., surveillance, sports analytics, long-form broadcasting).

Future Directions

Further progress will likely require:

• Explicit temporal memory mechanisms and tracking modules integrated into MLLMs.
• Novel training objectives or datasets emphasizing compositional evidence aggregation, multi-hop inference, and evidence span supervision.
• Architectures capable of dynamically attending to or indexing sparse evidence over long contexts, rather than fixed sliding-window or dense sampling techniques.

As ultra-long-form multimodal reasoning moves toward practical deployment, benchmarks such as EC-Bench will become critical in measuring progress and revealing the limits of emerging MLLM technology.

Conclusion

EC-Bench offers a robust, diagnostic evaluation suite targeting the hardest aspects of long-form video understanding: enumeration, counting, and evidence grounding. Experimental results establish the large headroom between current MLLMs and human performance, particularly in temporal composition and evidence aggregation. The analyses presented in EC-Bench will inform the next generation of multimodal models and serve as a driver for innovations in persistent memory, temporal relational inference, and scalable entity tracking (2603.29943).