ToMATO Pipeline: Temporal Reasoning Benchmark

• ToMATO Pipeline is a comprehensive framework that assesses temporal reasoning by analyzing multi-frame, order-sensitive video data in multimodal foundation models.
• It employs rigorously curated datasets and formal metrics such as multi-frame gain, frame-order sensitivity, and frame information disparity to detect model shortcomings.
• The pipeline features a reproducible, zero-shot evaluation protocol with public code release, driving progress toward bridging the human–model performance gap in visual analysis.

The ToMATO Pipeline refers to a collection of rigorously designed frameworks and algorithms, appearing under different acronyms and domains, but centrally characterized by their methodical, end-to-end structure for the assessment of temporal reasoning in multimodal foundation models (MFMs). Its primary instantiation is as the benchmarking protocol for the TOMATO dataset, targeted at quantifying genuine visual temporal reasoning by requiring model decisions to exhibit explicit dependence on multi-frame, order-sensitive visual input. The pipeline is built on formal metrics, curated data, and reproducible zero-shot evaluation protocols to serve as a testbed for state-of-the-art and next-generation multimodal models (Shangguan et al., 2024).

1. Data and Video Collection Methodology

The ToMATO Pipeline begins with comprehensive video acquisition and annotation, assembling a dataset specifically structured to diagnose temporal reasoning competence in MFMs. Sources include:

• YouTube (Creative Commons): 398 videos edited from 171 candidates, including frame reversal, speed modulation, mirroring, and concatenation to engineer explicit temporal structure.
• Existing Video QA Benchmarks: 214 videos sampled or adapted from Music-AVQA (“which instrument plays first/last?”), CLEVRER (collision counts), TGIF-QA, and Perception-Test, always re-annotated or augmented to enforce temporal requirements.
• Self-recorded and Algorithmically Simulated Videos: 805 human-acted, object-centric, and fully simulated clips capturing gesture, physical dynamics, or abstract motion, often modified to introduce counterfactual transformations and compositional temporal patterns.

In total, the assembled dataset comprises 1,417 videos (683 unique source clips): 588 human-centric, 596 object-centric, and 233 simulated, sampled at 16 uniformly spaced frames per video (frame count adjusted only if models are restricted).

2. Benchmark Construction and Temporal Task Definition

Each video is paired with a multiple-choice QA entry belonging to one of six defined temporal-reasoning tasks, all of which necessitate reasoning over frame transitions:

1. Rotation: Direction(s) of rotational movement.
2. Direction: Path or displacement vector of an agent or object.
3. Velocity & Frequency: Acceleration pattern, periodicity, or qualitative speed trend.
4. Shape & Trend: Geometry of trajectory or composite path.
5. Visual Cues: Event chronology requiring precise visual onset detection (e.g., first/last occurrence).
6. Action Count: Frequency of specified atomic events.

Task allocation and demonstration type are balanced across human, object, and simulated categories to prevent spurious statistical correlations. Annotation is administered in three stages: creation of questions and distractors, independent cross-checking by a secondary annotator, and group adjudication for error resolution. The pipeline yields 1,484 QA items, with questions averaging 11.7 tokens and 5.2 answer choices, each linked to a 4×4 grid display of frames for context.

3. Formal Metrics for Temporal Reasoning

The ToMATO pipeline enforces quantitative evaluation founded on three derived metrics, each addressing a specific class of shortcut or deficiency in temporal reasoning benchmarks:

• Multi-Frame Gain ($\kappa$): Quantifies the relative accuracy gain in using $m$ frames over a single frame,

$$\kappa = \frac{\operatorname{Acc}(m\ \text{frames})}{\operatorname{Acc}(1\ \text{frame})} - 1$$

High $\kappa$ indicates unavoidable temporal aggregation.

• Frame-Order Sensitivity ($\tau$): Detects model reliance on correct temporal ordering,

$\tau = \frac{\operatorname{Acc}(m\ \text{frames})}{\operatorname{Acc}(\text{shuffled $m$ frames})} - 1$

High $\tau$ exposes models that degrade without temporal sequence.

• Frame Information Disparity ($\rho$): Measures whether a subset of frames encodes a disproportionate share of solution-relevant information,

$$\rho = \frac{\operatorname{Acc}(\text{hand-picked 1 frame})}{\operatorname{Acc}(\text{random 1 frame})} - 1$$

Low $\rho$ ensures that shortcut solutions via "superframes" are not feasible.

These metrics are computed by repetitive scoring of models under modified input settings (single frames, shuffled frames, maximally informative frames), directly quantifying the degree to which temporal reasoning—as opposed to framewise or order-invariant pattern recognition—is required.

4. Model Evaluation Protocol and Experimental Setup

The evaluation framework comprises:

• Models Benchmarked: 7 proprietary MFMs (e.g., GPT-4o, Claude 3 Opus/Sonnet, Gemini 1.5, Reka) and 16 open-source MFMs (including Qwen2-VL 72B, Video-CCAM, Video-LLaVA, VideoLLaMA2, InternVL 2).
• Baselines: Random guessing (with/without video), frequent-answer baseline, human performance (with video and with frames only).
• Inference Pipeline: Zero-shot protocols only, preprocessing all frames at 1 FPS with model-specific preprocessing, deterministic decoding (temperature 0, top-p 1), and answer extraction via regex with fallback weighted to GPT-4o-mini.
• Aggregation: Model performance is scored as (# correct QAs) / (total QAs), both per task-type and overall. The human–model performance gap is explicitly measured (e.g., best MFM at 37.9%, human at 95.2%).

Notably, frame count ablation reveals a plateau in MFM performance beyond approximately 8 frames, indicating architectural or attention-window limitations.

5. Findings, Analysis, and Limitations

Empirical evaluation yields several robust observations:

• Sizable Human–Model Gap: The best performing open-source MFM (Qwen2-VL-72B, 37.9%) narrowly surpasses proprietary GPT-4o (37.7%), but remains 57.3 percentage points below human-level video understanding (95.2%).
• Metrics Reveal Temporal Insufficiency in Existing Benchmarks: Prior video QA benchmarks typically exhibit $m$0, $m$1, $m$2. ToMATO enforces $m$3, $m$4, $m$5, sharp transitions indicating that naive shortcuts are unavailable.
• Qualitative Error Modes: MFMs successfully recognize events within individual frames but fail to integrate these into coherent temporal analyses. Errors often reflect use of commonsense priors not grounded in video evidence, sensitivity to noise or distractor manipulations (e.g., counterfactual drop), and a lack of sequence-level compositionality unless explicit temporal encoding (e.g., M-RoPE embedding in Qwen2) is included.
• Frame Window Effects: Performance stagnates after 8 frames, suggesting insufficient attention span or temporal modeling in current MFM architectures.

6. Reproducibility and Framework Design

Reproducibility is maintained via public code release (GitHub: yale-nlp/TOMATO), complete random seed control, and a modular, script-driven workflow:

• Data processing: Python 3.10, ffmpeg, pandas/json.
• Model evaluation: OpenAI Python SDK (proprietary), HuggingFace Transformers/accelerate (open-source, NVIDIA A100).
• Modular scripts cover data collection, preprocessing, annotation UI, metrics computation, and inference aggregation.
• Provided Dockerfile and environment.yml guarantee cross-platform execution and reproducibility from data assembly through model evaluation.

The full computation pipeline—data acquisition, annotation, metric evaluation, and scoring—is documented for transparency and extension by future model developers.

7. Significance and Outlook

The ToMATO Pipeline establishes a rigorous standard for temporal reasoning evaluation in multimodal video QA, eschewing pitfalls of prior benchmarks by enforcing explicit multi-frame, order-sensitive, and non-local requirements. Its public release, coupled with detailed reproducibility infrastructure, positions it as a reference dataset and protocol for tracking progress in video-based MFM research, particularly as future architectures attempt to bridge the human–model gap and demonstrate genuine sequential scene comprehension (Shangguan et al., 2024).