TVBench: Temporal Video Benchmark

Updated 6 November 2025

TVBench is an open-source video-language benchmark engineered to isolate genuine temporal reasoning by eliminating spatial, textual, and world knowledge biases.
It employs a balanced, template-generated multiple-choice QA format that challenges models to correctly order and interpret sequential video data.
The benchmark reveals that only temporal-specialized models significantly outperform random baselines, as accuracy sharply drops when video frames are shuffled or reversed.

TVBench refers to a rigorously constructed, open-source video-language benchmark specifically designed to evaluate temporal reasoning in video LLMs (VideoLLMs) and multimodal LLMs (MLLMs). Its principal contribution is to resolve fundamental evaluation shortcomings in earlier video-language benchmarks by directly measuring and isolating genuine temporal understanding, as opposed to static spatial or text-based pattern recognition. TVBench is explicitly constructed to remove spatial, textual, and world knowledge biases, and is regarded as a critical diagnostic resource for the next generation of temporal video-language systems (Cores et al., 10 Oct 2024).

1. Motivation and Evaluation Gaps in Existing Video Benchmarks

Prevailing video-language datasets—such as MVBench and numerous VideoQA benchmarks—suffer from three primary deficiencies:

Spatial Bias: Static single frames often suffice; temporal information is not required.
Textual Bias: Poorly designed questions/answers allow text-only models (LLMs) to answer without visual data, due to overly informative natural-language cues.
World Knowledge Reliance: Many questions are answerable using background, commonsense, or domain-specific world knowledge, rather than video content.

These issues mean that state-of-the-art models can achieve strong reported performance without any actual temporal reasoning, as demonstrated empirically by the near-random performance of such models on TVBench contrasted with high scores on prior datasets under shuffle/reversal ablations (Cores et al., 10 Oct 2024). Open-ended question formats further exacerbate this problem, as automatic LLM-based evaluation introduces unreliability and confounds.

2. Benchmark Design Principles and Construction

TVBench was explicitly engineered to address these deficiencies through several core design principles:

Temporal Challenge by Construction: For each question, distractor answers are chosen such that only correct temporal reasoning (i.e., understanding sequence and order in the video) enables resolution.
Balanced and Template-Generated MCQA: Questions are generated using templates for unbiased, minimal language. Answer options are balanced such that each is correct an equal number of times. No LLM-based QA generation is used for final data.
Domain Knowledge Exclusion: All questions are operationalizable strictly from video observation; no external, commonsense, or named-entity knowledge is required or beneficial.
Multiple-Choice QA Format: MCQA allows for unambiguous, reproducible, accuracy-based evaluation, circumventing the subjectivity and unreliability of open-ended LLM-based grading.
Dataset Construction: Source videos are curated from diverse, publicly available datasets—including Perception Test, CLEVRER, STAR, MoVQA, Charades-STA, NTU RGB+D, and FunQA—spanning synthetic/real, third-/first-person, and varied scenes. Human-audited, template-guided QA pairs are generated, with candidate pool rotation to ensure dataset balance (Cores et al., 10 Oct 2024).

Task Taxonomy

TVBench QA pairs span 10 task categories that demand temporal reasoning:

Task	Description (requires...)
Action Count	Counting repeated actions (segmentation/count)
Object Count	Counting objects over time
Action Sequence	Inferring order of occurrence
Object Shuffle	Object tracking during occlusion
Scene Transition	Identifying transitions/orderings
Action Localization	Pinpointing when an action occurs
Action Antonym	Distinguishing action opposites
Unexpected Action	Localizing creative/amusing events
Egocentric Sequence	Order of actions from first-person
Moving Direction	Inferring movement trajectory

Overall, TVBench provides 2,654 QA pairs carefully generated to enforce temporal reasoning (Cores et al., 10 Oct 2024).

3. Evaluation Protocols and Metrics

Evaluation on TVBench adheres to a strict protocol to ensure only genuine temporal intelligence is measured:

Accuracy Metric: Defined as standard proportion correct, with a random baseline of 25% (for 4-way, or 50% for 2-way tasks).
Temporal Robustness Ablation: For each model, additional runs are performed with shuffled or reversed video frames. Models with true temporal comprehension should show a sharp drop to random baseline in these ablations; models leveraging only spatial or textual patterns are unaffected.
Input Variants: Models are tested with text-only inputs (LLMs only), image-only (single random frame), native video (temporally ordered), video-shuffled, and video-reversed.
Scoring: For any experiment,

$\text{Accuracy} = \frac{\text{Number of correct answers}}{\text{Total number of questions}}$

On TVBench, only temporal models show a significant gap between ordered and shuffled/reversed accuracy, a property not observed in prior datasets (Cores et al., 10 Oct 2024).

4. Empirical Findings and Model Performance

TVBench revealed that recent state-of-the-art video-LLMs (including powerful multimodal Transformers) perform only marginally better than random baselines, unless specifically trained for temporal reasoning. Salient findings include:

Model Category	TVBench Accuracy	Change on Shuffle/Reverse	MVBench Accuracy
Text-only LLMs	~33–34% (random)	No significant drop	~35–38%
Image-only (single frame)	~34–36% (random)	No significant drop	~44–48%
Standard Video-language	~33–45%	Small or no drop	~46–68%
Temporal-specialized	46–54% (best: 20.5% above random)	Sharp drop to random	~67–68%

Notably, only models such as Tarsier-34B and Gemini 1.5 Pro surpass 20% above random on TVBench; accuracy collapses when video order is disrupted, indicating reliance on temporal modeling. On MVBench and similar datasets, shuffling/reversing video has little effect, confirming the lack of temporal challenge in these tests (Cores et al., 10 Oct 2024).

TVBench thus discriminates effectively between models with and without temporal competence, a property unmatched by previous benchmarks or open-ended QA evaluations.

5. Comparative Impact and Addressing Benchmark Limitations

By construction, TVBench addresses known pitfalls and provides a critical tool for the community:

Spatial/Textual/World Knowledge Deconfounding: No prior benchmark systematically removes these biases while enforcing temporal reasoning. TVBench's balanced design, minimal template language, and candidate rotation directly achieve this goal (Cores et al., 10 Oct 2024).
Diagnostic Value: Sharp performance collapse under temporal ablations directly measures genuine temporal reasoning capacity, a critical property absent from MVBench, VideoQA, or open-ended grading approaches.
Implications for Model Development: TVBench quantitatively demonstrates that even leading models are generally deficient in temporal reasoning unless explicitly designed for the task. As such, it provides essential validation data for architecture and training innovations targeting temporal intelligence.
Integration With Related Research: TVBench is referenced and utilized as a principal diagnostic in contemporary studies examining the role of video versus image pretraining in VideoLLMs (Lydakis et al., 7 Jun 2025), the effect of fine-grained motion-comprehension datasets (Tu et al., 19 Mar 2025), and for benchmarking advances in real-time video reasoning (Xun et al., 4 May 2025). In each case, its strict requirements reveal bottlenecks and enable genuine cross-model comparison.

6. Availability, Best Practices, and Extensions

TVBench's resources, including the dataset, balanced QA pairs, evaluation code, and details of its MCQA design are openly available (Cores et al., 10 Oct 2024). Recommended practices for future benchmarking include:

Temporal Ablations as Required Protocol: Evaluate each candidate model under shuffle/reversal conditions to assert temporal dependence.
Transparent Reporting: Publish not only accuracy but also ablation and baseline (text-/image-only) figures.
Complementary Use: TVBench should complement—rather than replace—other benchmarks focusing on spatial/object reasoning, but is the gold standard for temporal intelligence.

A plausible implication is that further work should adopt TVBench-style MCQA and ablation-based metrics to measure progress in multimodal and video-LLM development.

7. Significance and Future Directions

TVBench marks a pivotal advance by providing an objective and exacting methodology for temporal reasoning evaluation in video-language systems. Its structural approach ensures that only genuinely temporal models can systematically outperform random baselines. This characteristic has motivated its immediate adoption in methodological assessments and data-efficiency studies, revealing bottlenecks in current architectures and training paradigms (Lydakis et al., 7 Jun 2025, Tu et al., 19 Mar 2025). TVBench is expected to serve as the primary diagnostic for temporal reasoning in VideoLLMs and MLLMs, informing research priorities toward models capable of true spatiotemporal understanding (Cores et al., 10 Oct 2024).

Reference:

Cores, D., Dorkenwald, M., Mucientes, M., Snoek, C.G.M., Asano, Y.M.: TVBench: Redesigning Video-Language Evaluation (Cores et al., 10 Oct 2024)