Lost in Time: A New Temporal Benchmark for VideoLLMs (2410.07752v3)

Abstract: LLMs have demonstrated impressive performance when integrated with vision models even enabling video understanding. However, evaluating video models presents its own unique challenges, for which several benchmarks have been proposed. In this paper, we show that the currently most used video-language benchmarks can be solved without requiring much temporal reasoning. We identified three main issues in existing datasets: (i) static information from single frames is often sufficient to solve the tasks (ii) the text of the questions and candidate answers is overly informative, allowing models to answer correctly without relying on any visual input (iii) world knowledge alone can answer many of the questions, making the benchmarks a test of knowledge replication rather than video reasoning. In addition, we found that open-ended question-answering benchmarks for video understanding suffer from similar issues while the automatic evaluation process with LLMs is unreliable, making it an unsuitable alternative. As a solution, we propose TVBench, a novel open-source video multiple-choice question-answering benchmark, and demonstrate through extensive evaluations that it requires a high level of temporal understanding. Surprisingly, we find that most recent state-of-the-art video-LLMs perform similarly to random performance on TVBench, with only a few models such as Qwen2-VL, and Tarsier clearly surpassing this baseline.

• The paper introduces TVBench, a novel benchmark that requires models to utilize temporal context rather than relying on static frames or textual shortcuts.
• It employs simplified textual cues and tasks like action sequence recognition and object motion tracking to mitigate pre-existing language biases.
• Empirical results show state-of-the-art models perform near random chance on TVBench, underscoring the benchmark's effectiveness in assessing true temporal understanding.

Evaluating Temporal Understanding in Video-LLMs: A Critical Review

The paper "TVBench: Redesigning Video-Language Evaluation" addresses a significant gap in the current evaluation protocols for video-LLMs. The authors identify several critical shortcomings inherent in existing benchmarks, notably MVBench, and propose a novel benchmark, TVBench, to more rigorously assess temporal understanding in video-LLMs.

Issues with Existing Benchmarks

Current video-language benchmarks, such as MVBench, are found to be inadequate in evaluating true temporal reasoning. The authors highlight three primary issues:

1. Static Frame Dependence: Many tasks within existing benchmarks can be solved using information from a single frame, rendering temporal evaluation ineffective.
2. Textual Bias: Overly informative textual descriptors allow models to predict answers without processing visual content, leading to answers based purely on pre-existing LLM biases.
3. World Knowledge Overreliance: Benchmarks like MVBench include questions that can be answered using general world knowledge, bypassing the need for video analysis.

TVBench: A Novel Solution

In response to these issues, the paper introduces TVBench, a temporal video-language benchmark designed to necessitate temporal reasoning. TVBench focuses on creating more challenging tasks that cannot be resolved without understanding the sequence of video frames. Key features of TVBench include:

• Temporal Challenges: Tasks are explicitly designed to require temporal context, such as action sequence recognition, object motion tracking, and unexpected action detection.
• Textual Simplification: Utilizing uniform question templates ensures that the textual component does not hint at the correct answer.
• Balanced Data: Careful structuring ensures that candidate answers do not allow models to leverage pre-existing linguistic biases.

Empirical Evaluation

The paper reports that recent state-of-the-art models falter on TVBench, performing close to random chance, as opposed to their comparatively high performance on MVBench. More advanced temporal models like Tarsier and Gemini 1.5 Pro distinguish themselves with notably better results, underscoring the necessity of temporal reasoning capabilities.

For instance, shuffling or reversing the video frames significantly reduces model accuracy on TVBench, contrasting with MVBench, where models are less affected. This stark performance disparity highlights TVBench's effectiveness in focusing on temporal understanding.

Implications for Future AI Research

TVBench serves as a more reliable mechanism for evaluating video-LLMs' temporal understanding, offering a more rigorous assessment of model capabilities. This benchmark paves the way for future developments in AI, encouraging the design of models that truly understand video content beyond static or textual cues.

The paper also suggests that improving temporal reasoning can be achieved by diversifying training datasets and enhancing model architectures to better capture dynamic content. As AI progresses, TVBench may act as an essential tool in guiding advancements and ensuring models develop comprehensive temporal comprehension.

Conclusion

The introduction of TVBench represents an essential step toward more reliable evaluations in the field of video-language understanding. By addressing the inadequacies of current benchmarks, this work provides a foundation upon which future models can be developed and assessed, ensuring progress is both meaningful and measurable.