TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models (2410.10818v2)

Abstract: Understanding fine-grained temporal dynamics is crucial for multimodal video comprehension and generation. Due to the lack of fine-grained temporal annotations, existing video benchmarks mostly resemble static image benchmarks and are incompetent at evaluating models for temporal understanding. In this paper, we introduce TemporalBench, a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. TemporalBench consists of ~10K video question-answer pairs, derived from ~2K high-quality human annotations detailing the temporal dynamics in video clips. As a result, our benchmark provides a unique testbed for evaluating various temporal understanding and reasoning abilities such as action frequency, motion magnitude, event order, etc. Moreover, it enables evaluations on various tasks like both video question answering and captioning, both short and long video understanding, as well as different models such as multimodal video embedding models and text generation models. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench, demonstrating a significant gap (~30%) between humans and AI in temporal understanding. Furthermore, we notice a critical pitfall for multi-choice QA where LLMs can detect the subtle changes in negative captions and find a centralized description as a cue for its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such bias. We hope that TemporalBench can foster research on improving models' temporal reasoning capabilities. Both dataset and evaluation code will be made available.

• The paper demonstrates that state-of-the-art multimodal video models show major deficiencies in fine-grained temporal reasoning, with GPT-4o achieving only 38.5% accuracy on temporal QA tasks.
• It leverages approximately 10,000 video Q&A pairs and introduces a Multiple Binary Accuracy metric to challenge models across tasks like captioning, grounding, and varying video lengths.
• The study highlights the need for improved temporal dynamics understanding, paving the way for future advances in causal inference and contrastive learning methods in video analysis.

TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

Overview

"TemporalBench" introduces a benchmark specifically designed to evaluate the fine-grained temporal understanding capabilities of multimodal video models. This initiative addresses the deficiency in current video understanding benchmarks that largely mirror those used for static images, thereby failing to assess models effectively for temporal dynamics. Comprising approximately 10,000 video question-answer pairs derived from about 2,000 human annotations, TemporalBench emphasizes the evaluation of temporal reasoning abilities such as action frequency and event order.

Key Components and Methodology

TemporalBench is constructed with an array of tasks, including video question answering, captioning, and understanding videos of varying lengths. Several characteristics distinguish this benchmark:

• Fine-grained Action Understanding: The benchmark emphasizes intricate action details, facilitating the creation of challenging negative captions that can subtly differ from positive ones.
• Short and Long Video Evaluation: It supports evaluations on videos ranging from short clips (less than 20 seconds) to longer durations (up to 20 minutes) by concatenating clips from the same source.
• Versatility in Model Evaluation: TemporalBench can extend to tasks such as video captioning, grounding, and generation, catering to both video embedding models and generative models.

Findings and Results

Significant gaps remain between current state-of-the-art models and human performance in understanding temporal dynamics. For instance, models like GPT-4o achieved a mere 38.5% accuracy, highlighting a substantial performance gap compared to human results. The benchmark also reveals a critical limitation in multi-choice QA, where models may exploit linguistic cues rather than genuinely understanding video content. Accordingly, a Multiple Binary Accuracy (MBA) metric is proposed to mitigate this bias by converting multi-choice questions into multiple binary questions, thereby challenging models to demonstrate true understanding.

Implications and Future Directions

TemporalBench stands to influence future developments in AI by highlighting the need for enhancement in models' temporal reasoning capabilities. This benchmark could serve as a catalyst for various theoretical and practical advancements, including improvement in video spatio-temporal localization and causal inference tasks. Moreover, it provides a robust testbed for scrutinizing contrastive learning-based models and large multimodal models. The insights gleaned may propel further research to bridge the evident gap between human and model performance in temporal understanding.

Concluding Remarks

TemporalBench represents a significant step toward refining the temporal understanding of multimodal video models. By providing a comprehensive evaluation framework, it challenges current AI models and sets the stage for the continued enhancement of temporal reasoning capabilities in AI systems. The benchmark is a valuable tool for researchers dedicated to advancing the field of video understanding and modeling complex temporal sequences.