How Important are Videos for Training Video LLMs? (2506.06928v1)

Abstract: Research into Video LLMs has progressed rapidly, with numerous models and benchmarks emerging in just a few years. Typically, these models are initialized with a pretrained text-only LLM and finetuned on both image- and video-caption datasets. In this paper, we present findings indicating that Video LLMs are more capable of temporal reasoning after image-only training than one would assume, and that improvements from video-specific training are surprisingly small. Specifically, we show that image-trained versions of two LLMs trained with the recent LongVU algorithm perform significantly above chance level on TVBench, a temporal reasoning benchmark. Additionally, we introduce a simple finetuning scheme involving sequences of annotated images and questions targeting temporal capabilities. This baseline results in temporal reasoning performance close to, and occasionally higher than, what is achieved by video-trained LLMs. This suggests suboptimal utilization of rich temporal features found in real video by current models. Our analysis motivates further research into the mechanisms that allow image-trained LLMs to perform temporal reasoning, as well as into the bottlenecks that render current video training schemes inefficient.

• The paper shows that robust temporal reasoning in Video LLMs is achievable through image-text pretraining, questioning the added value of video data.
• The methodology demonstrates that pseudo-video finetuning yields temporal reasoning performance comparable to traditional video training, offering a cost-effective alternative.
• The study implies that optimizing text and image pretraining can reduce dependency on costly video data while maintaining effective temporal feature capture.

Insights into the Role of Videos in Training Video LLMs

The paper "How Important are Videos for Training Video LLMs?" provides a comprehensive evaluation of the role videos play in training Video LLMs. This research is motivated by the observation that Video LLMs, which have typically been trained on a combination of text, images, and video datasets, might already possess temporal reasoning capabilities derived from image pretraining alone. The authors question the necessity and efficacy of video datasets in fostering the temporal reasoning capabilities of Video LLMs.

Key Findings

The paper reveals several intriguing results:

1. Temporal Reasoning from Image Training: Video LLMs, specifically those trained with the LongVU algorithm, exhibit significant temporal reasoning capabilities even when trained solely on image datasets. This result suggests that substantial temporal understanding may already be embedded within these models through their image-text pretraining phase.
2. Performance using Pseudo Videos: The paper introduces a finetuning scheme that uses sequences of annotated images to simulate video dynamics. This pseudo-video approach yielded temporal reasoning performance comparable to—and sometimes exceeding—that of video-trained LLMs. The findings imply that the utilization of real video datasets might be suboptimal in current training architectures.
3. Limited Impact of Video Training: The improvement gains from actual video-specific training were surprisingly limited, indicating potential inefficiencies in leveraging the temporal features afforded by real video datasets.

Implications

The results of this research offer both practical and theoretical implications for the development of Video LLMs:

• Optimization of Training Data Usage: The findings suggest a more optimized use of text and image datasets during the pretraining phase might reduce reliance on video datasets for temporal reasoning tasks.
• Architectural Advancements: There is room to explore whether advancements in architecture could better capture the richness of temporal features in videos.
• Cost-Effective Training: Given the computational expenses associated with video data, optimizing training processes by leveraging pseudo videos could lead to cost-effective strategies.

Speculations on Future Developments

In light of these findings, several future research directions could be pursued:

• Mechanisms of Temporal Reasoning: Investigating the underlying mechanisms that enable image-trained LLMs to perform temporal reasoning might provide insights into how these models process sequential data.
• Efficiency in Video Utilization: Research into why current training schemes might inefficiently leverage video data could lead to the development of algorithms or architectures better suited for capturing video-specific features.
• Benchmarks and Evaluation: The creation of additional benchmarks that target specific temporal reasoning capabilities could help quantify improvements and identify gaps in current models.

In summary, the paper challenges existing paradigms regarding the training of Video LLMs by questioning the extent to which video datasets contribute to temporal reasoning capabilities. These findings open the door to re-evaluating how models are trained and how different data modalities can be efficiently and effectively utilized to advance the capabilities of AI systems in video understanding tasks.