VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges (2409.01071v1)

Abstract: Recent advancements in large-scale video-LLMs have shown significant potential for real-time planning and detailed interactions. However, their high computational demands and the scarcity of annotated datasets limit their practicality for academic researchers. In this work, we introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens within bridge layers to allow for the encoding of entire video sequences alongside historical visual data, effectively preserving semantic continuity and enhancing model performance across various tasks. This approach includes recurrent memory tokens and a SceneTilling algorithm, which segments videos into independent semantic units to preserve semantic integrity. Empirically, VideoLLaMB significantly outstrips existing video-LLMs, demonstrating a 5.5 points improvement over its competitors across three VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive results on the MVBench show that VideoLLaMB-7B achieves markedly better results than previous 7B models of same LLM. Remarkably, it maintains robust performance as PLLaVA even as video length increases up to 8 times. Besides, the frame retrieval results on our specialized Needle in a Video Haystack (NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately identifying specific frames within lengthy videos. Our SceneTilling algorithm also enables the generation of streaming video captions directly, without necessitating additional training. In terms of efficiency, VideoLLaMB, trained on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear GPU memory scaling, ensuring both high performance and cost-effectiveness, thereby setting a new foundation for long-form video-LLMs in both academic and practical applications.

• The paper introduces recurrent memory tokens that preserve historical context and maintain semantic continuity over long video sequences.
• It incorporates the SceneTilling algorithm to segment videos into semantic units, improving task performance such as VideoQA and egocentric planning.
• Empirical evaluations demonstrate that VideoLLaMB significantly outperforms baseline models in long-context video understanding.

VideoLLaMB (Long-context Video Understanding with Recurrent Memory Bridges) is a framework designed to enhance the ability of video-LLMs to process and understand long video sequences by utilizing recurrent memory tokens within bridge layers. This approach allows the encoding of extensive video content alongside historical visual data, preserving semantic continuity and improving model performance across several tasks.

The key innovation in VideoLLaMB is the integration of temporal memory tokens, which facilitate the historical context retention throughout video sequences. Additionally, the inclusion of a SceneTilling algorithm segments videos into independent semantic units. This segmentation helps maintain the semantic integrity of longer clips, making it easier for the model to manage extended contexts. Empirical evaluations demonstrate that VideoLLaMB outperforms existing models by notable margins in tasks such as VideoQA benchmarks and egocentric planning, reflecting its robust long-context video understanding capabilities (2409.01071).

For benchmarking long-term video recognition, previous works such as MeMViT (Memory-Augmented Multiscale Vision Transformer) offer relevant insights. MeMViT addresses similar challenges by employing a memory caching mechanism to reference prior context, thus extending temporal support significantly without substantial computational overhead (2201.08383). Moreover, integrating these methods with frameworks like LongBench for evaluating long-context understanding could provide comprehensive performance metrics for such models (2308.14508).

Comparatively, VideoLLaMB introduces unique features compared to models like Video-ChatGPT and Video-LLaMA, which also focus on leveraging LLMs for video comprehension. Unlike VideoLLaMB, Video-ChatGPT emphasizes conversational understanding and training a model on large datasets of video-instruction pairs to generate human-like conversations about videos (2306.05424). Video-LLaMA, on the other hand, combines visual and auditory content processing to enhance video comprehension, demonstrating capabilities in aligning multiple modalities through pre-trained encoders (2306.02858).

Ultimately, VideoLLaMB's approach of employing memory tokens and a specialized scene segmentation algorithm sets it apart as a model optimized for maintaining semantic continuity over long video sequences, while also being computationally efficient, making it a notable advancement in the domain of long-context video understanding.