Unifying Specialized Visual Encoders for Video Language Models (2501.01426v2)

Abstract: The recent advent of LLMs has ushered sophisticated reasoning capabilities into the realm of video through Video LLMs (VideoLLMs). However, VideoLLMs currently rely on a single vision encoder for all of their visual processing, which limits the amount and type of visual information that can be conveyed to the LLM. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge. Spatio-temporally aligning the features from each encoder allows us to tackle a wider range of open-ended and multiple-choice video understanding questions and outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks, while also having a better Video-ChatGPT score. We also improve upon SeViLA, the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV introduces minimal extra parameters and trains faster than equivalent single-encoder methods while parallelizing the visual processing. Finally, we provide qualitative evidence that MERV successfully captures domain knowledge from each of its encoders. Our results offer promising directions in utilizing multiple vision encoders for comprehensive video understanding.

• The paper introduces MERV, a novel approach that unifies multiple frozen visual encoders to improve video language models.
• It employs a feature fusion strategy using adaptive pooling and cross-attention to integrate spatial, temporal, and language cues effectively.
• Experimental evaluations demonstrate up to 3.7% accuracy improvements and faster training times compared to single-encoder models.

Unifying Specialized Visual Encoders for Video LLMs

This paper addresses a significant limitation inherent in current Video LLMs (VideoLLMs), which primarily rely on a single visual encoder to process visual information. The authors propose an innovative method, termed MERV (Multi-Encoder Representation of Videos), that integrates multiple frozen visual encoders into a unified VideoLLM framework. This approach aims to comprehensively enhance video understanding by leveraging the distinct capabilities of each encoder type.

Methodology and Contributions

The MERV model employs multiple encoders, each specializing in different aspects of visual information:

• DINOv2 serves as the spatial expert, trained on unsupervised image data for understanding object parts and semantics.
• ViViT acts as the temporal expert, focusing on interactions between frames and capturing temporal dependencies.
• SigLIP is utilized as the image-language contrastive expert, capturing joint image-text embeddings.
• LanguageBind serves as the video-language contrastive expert, trained on multi-modal datasets including videos.

A noteworthy component of MERV is its feature fusion strategy that spatio-temporally aligns and projects multi-encoder outputs into a common space. This is achieved through adaptive pooling and linear transformation, followed by cross-attention to achieve an effective fusion that retains the differentiated strengths of each encoder. This methodology allows the model to flexibly draw on the capabilities of each encoder according to the requirements of the specific video task.

Experimental Evaluation and Results

The authors conduct a thorough evaluation of MERV against state-of-the-art VideoLLMs on a wide array of video understanding benchmarks. MERV consistently outperforms prior models, with improvements in accuracy up to 3.7% on datasets such as ActivityNet-QA. Furthermore, on the complex zero-shot Perception Test, MERV shows a notable 2.2% improvement over previous leading models like SeViLA.

An important experimental result highlighted in the paper is the demonstration that MERV, even with minimal additional parameters, trains faster than models reliant on a single vision encoder. This efficiency is largely attributed to parallel visual processing, which minimizes runtime overhead from having multiple encoders.

Analysis and Insights

The paper provides an insightful analysis of the distribution of tasks that each encoder configuration excels in, showing that multiple encoders extend the spectrum of tasks beyond the capability of any individual encoder. For instance, the ViViT encoder exhibits superior performance in temporally demanding tasks, while SigLIP provides advantages in tasks requiring vision-language alignment.

A qualitative evaluation on the Something-Something v2 dataset further illustrates MERV's ability to distinguish fine-grained temporal actions in video sequences. These findings indicate that the mixture of specialized encoders provides a holistic understanding that would be unattainable with any single encoder type.

Future Implications

The integration of multiple visual encoders presents new avenues for improving VideoLLMs by allowing them to harness a greater breadth of visual expertise. Future research could explore expanding to include other modalities like 3D vision and audio to further enhance the understanding of complex video data. Additionally, investigating more fine-grained control over the fusion strategy could improve parameter efficiency and task-specific adaptability.

In conclusion, this study demonstrates a methodologically sound approach to improving video understanding in AI models, emphasizing the value of a diverse ensemble of visual encoders. By unifying these specialized encoders, MERV sets the stage for more nuanced and comprehensive video LLMs in future research.