EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone (2307.05463v2)

Abstract: Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks. However, existing egocentric VLP frameworks utilize separate video and language encoders and learn task-specific cross-modal information only during fine-tuning, limiting the development of a unified system. In this work, we introduce the second generation of egocentric video-language pre-training (EgoVLPv2), a significant improvement from the previous generation, by incorporating cross-modal fusion directly into the video and language backbones. EgoVLPv2 learns strong video-text representation during pre-training and reuses the cross-modal attention modules to support different downstream tasks in a flexible and efficient manner, reducing fine-tuning costs. Moreover, our proposed fusion in the backbone strategy is more lightweight and compute-efficient than stacking additional fusion-specific layers. Extensive experiments on a wide range of VL tasks demonstrate the effectiveness of EgoVLPv2 by achieving consistent state-of-the-art performance over strong baselines across all downstream. Our project page can be found at https://shramanpramanick.github.io/EgoVLPv2/.

Overview of EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone

The paper presents EgoVLPv2, a system designed for the pre-training of video-LLMs with a specific focus on egocentric video data. This work stands out by incorporating a cross-modal fusion strategy directly within the backbone of video and language encoders, which marks a significant shift from previous approaches that relied on separate encoders or added fusion layers on top of dual encoders. This method allows EgoVLPv2 to learn robust video-text representations during the pre-training phase itself, thereby enhancing its capability to handle various video-language (VL) tasks whilst minimizing fine-tuning costs.

Background and Motivation

Egocentric VLP has gained traction due to its potential in handling diverse vision and language tasks, which is crucial for applications requiring an understanding of personal, first-person perspectives in videos. Traditional frameworks, however, heavily depend on task-specific learning during fine-tuning phases, which restricts their generalization and increases computational overheads. To address these limitations, the authors propose EgoVLPv2, which aggregates cross-modal information directly within its architecture, making it more efficient and versatile across multiple downstream tasks.

Methodology

The backbone's fusion strategy, central to EgoVLPv2, is built on integrating cross-modal attention mechanisms within existing video (TimeSformer) and language (RoBERTa) backbones. This integration avoids the introduction of additional parameters typically associated with stacked fusion layers and allows the model to seamlessly switch between and execute a blend of dual and fusion encoding tasks. Moreover, it maintains computational efficiency and reduces the model size and resource requirements relative to existing methods.

During pre-training, the model optimizes for three key objectives: Egocentric Noise Contrastive Estimation (EgoNCE), Masked LLMing (MLM), and Video-Text Matching (VTM), allowing it to generalize better to unseen tasks with minimal adjustments.

Experimental Results

The evaluation across several egocentric datasets demonstrates that EgoVLPv2 consistently reaches state-of-the-art performance benchmarks. For instance, it showcases significant improvements in egocentric video understanding tasks such as video-text retrieval, video grounding, and video summarization tasks. In particular, the authors highlight a notable improvement in intra-video multiple-choice questions where the task involves distinguishing between visually similar video sequences.

The results underscore the improved cross-modal representations inherently captured by the EgoVLPv2 framework due to the fusion in the backbone strategy. This strength translates to enhanced zero-shot and fine-tuning capabilities across diverse tasks ranging from video retrieval to activity recognition in both constrained and open-world settings.

Implications and Future Directions

The implications of EgoVLPv2 are significant for AI applications involving personal assistant devices and wearable technology that require adaptive and context-aware interaction capabilities. The authors acknowledge the limitations in handling very fine-grained actions and propose that future iterations could explore higher-resolution inputs and richer contextual cues that leverage multi-sensory data.

From a theoretical standpoint, the integration of fusion strategies directly within the model backbone represents a paradigm shift that could influence future VL research, particularly in optimizing for both computational efficiency and task flexibility.

Overall, the work is a noteworthy contribution to the ongoing development of egocentric AI models, with the potential to redefine how video-language tasks are approached in both academic and industry settings.