Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties (2311.17041v4)

Abstract: A major reason behind the recent success of LLMs is their \textit{in-context learning} capability, which makes it possible to rapidly adapt them to downstream text-based tasks by prompting them with a small number of relevant demonstrations. While large vision-LLMs (VLMs) have recently been developed for tasks requiring both text and images, they largely lack in-context learning over visual information, especially in understanding and generating text about videos. In this work, we implement \textbf{E}mergent \textbf{I}n-context \textbf{Le}arning on \textbf{V}ideos (\eilev{}), a novel training paradigm that induces in-context learning over video and text by capturing key properties of pre-training data found by prior work to be essential for in-context learning in transformers. In our experiments, we show that \eilev-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions. Furthermore, we demonstrate that these key properties of bursty distributions, skewed marginal distributions, and dynamic meaning each contribute to varying degrees to VLMs' in-context learning capability in narrating procedural videos. Our results, analysis, and \eilev{}-trained models yield numerous insights about the emergence of in-context learning over video and text, creating a foundation for future work to optimize and scale VLMs for open-domain video understanding and reasoning. Our code and demo are available at \url{https://github.com/yukw777/EILEV}.

• The paper introduces EILEV, a novel approach that improves in-context learning in vision-language models for egocentric videos.
• It adapts VLM architectures with a frozen language model interface and targeted data sampling to mimic effective text distributional properties.
• Results demonstrate superior narrative generation and generalization across novel video scenarios while reducing dependency on massive datasets.

Efficient In-Context Learning in Vision-LLMs for Egocentric Videos

The paper "Efficient In-Context Learning in Vision-LLMs for Egocentric Videos" by Keunwoo Peter Yu et al. presents a paper on enhancing in-context learning capabilities within vision-LLMs (VLMs), specifically targeting the domain of egocentric videos. This investigation recognizes the limitations inherent in current methods that require vast collections of naturalistic data, which are both cost-prohibitive and time-consuming to acquire. The paper proposes a novel method named EILEV (Efficient In-Context Learning on Egocentric Videos) designed to elicit in-context learning capabilities in VLMs without necessitating massive datasets.

Methodological Advancements

EILEV introduces architectural adaptations and data-oriented strategies to enhance the in-context learning capabilities of VLMs, especially for egocentric video content. Key components of this approach include:

1. Context Processing: The paper adapts existing VLM architectures to facilitate the processing of interleaved video and text data. By leveraging a frozen LLM as a universal interface, this adaptation enables the model to handle context from both modalities effectively.
2. Data Sampling Strategy: A critical aspect of EILEV is the method for creating training data that possesses specific distributional characteristics known to support in-context learning. These include:
   • Clusters of verbs and nouns to reproduce bursty distributions.
   • Marginal distributions with a long tail of infrequent items to emphasize less common actions.
   • Inclusion of homonyms and synonyms to introduce ambiguity and reliance on context for disambiguation.

These strategies aim to replicate the properties observed in text-only LLMs which have demonstrated significant in-context learning abilities.

Evaluation and Results

The performance of the EILEV-trained models is evaluated against traditional models such as Kosmos-2 and Otter, which are trained on larger datasets. Notable outcomes of this evaluation include:

• Superior In-Context Learning Performance: The EILEV-trained models consistently outperform larger VLMs in generating narrative descriptions for unseen egocentric video clips at various shot levels, from one-shot to sixteen-shot scenarios.
• Generalization to Out-of-Distribution and Novel Actions: The models excel in synthesizing action narratives for not only the original dataset but also generalize effectively to novel and out-of-distribution datasets such as EPIC-KITCHENS-100. This demonstrates the model's ability to leverage in-context examples for adaptation to new, unseen tasks.
• Minimal Dependence on In-Weights Learning: Despite the training being on common actions, the variance in performance as novel, rare actions are introduced indicates reliance on contextual information rather than solely on pre-encoded knowledge.

Implications and Future Directions

The paper highlights the potential of deploying EILEV-trained VLMs in cost-restrictive environments such as interactive task-guidance systems, where the ease of adaptation and low requirement for extensive training data are advantageous. This has significant implications for embodied AI applications and real-time processing systems requiring adaptability and context understanding.

Moreover, the results advocate for more nuanced approaches to integrating in-context learning in VLMs, moving beyond mere scale and data volume. Future research could delve further into optimizing these methods for broader applications or extending them to other modalities.

In conclusion, this paper contributes to the advancement of efficient learning paradigms within the field of vision-language integration, emphasizing strategic data and architecture configurations that may reshape the future landscape of adaptive AI systems.