VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (2410.05160v3)

Abstract: Embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering. Recently, there has been a surge of interest in developing universal text embedding models that can generalize across tasks (e.g., MTEB). However, progress in learning universal multimodal embedding models has been relatively slow despite its importance and practicality. In this work, we aim to explore the potential for building universal embeddings capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training and 16 evaluation datasets covering both in-distribution and out-of-distribution tasks, and (2) VLM2Vec (Vision-LLM -> Vector), a contrastive training framework that converts any state-of-the-art vision-LLM into an embedding model via training on MMEB. Unlike previous models such as CLIP and BLIP, which encodes text or images independently without any task instruction, VLM2Vec can process any combination of images and text to generate a fixed-dimensional vector based on task instructions. We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split. Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in MMEB. We show that VLMs are secretly strong embedding models.

• The paper introduces the MMEB benchmark and VLM2Vec framework, achieving an average improvement of 17.3 points over existing models.
• The paper employs contrastive training with a standard InfoNCE loss to fuse vision and language modalities into fixed-dimensional embeddings.
• The paper demonstrates that VLM2Vec outperforms models like CLIP and BLIP with an 11.6-point gain on out-of-distribution datasets, underscoring its scalability.

VLM2Vec: Training Vision-LLMs for Massive Multimodal Embedding Tasks

The research paper titled "VLM2Vec: Training Vision-LLMs for Massive Multimodal Embedding Tasks" presents a novel approach for developing universal multimodal embedding models. This work addresses the relatively slow progress in the field compared to text embeddings and proposes two main contributions: the Massive Multimodal Embedding Benchmark (MMEB) and the VLM2Vec framework.

Key Contributions and Methodology

1. Massive Multimodal Embedding Benchmark (MMEB):
   • MMEB is introduced as a comprehensive benchmark encompassing 36 datasets across four meta-tasks: classification, visual question answering (VQA), retrieval, and visual grounding.
   • The structure includes 20 in-distribution datasets for training and 16 out-of-distribution datasets for evaluation. This setup is designed to assess the generalization capacity of embedding models.
   • All tasks are reframed as ranking problems, providing a standardized evaluation framework across varied multimodal tasks.
2. VLM2Vec Framework:
   • VLM2Vec transforms any state-of-the-art vision-LLM into a fixed-dimensional embedding model via contrastive training on the MMEB data.
   • Using Phi-3.5-V as the backbone, VLM2Vec provides a deep integration of vision and language features within a transformer architecture.
   • The framework's contrastive training utilizes a standard InfoNCE loss function to optimize multimodal embeddings. It emphasizes instruction-following capabilities, enhancing generalizability across diverse tasks.
   • GradCache is employed during training to enable large batch sizes, crucial for improving performance by increasing the number of random negatives.

Experimental Evaluation

The experimental results of the VLM2Vec framework demonstrate significant improvements over existing models like CLIP, BLIP, and UniIR. Key findings include:

• VLM2Vec achieves an average improvement of 17.3 points across all 36 datasets in MMEB and an 11.6-point increase on 16 out-of-distribution datasets during zero-shot evaluation.
• This performance evidences the model's capability to generalize and handle various combinations of text and image modalities effectively.
• The paper further explores the impact of different training parameters, illustrating that larger batch sizes and structured instructions greatly enhance performance.

Implications and Future Directions

The introduction of MMEB and the VLM2Vec framework presents notable theoretical and practical implications:

• Theoretical Implications:
  • The research advances understanding of multimodal embeddings by proposing methodologies for universal modeling across text and visual modalities.
  • The development of MMEB sets a new standard for evaluation, encouraging further exploration into multimodal representation learning.
• Practical Implications:
  • The capability of the VLM2Vec model to generalize across unseen tasks offers practical applications in domains requiring efficient handling of diverse multimodal data, such as autonomous systems and cross-modal retrieval tasks.

The paper opens several avenues for future work, especially in refining the integration of vision-LLMs to enhance fine-grained understanding and reasoning capabilities. Continuation of this line of research can potentially lead to more robust deployment of such models in real-world scenarios, aligning closer to human-like understanding across multimodal inputs.

In conclusion, VLM2Vec presents a fundamental step forward in the field of multimodal embeddings, setting the stage for future advancements in AI-driven understanding and processing of multimodal data. The research underscores the critical importance of comprehensive benchmarks and well-integrated model architectures in achieving significant improvements in machine-based perception and interaction.