Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment (2208.13628v2)

Abstract: Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder. Although pretrained on four times less data, our ViCHA strategy outperforms other approaches on several downstream tasks such as Image-Text Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The code will be made publicly available here: https://github.com/mshukor/ViCHA

Efficient Vision-Language Pretraining with ViCHA

The paper introduces ViCHA, an innovative framework designed for efficient vision-language pretraining. This research tackles the escalating computational demands of scaling vision-LLMs, presenting a compelling alternative that alleviates the heavy reliance on large datasets and extensive resources.

Core Contributions

ViCHA's methodology is defined by three pivotal components:

1. Hierarchical Cross-Modal Alignment Loss: Unlike conventional strategies that employ a singular alignment loss predominantly at the final layer of encoders, ViCHA introduces a hierarchical approach. By aligning representations at multiple layers within the vision and text transformers, the method effectively captures various semantic levels. This multilevel alignment is pivotal in refining vision-language interactions, ensuring a more thorough assimilation of multimodal features.
2. Visual Concepts for Enhanced Image Encoding: The framework leverages existing foundation models like CLIP to extract image-level annotations, termed Visual Concepts (VCs). By integrating these concepts directly with visual tokens, ViCHA augments the visual representation with semantically rich contextual information. This enhancement is shown to significantly ease the burden of aligning textual and visual modalities, especially during early training stages.
3. Self-Supervised Masked Image Modeling: A self-supervised technique based on the masked image modeling paradigm enhances the visual encoder's robustness. The approach is inspired by Masked Autoencoders (MAE) and is fine-tuned to optimize the learning process from both local and global image features.

Results and Implications

Despite training on significantly less data—approximately a quarter of that used in comparable frameworks—ViCHA demonstrates superior performance across several downstream tasks, including Image-Text Retrieval, Visual Question Answering (VQA), Visual Reasoning, Visual Entailment, and Visual Grounding. This efficiency is attributed largely to the innovative training recipes and visual-semantic enrichment strategies that form the backbone of ViCHA.

The practical implications of this research are notable:

• Resource Accessibility: The framework democratizes access to high-performing vision-LLMs for academic labs with limited computational resources.
• Sustainability: By optimizing data utilization and reducing computational load, ViCHA offers a path toward more sustainable AI research and deployment practices.

Furthermore, the theoretical implications suggest a paradigm shift in model design, focusing on the quality of learning methodologies rather than sheer scale. The hierarchical alignment approach and VC integration might inspire future research to explore deeper modality interactions and innovate beyond standard pretraining schemes.

Future Developments

This research opens several avenues for future exploration. Enhanced methods for extracting and integrating visual concepts could further refine multimodal representations. Exploring alternate foundational models beyond CLIP for visual concept extraction might also offer gains in diversity and domain-specific applicability. Additionally, fine-tuning alignment strategies to capture even finer granularities of image-text relationships could further enhance model robustness and accuracy.

In conclusion, the ViCHA framework sets a new benchmark for efficient vision-language pretraining by intelligently leveraging hierarchical alignment and enriched visual concepts. Its potential to scale effectively while maintaining high performance is a landmark achievement, providing a sustainable and accessible model for future multimodal AI advancements.