ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models (2402.11684v2)

Abstract: Large vision-LLMs (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

Harnessing GPT-4V-Synthesized Data for Efficient Lite Vision-LLM Training

Introduction to ALLaVA

The advent of Large Vision-LLMs (LVLMs) marks a significant advancement in the AI field, enabling the integration of visual and textual data processing in a manner akin to human cognition. However, the extensive computational resources required for such models pose a challenge, especially for their deployment on edge devices. In response, this paper introduces ALLaVA, a lite version of a Vision-LLM (VLM), which leverages high-quality, synthetic data generated by GPT-4V to achieve efficiency without significant performance degradation. By adopting a synthetic data creation strategy that encompasses detailed captioning, complex reasoning, and answer generation, ALLaVA manages to bridge the performance gap typically observed between lite and standard-sized LVLMs.

Rethinking Existing LVLM Strategies

Existing approaches to align and instruct LVLMs often suffer from two significant drawbacks:

1. Alignment Issues: Traditional methods rely on coarse-grained, often noisy, caption data for image-text alignment, which limits the model's ability to accurately process visual information.
2. Simplistic Visual Instructions: The questions or instructions used to guide LVLMs' understanding and interaction with visual data tend to be overly simplistic, failing to challenge or fully utilize the model's potential for complex reasoning.

In light of these challenges, ALLaVA proposes a holistic overhaul of data strategies, emphasizing the generation of high-quality, fine-grained captions and complex, instruction-based Q&As tailored to augment both the alignment and instructional tuning of LVLMs.

ALLaVA's Methodology

ALLaVA stands out by introducing a novel data synthesis technique that effectively generates detailed captions followed by complex Q&A pairs from images. This approach addresses the critical need for high-quality training data in lite LVLMs by focusing on several areas:

• High-Quality Data Generation: Using GPT-4V’s advanced capabilities to produce elaborately detailed captions and complex reasoning instructions, broadening the model's exposure to varied, intricate visual-textual scenarios.
• Image Source Diversity: Incorporating images from Vision-FLAN and LAION datasets to ensure a wide representation of visual content, thus making the model robust across different visual domains.
• Efficient Training with Lite Models: Demonstrating the feasibility of training a less resource-intensive model without compromising on the breadth of language and vision comprehension abilities.

Experimental Validations and Observations

ALLaVA's efficacy is underscored through rigorous benchmarking against 12 diverse LVLM benchmarks, where it not only competes closely with 3B-parameter models but also showcases commendable performance in direct comparison with more sizable counterparts. The model's success across textual and multimodal tasks illustrates the potency of the employed synthetic data strategy in enhancing lite LVLMs' capabilities.

Future Avenues and Conclusion

Although ALLaVA demonstrates significant strides towards making efficient yet powerful LVLMs a reality, future research could further scale the synthetic data or explore additional dimensions of data quality and complexity. The open-sourcing of the ALLaVA model and dataset is poised to catalyze further innovations in the development of lite, efficient, and capable LVLMs.

The progression encapsulated in the ALLaVA paper illustrates a promising trajectory towards democratizing advanced AI capabilities through lite models, making sophisticated vision-language processing accessible across a broader spectrum of computational environments.