Visual Instruction Tuning (2304.08485v2)

Abstract: Instruction tuning LLMs using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

• The paper presents LLaVA, which merges a CLIP visual encoder with Vicuna LLM using a lightweight projection matrix to align visual and textual data.
• It employs a two-stage training approach where multimodal instruction data is generated via GPT-4 and ChatGPT to enhance model performance.
• The model achieves state-of-the-art results with over 90% accuracy on ScienceQA, demonstrating superior visual understanding compared to previous multimodal models.

Visual Instruction Tuning

Introduction

The paper "Visual Instruction Tuning" (2304.08485) addresses the challenge of aligning large multimodal models (LMMs) with human visual and linguistic instructions to perform various tasks. It introduces LLaVA, an innovative model integrating a vision encoder and a LLM for enhanced understanding across modalities.

Architecture and Model Design

LLaVA employs a robust architecture where the CLIP visual encoder is coupled with Vicuna, a leading LLM known for its instruction-following capabilities (Figure 1).

Figure 1: LLaVA network architecture.

The architecture involves a meticulous alignment process that adapts visual features using a lightweight projection matrix for integration into the token space of the LLM. This design choice facilitates efficient connection and grounding of visual input within language-based processing, a crucial step for achieving seamless multimodal interaction.

Data Generation and Training

A significant contribution of the paper is its methodology for generating multimodal instruction-following data. Utilizing GPT-4 and ChatGPT, the authors reformulate existing image-text pairs into instructive datasets that enrich the model's learning landscape with complex reasoning and conversational nuances. The training process is bifurcated into two stages:

1. Pre-training Alignment: This stage ensures the compatible alignment of visual features with word embeddings, focusing only on the projection matrix.
2. End-to-End Fine-tuning: In this phase, the model is enhanced using the comprehensive instruction-tuning dataset, allowing the LLM to embroil itself deeply in task-specific adjustments across modalities.

Experimental Evaluation

The LLaVA model was subjected to rigorous experimental scrutiny using novel benchmarks designed to test its instruction-following capabilities and generalization across unseen tasks.

Multimodal Chatbot

A chatbot application demonstrated LLaVA's aptitude for visual understanding and interactive task execution comparable to multimodal GPT-4, often outperforming other models like BLIP-2 and OpenFlamingo in terms of following human-like instructions (Table 1).

ScienceQA Benchmark

ScienceQA, an advanced reasoning dataset, provided an opportunity to assess LLaVA's precision in question-answering tasks. The model achieved an impressive 90.92% accuracy, and when combined with GPT-4, reached a state-of-the-art accuracy of 92.53%, highlighting the model's synergistic potential when supplemented with external LLMs (Figure 2).

Figure 2: LLaVA generates HTML/JS code for an interactive website based on user sketch inputs.

Implications and Future Directions

The implications of this research are profound, suggesting a trajectory towards creating universal multimodal assistants capable of nuanced understanding and interaction. The method of visual instruction tuning presented in this paper could serve as a blueprint for future LMMs aiming for enhanced alignment and task diversity.

Further research is encouraged in exploring more sophisticated model architectures and broader datasets to bolster the robustness and adaptability of future models. Moreover, exploring emergent behaviors and generalization capabilities of LMMs could unlock new paradigms in AI-driven problem-solving across sectors.

Conclusion

"Visual Instruction Tuning" exemplifies a pivotal advancement in multimodal AI, showcasing a scalable approach to integrating visual and linguistic modalities systematically. LLaVA's framework and outcomes not only reflect the intricacies of sophisticated multimodal learning but also pave the way for significant advancements in developing AI models that are more adept at understanding and executing complex human-driven tasks.