Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Visual Instruction Tuning (2304.08485v2)

Published 17 Apr 2023 in cs.CV, cs.AI, cs.CL, and cs.LG

Abstract: Instruction tuning LLMs using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Langchain. https://github.com/hwchase17/langchain, 2022.
  2. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.
  3. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
  4. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.
  5. Openflamingo, March 2023.
  6. Instruct pix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
  7. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  8. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
  9. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  10. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  11. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  12. CVinW. Computer vision in the wild. https://github.com/Computer-Vision-in-the-Wild/CVinW_Readings, 2022.
  13. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  14. Reinforce data, multiply impact: Improved model accuracy and robustness with dataset reinforcement. arXiv preprint arXiv:2303.08983, 2023.
  15. Make-a-scene: Scene-based text-to-image generation with human priors. ArXiv, abs/2203.13131, 2022.
  16. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends® in Computer Graphics and Vision, 2022.
  17. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023.
  18. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.11559, 2022.
  19. Towards learning a generic agent for vision-and-language navigation via pre-training. In CVPR, 2020.
  20. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023.
  21. Openclip. July 2021. If you use this software, please cite it as below.
  22. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022.
  23. Visual prompt tuning. In ECCV, 2022.
  24. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823, 2023.
  25. Language-driven semantic segmentation. ICLR, 2022.
  26. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 2023.
  27. ELEVATER: A benchmark and toolkit for evaluating language-augmented visual models. In NeurIPS Track on Datasets and Benchmarks, 2022.
  28. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  29. Grounded language-image pre-training. In CVPR, 2022.
  30. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093, 2023.
  31. Microsoft COCO: Common objects in context. In ECCV, 2014.
  32. Improved baselines with visual instruction tuning, 2023.
  33. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
  34. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022.
  35. OpenAI. ChatGPT. https://openai.com/blog/chatgpt/, 2023.
  36. OpenAI. Gpt-4 technical report, 2023.
  37. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  38. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023.
  39. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv: 2111.10050, 2021.
  40. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
  41. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
  42. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022.
  43. High-resolution image synthesis with latent diffusion models. CVPR, pages 10674–10685, 2022.
  44. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.11487, 2022.
  45. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
  46. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023.
  47. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  48. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  49. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  50. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022.
  51. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  52. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022.
  53. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023.
  54. Unified contrastive learning in image-text-label space. CVPR, 2022.
  55. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023.
  56. Scaling autoregressive models for content-rich text-to-image generation. ArXiv, abs/2206.10789, 2022.
  57. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
  58. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08131, 2023.
  59. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023.
  60. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  61. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
  62. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793–16803, 2022.
  63. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270, 2022.
Citations (2,976)
View on Semantic Scholar

Summary

  • The paper introduces LLaVA, a model that integrates a vision encoder with an LLM using machine-generated instruction data.
  • The approach achieves an 85.1% relative score compared to GPT-4 and sets a new state-of-the-art with a 92.53% accuracy on Science QA.
  • The work provides an open-source data pipeline and model framework that advances multimodal instruction tuning for practical applications.

Visual Instruction Tuning: A Formal Overview

The paper "Visual Instruction Tuning" authored by Liu et al. presents a methodology to enhance LLMs by connecting them with a vision encoder, culminating in an end-to-end large multimodal model named LLaVA. LLaVA stands for "Large Language and Vision Assistant," focusing on effectively interpreting and following multimodal instructions, bridging the domains of language processing and computer vision.

Abstract Summary

The authors introduce a novel approach to instruction tuning in the multimodal domain, specifically targeting visual and language understanding. They leverage machine-generated instruction-following data to enhance the zero-shot capabilities of LLMs for new tasks. LLaVA, an end-to-end trained model, incorporates a vision encoder with an LLM, resulting in superior multimodal chat abilities. Notably, LLaVA achieves an impressive 85.1% relative score compared to GPT-4 on a synthetic dataset. Furthermore, when fine-tuned on Science QA, it achieves a new state-of-the-art (SoTA) accuracy of 92.53%. The paper also outlines the release of GPT-4-generated visual instruction tuning data, the model, and associated code.

Core Motivation and Objectives

One of the primary goals in AI research is to develop general-purpose assistants capable of effectively following multimodal instructions. The current landscape of AI includes models with strong capabilities in open-world visual understanding. However, they often operate with a fixed interface, limiting interactivity and adaptability. On the other hand, LLMs like ChatGPT and GPT-4 serve as universal interfaces, representing various task instructions explicitly in language, guiding the model to the task of interest.

The paper aims to extend the instruction-tuning paradigm to the multimodal space, introducing visual instruction tuning to build a general-purpose visual assistant.

Key Contributions

The paper makes several significant contributions:

  1. Multimodal Instruction-Following Data: The authors address the scarcity of vision-language instruction-following data by presenting a data reformation pipeline. This pipeline utilizes ChatGPT and GPT-4 to convert image-text pairs into appropriate instruction-following formats.
  2. Large Multimodal Models: LLaVA is developed by connecting the visual encoder of CLIP with the language decoder Vicuna, fine-tuning it end-to-end on generated instructional vision-language data. This approach demonstrates the efficacy of using machine-generated data for multimodal model instruction-tuning.
  3. Multimodal Instruction-Following Benchmark: LLaVA-Bench is introduced, consisting of two challenging benchmarks with diverse selections of paired images, instructions, and detailed annotations.
  4. Open-Source Release: The authors release the generated multimodal instruction data, codebase, model checkpoints, and a visual chat demo.

Experimental Results

Multimodal Chatbot

The LLaVA model demonstrates significant multimodal chat capabilities, akin to those of GPT-4. The chatbot experiment reveals LLaVA's ability to understand and respond to visual inputs accurately. Quantitatively, LLaVA achieves an 85.1% relative score compared to text-only GPT-4, which uses text descriptions of visual inputs.

Science QA

For the Science QA dataset, LLaVA, when fine-tuned, achieves an accuracy of 90.92%, nearing the SoTA performance. Moreover, combining LLaVA's predictions with those from text-only GPT-4 yields a new SoTA accuracy of 92.53%. This ensemble approach highlights the complementary strengths of LLaVA and GPT-4.

Implications and Future Directions

Practical Implications

The development of LLaVA represents a significant advancement in building general-purpose visual assistants. It demonstrates how multimodal models can be fine-tuned to understand and respond to complex visual instructions. The open-source release of LLaVA paves the way for broader application and experimentation, potentially leading to more sophisticated AI-driven solutions in various domains such as healthcare, autonomous driving, and education.

Theoretical Implications

The approach of visual instruction tuning introduces a new dimension to multimodal learning, emphasizing the importance of aligning visual and language representations. The data augmentation techniques employed could be extended further to improve the robustness and generalization capabilities of multimodal models.

Future Developments

Future research could explore more sophisticated schemes to connect image and language representations. Additionally, focusing on minimizing biases and improving the interpretability of multimodal models will be imperative. Another promising direction involves scaling the pretraining datasets and model sizes, potentially leveraging larger LLaMA models for enhanced performance.

Conclusion

"Visual Instruction Tuning" by Liu et al. bridges a critical gap between visual and language understanding, leveraging machine-generated instruction-following data to create an effective multimodal assistant. Through comprehensive experiments and significant practical contributions, this paper lays the groundwork for future advancements in multimodal AI, fostering improved general-purpose assistance capabilities.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube