Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding (2306.17107v2)

Published 29 Jun 2023 in cs.CV and cs.CL

Abstract: Instruction tuning unlocks the superior capability of LLMs (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Training language models to follow instructions with human feedback, 2022.
  2. Scaling instruction-finetuned language models, 2022.
  3. Visual instruction tuning, 2023a.
  4. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.
  5. Chunyuan Li. Large multimodal models: Notes on cvpr 2023 tutorial. ArXiv, abs/2306.14895, 2023.
  6. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
  7. Learning transferable visual models from natural language supervision, 2021.
  8. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts, 2021.
  9. Microsoft coco: Common objects in context, 2015.
  10. On the hidden mystery of ocr in large multimodal models, 2023b.
  11. Llama-adapter v2: Parameter-efficient visual instruction model, 2023.
  12. OpenAI. Gpt-4 technical report, 2023.
  13. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
  14. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, 2022a.
  15. Self-instruct: Aligning language models with self-generated instructions, 2022b.
  16. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  17. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  18. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023.
  19. Llama: Open and efficient foundation language models, 2023.
  20. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023c.
  21. Large language models are not fair evaluators, 2023.
  22. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  23. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023a.
  24. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2023.
  25. Audiogpt: Understanding and generating speech, music, sound, and talking head. ArXiv, abs/2304.12995, 2023.
  26. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023b.
  27. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
  28. Llama-adapter: Efficient fine-tuning of language models with zero-init attention, 2023c.
  29. mplug-owl: Modularization empowers large language models with multimodality, 2023.
  30. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  31. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019.
  32. Cream: Visually-situated natural language understanding with contrastive reading model and frozen large language models, 2023.
  33. Dit: Self-supervised pre-training for document image transformer. Proceedings of the 30th ACM International Conference on Multimedia, Oct 2022. doi: 10.1145/3503161.3547911. URL http://dx.doi.org/10.1145/3503161.3547911.
  34. Evaluation of deep convolutional nets for document image classification and retrieval, 2015.
  35. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023b.
  36. Openflamingo, March 2023. URL https://doi.org/10.5281/zenodo.7733589.
  37. Language models are few-shot learners, 2020.
  38. Promptcap: Prompt-guided task-aware image captioning, 2022.
  39. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022.
  40. Multimodal chain-of-thought reasoning in language models, 2023d.
  41. Chameleon: Plug-and-play compositional reasoning with large language models. ArXiv, abs/2304.09842, 2023.
  42. Clippo: Image-and-language understanding from pixels only, 2022.
  43. Icdar 2019 competition on scene text visual question answering. 2019 International Conference on Document Analysis and Recognition (ICDAR), Sep 2019. doi: 10.1109/icdar.2019.00251. URL http://dx.doi.org/10.1109/ICDAR.2019.00251.
  44. Towards vqa models that can read. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cvpr.2019.00851. URL http://dx.doi.org/10.1109/CVPR.2019.00851.
  45. Docvqa: A dataset for vqa on document images, 2020.
  46. Doremi: Optimizing data mixtures speeds up language model pretraining, 2023.
  47. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
  48. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. doi: 10.1109/CVPR.2009.5206848.
  49. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github.com/openimages, 2017.
  50. Judging a book by its cover, 2016.
Citations (182)
View on Semantic Scholar

Summary

  • The paper presents LLaVAR, which significantly improves text-rich image comprehension using OCR-enhanced datasets and a two-stage training process.
  • The model leverages a CLIP-ViT visual encoder with a Vicuna-language decoder to achieve up to 20% higher accuracy on VQA benchmarks like TextVQA and OCR-VQA.
  • The approach highlights the impact of high-resolution inputs and extensive data from 422K images for advancing visual instruction tuning in real-world applications.

Enhanced Visual Instruction Tuning for Text-Rich Image Understanding: An Overview of LLaVAR

The paper "LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding" presents advancements in the field of visual instruction tuning by addressing the limitations of current models in comprehending textual elements within images. Instruction tuning has significantly improved the utility of LLMs in human interaction tasks. These models, when augmented with visual encoders, possess the potential for comprehensive human-agent interaction based on visual inputs. However, their efficacy diminishes when tasked with dissecting and understanding the textual intricacies present within images - an area crucial for enriched visual comprehension.

Methodology and Data Collection

This paper introduces LLaVAR, a model designed to augment the visual instruction tuning capability of its predecessors by focusing on text-rich images. The authors embark on enhancing the current visual instruction tuning pipeline by accumulating a substantial dataset from the LAION repository, composed of 422K text-rich images such as movie posters and book covers. These images serve as a foundation to collect textual data using Optical Character Recognition (OCR) tools. This dataset aids in overcoming the operational challenges posed by existing models that predominantly train on natural images, which lack embedded textual information.

Moreover, the authors employ GPT-4 to process the gathered OCR data and image captions, generating 16K question-answer conversational pairs tailored for text-rich images. These pairs provide high-quality, instruction-following examples crucial for the model training phases.

Model Architecture and Training Process

LLaVAR utilizes the CLIP-ViT architecture in conjunction with a Vicuna-language decoder. The model undergoes a two-stage training process:

  1. Pre-training Stage: This involves aligning visual features with a language decoder using a trainable projection matrix. Here, the integration of both the newly collected noisy data and existing pre-training datasets lays the groundwork for feature alignment without finetuning the decoder.
  2. Fine-tuning Stage: During this phase, the model incorporates high-quality instruction-following pairs. Both the feature projection matrix and the decoder undergo training to refine question-answering capabilities.

Results and Implications

LLaVAR demonstrates substantial improvement over the original LLaVA and other models across four text-based Visual Question Answering (VQA) datasets, such as ST-VQA, OCR-VQA, TextVQA, and DocVQA, with observable accuracy improvements by up to 20%. The model's high-resolution capability, marked by upgrades from 2242224^2 to 3362336^2, further underscores its enhanced proficiency in capturing and interpreting small textual details—a field traditionally challenging for standard models due to resolution constraints.

The strength of LLaVAR lies in its strategic data augmentation and instruction-following training, enhancing the encoding and decoding robustness through OCR and high-resolution inputs. This improvement is crucial in real-world applications, where combination of textual and visual information processing plays a pivotal role—spanning areas from navigating digital content to autonomous vehicle navigation systems that rely on traffic signs.

Future Directions

The progression reflected in LLaVAR paves the way for exploring even higher resolution capabilities and more sophisticated visual encoders. Future endeavors might benefit from expanding datasets further or implementing domain reweighting strategies to maximize data utility. Moreover, enhancing computation efficiency in both high-resolution and multimodal settings remains a pivotal direction for future research endeavors.

Conclusion

By focusing on augmenting the text recognition capabilities of visual instruction models through substantial datasets and strategic model adaptations, LLaVAR represents a meaningful stride in visual language processing. This paper not only highlights the current limitations and avenues for advancement but also sets a definitive blueprint for future AI models underpinned by improved visual-textual comprehension in various computational domains.