Papers
Topics
Authors
Recent
Search
2000 character limit reached

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Published 21 Nov 2023 in cs.CV | (2311.12793v2)

Abstract: In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data. To address this bottleneck, we introduce the ShareGPT4V dataset, a pioneering large-scale resource featuring 1.2 million highly descriptive captions, which surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. Specifically, ShareGPT4V originates from a curated 100K high-quality captions collected from advanced GPT4-Vision and has been expanded to 1.2M with a superb caption model trained on this subset. ShareGPT4V first demonstrates its effectiveness for the Supervised Fine-Tuning (SFT) phase, by substituting an equivalent quantity of detailed captions in existing SFT datasets with a subset of our high-quality captions, significantly enhancing the LMMs like LLaVA-7B, LLaVA-1.5-13B, and Qwen-VL-Chat-7B on the MME and MMBench benchmarks, with respective gains of 222.8/22.0/22.3 and 2.7/1.3/1.5. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM based on a simple architecture that has remarkable performance across a majority of the multi-modal benchmarks. This project is available at https://ShareGPT4V.github.io to serve as a pivotal resource for advancing the LMMs community.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Sharegpt. https://sharegpt.com/, 2023.
  2. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023a.
  3. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023b.
  4. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023a.
  6. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023b.
  7. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  8. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
  9. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  10. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  11. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  12. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360, 2021.
  13. Improving clip training with language rewrites. arXiv preprint arXiv:2305.20088, 2023.
  14. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023.
  15. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023.
  16. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108, 2023.
  17. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  18. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018.
  19. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  20. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
  21. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  22. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73, 2017.
  23. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXiv:2310.07699, 2023.
  24. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
  25. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023b.
  26. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022a.
  27. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023c.
  28. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022b.
  29. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  30. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
  31. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023b.
  32. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023c.
  33. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023d.
  34. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022.
  35. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv preprint arXiv:2305.15023, 2023.
  36. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  37. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019.
  38. Improving multimodal datasets with image captioning. arXiv preprint arXiv:2307.10350, 2023.
  39. OpenAI. Chatgpt. https://chat.openai.com/, 2023a.
  40. OpenAI. Gpt-4v(ision) system card. 2023b.
  41. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
  42. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  43. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
  44. Improving language understanding by generative pre-training. 2018.
  45. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  46. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  47. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.00855, 2015.
  48. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  49. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146–162. Springer, 2022.
  50. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  51. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
  52. InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
  53. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  54. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  55. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023.
  56. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023.
  57. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
  58. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023.
  59. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067–36080, 2022.
  60. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112, 2023a.
  61. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023b.
  62. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Citations (408)
View on Semantic Scholar

Summary

  • The paper introduces the ShareGPT4V dataset with 1.2M detailed captions to significantly enhance modality alignment in multi-modal models.
  • It employs GPT4-Vision and Share-Captioner to generate rich, descriptive captions incorporating world knowledge and spatial relationships.
  • Integrating ShareGPT4V in pre-training and fine-tuning phases notably improved benchmarks for models like LLaVA-7B and Qwen-VL-Chat-7B.

An Expert Overview of "ShareGPT4V: Improving Large Multi-Modal Models with Better Captions"

The paper "ShareGPT4V: Improving Large Multi-Modal Models with Better Captions" by Lin Chen et al. provides a comprehensive examination of the current limitations in large multi-modal models (LMMs) and addresses these challenges by introducing the ShareGPT4V dataset. This dataset consists of a substantial compilation of 1.2 million meticulously curated captions designed to enhance modality alignment between visual and textual data in LMMs.

Motivation and Contribution

In the landscape of LMMs, modality alignment is paramount, yet it is often hampered by the limited quality of existing image-text datasets. The authors argue that most of the datasets prioritize the main subjects with simple captions, thereby reducing the richness and granularity necessary for effective alignment. To counter this, ShareGPT4V introduces high-caliber captions enriched with detailed descriptions that incorporate world knowledge, spatial relationships, object properties, and aesthetic assessments.

The key contribution of the paper is twofold:

  1. Establishment of the ShareGPT4V Dataset: This dataset features 100K captions generated using the sophisticated GPT4-Vision model and extends to 1.2M captions through a specialized caption model named Share-Captioner.
  2. Enhancement of Pre-Training and Supervised Fine-Tuning Phases: In rigorous tests, replacing existing supervised fine-tuning datasets with ShareGPT4V captions significantly boosted the performance of several LMMs across various benchmarks. Furthermore, incorporating ShareGPT4V into pre-training phases resulted in the development of the ShareGPT4V-7B model, which exhibits outstanding results across multiple benchmarks.

Numerical Results

The empirical evaluation underscores the substantial impact of the ShareGPT4V dataset. For example, integrating these high-quality captions into models such as LLaVA-7B and Qwen-VL-Chat-7B led to notable performance improvements on key benchmarks like MME and MMBench, with gains of 222.8/22.0/22.3 and 2.7/1.3/1.5, respectively. ShareGPT4V-7B, despite its streamlined architecture and a parameter size of 7B, consistently outperforms existing models across 11 benchmarks, such as LCS and CQ-Bench, indicating the dataset's efficacy in enhancing modality alignment.

Implications and Future Directions

The advancements presented in this paper imply a paradigm shift in developing and utilizing multi-modal datasets for LMMs. By demonstrating the advantages of high-quality captions, the paper sets a precedent for how future LMM architectures and datasets could be structured. The utilization of highly descriptive and content-aware captions suggests that less emphasis could be placed on enlarging model parameters if the quality of training data is prioritized.

Future directions may explore further optimization of dataset curation processes and extension of this high-quality captioning approach into other languages and cultural contexts to enhance global applicability. Furthermore, the integration of ShareGPT4V-like datasets with new architectures designed to leverage such detailed data would likely present additional gains in LMM performance.

Conclusion

The work on ShareGPT4V underscores the importance of data quality in achieving optimal modality alignment in LMMs. By offering a robust dataset and showcasing its significant impact on model performance, this research propels the development of LMMs towards more nuanced and accurate multi-modal interactions. The community is encouraged to leverage the ShareGPT4V dataset for further research and development, as it provides a vital stepping stone towards more refined and capable LMM applications.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

GitHub

  1. ShareGPT4V 

Tweets

Sign up for free to view the 1 tweet with 248 likes about this paper.

Sign Up for Free