Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models (2403.00231v3)

Published 1 Mar 2024 in cs.CV and cs.CL

Abstract: Large vision-LLMs (LVLMs) excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains. To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions, sourced from 572K ArXiv papers spanning various scientific domains. Drawing from ArXivCap, we introduce ArXivQA, a question-answering dataset generated by prompting GPT-4V based on scientific figures. ArXivQA greatly enhances open-sourced LVLMs' mathematical reasoning capabilities, achieving a 10.4\% absolute accuracy gain on a multimodal mathematical reasoning benchmark. Furthermore, employing ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs. Evaluation results with state-of-the-art LVLMs underscore their struggle with the nuanced semantics of academic figures, while domain-specific training yields substantial performance gains. Our error analysis uncovers misinterpretations of visual context, recognition errors, and the production of overly simplified captions by current LVLMs, shedding light on future improvements.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Flamingo: a visual language model for few-shot learning. ArXiv preprint, abs/2204.14198.
  2. Openflamingo: An open-source framework for training large autoregressive vision-language models. ArXiv preprint, abs/2308.01390.
  3. Qwen technical report. arXiv preprint arXiv:2309.16609.
  4. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv preprint, abs/2308.12966.
  5. Introducing our multimodal models.
  6. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  7. Figure captioning with relation maps for reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1537–1545.
  8. Figure captioning with reasoning and sequence-level training. ArXiv preprint, abs/1906.02850.
  9. Sharegpt4v: Improving large multi-modal models with better captions. ArXiv preprint, abs/2311.12793.
  10. Pali-x: On scaling up a multilingual vision and language model. ArXiv preprint, abs/2305.18565.
  11. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  12. On the use of arxiv as a dataset. ArXiv preprint, abs/1905.00075.
  13. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv preprint, abs/2305.06500.
  14. A survey for in-context learning.
  15. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv preprint, abs/2306.13394.
  16. G-llava: Solving geometric problem with multi-modal large language model.
  17. Google. 2023. Bard.
  18. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258–3264.
  19. mplug-paperowl: Scientific diagram analysis with the multimodal large language model.
  20. ImageMagick Studio LLC. Imagemagick.
  21. DVQA: understanding data visualizations via question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5648–5656.
  22. Figureqa: An annotated figure dataset for visual reasoning. ArXiv preprint, abs/1710.07300.
  23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  24. The semantic scholar open data platform.
  25. Obelics: An open web-scale filtered dataset of interleaved image-text documents.
  26. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. ArXiv preprint, abs/2306.00890.
  27. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv preprint, abs/2301.12597.
  28. Silkie: Preference distillation for large visual language models. ArXiv preprint, abs/2312.10665.
  29. M33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPTIT: A large-scale dataset towards multi-modal multilingual instruction tuning. ArXiv preprint, abs/2306.04387.
  30. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81.
  31. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.
  32. Improved baselines with visual instruction tuning.
  33. Visual instruction tuning. ArXiv preprint, abs/2304.08485.
  34. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. ArXiv preprint, abs/2310.02255.
  35. Brielen Madureira. 2021. Flamingos and hedgehogs in the croquet-ground: Teaching evaluation of NLP systems for undergraduate students. In Proceedings of the Fifth Workshop on Teaching NLP, pages 87–91.
  36. OpenAI. 2023. Gpt-4v(ision) system card.
  37. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  38. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
  39. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763.
  40. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. ArXiv preprint, abs/2111.02114.
  41. Aligning large multimodal models with factually augmented rlhf. ArXiv preprint, abs/2309.14525.
  42. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971.
  43. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 11445–11465.
  44. The dawn of lmms: Preliminary explorations with gpt-4v (ision). ArXiv preprint, abs/2309.17421.
  45. Scicap+: A knowledge augmented dataset to study the challenges of scientific figure captioning. ArXiv preprint, abs/2306.03491.
  46. Capsfusion: Rethinking image-text data at scale. ArXiv preprint, abs/2310.20550.
  47. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. ArXiv preprint, abs/2311.16502.
  48. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
  49. Llavar: Enhanced visual instruction tuning for text-rich image understanding.
  50. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv preprint, abs/2304.10592.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Lei Li (1293 papers)
  2. Yuqi Wang (62 papers)
  3. Runxin Xu (30 papers)
  4. Peiyi Wang (48 papers)
  5. Xiachong Feng (28 papers)
  6. Lingpeng Kong (134 papers)
  7. Qi Liu (485 papers)
Citations (33)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Multimodal ArXiv
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets