Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (2403.13164v2)

Published 19 Mar 2024 in cs.LG

Abstract: LLMs famously exhibit emergent in-context learning (ICL) -- the ability to rapidly adapt to new tasks using few-shot examples provided as a prompt, without updating the model's weights. Built on top of LLMs, vision LLMs (VLLMs) have advanced significantly in areas such as recognition, reasoning, and grounding. However, investigations into \emph{multimodal ICL} have predominantly focused on few-shot visual question answering (VQA), and image captioning, which we will show neither exploit the strengths of ICL, nor test its limitations. The broader capabilities and limitations of multimodal ICL remain under-explored. In this study, we introduce a comprehensive benchmark VL-ICL Bench for multimodal in-context learning, encompassing a broad spectrum of tasks that involve both images and text as inputs and outputs, and different types of challenges, from {perception to reasoning and long context length}. We evaluate the abilities of state-of-the-art VLLMs against this benchmark suite, revealing their diverse strengths and weaknesses, and showing that even the most advanced models, such as GPT-4, find the tasks challenging. By highlighting a range of new ICL tasks, and the associated strengths and limitations of existing models, we hope that our dataset will inspire future work on enhancing the in-context learning capabilities of VLLMs, as well as inspire new applications that leverage VLLM ICL. The code and dataset are available at https://github.com/ys-zong/VL-ICL.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
  2. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023.
  3. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.
  4. Visual prompting via image inpainting. NeurIPS, 2022.
  5. Language models are few-shot learners. NeurIPS, 2020.
  6. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  7. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
  8. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023.
  9. Making llama see and draw with seed tokenizer. ICLR, 2024.
  10. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617, 2018.
  11. Measuring massive multitask language understanding. ICLR, 2021.
  12. Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi: 10.1109/TPAMI.2021.3079209.
  13. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
  14. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
  15. Generating images with multimodal language models. NeurIPS, 2023.
  16. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS, 36, 2023.
  17. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
  18. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023b.
  19. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023c.
  20. Microsoft coco: Common objects in context. In ECCV, 2014.
  21. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
  22. Visual instruction tuning. NeurIPS, 2023b.
  23. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/.
  24. Lost in the middle: How language models use long contexts. TACL, 2024b.
  25. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023c.
  26. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022.
  27. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
  28. Metaicl: Learning to learn in context. In NACL, 2022.
  29. R OpenAI. Gpt-4 technical report. arXiv, pages 2303–08774, 2023.
  30. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022.
  31. TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In CVPR, 2021.
  32. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TMLR, 2023.
  33. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023a.
  34. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023b.
  35. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
  36. Multimodal few-shot learning with frozen language models. In NeurIPS, 2021.
  37. T-NER: An all-round python library for transformer-based named entity recognition. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2021.
  38. Matching networks for one shot learning. In NeurIPS, 2016.
  39. Building a question answering test collection. In SIGIR, 2000.
  40. Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023a.
  41. Seggpt: Towards segmenting everything in context. In ICCV, 2023b.
  42. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv., 53(3), June 2020. doi: 10.1145/3386252.
  43. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
  44. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023.
  45. Can mllms perform text-to-image in-context learning? arXiv preprint arXiv:2402.01293, 2024.
  46. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112, 2023.
  47. Character-level convolutional networks for text classification. NeurIPS, 28, 2015.
  48. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915, 2023.
  49. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Yongshuo Zong (11 papers)
  2. Ondrej Bohdal (19 papers)
  3. Timothy Hospedales (101 papers)
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - ys-zong/VL-ICL (59 stars)