How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? (2404.12866v2)
Abstract: The increase in parameter size of multimodal LLMs (MLLMs) introduces significant capabilities, particularly in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. This effectiveness, however, hinges on the appropriate selection of in-context examples, a process that is currently biased towards visual data, overlooking textual information. Furthermore, the area of supervised retrievers for MLLMs, crucial for optimal in-context example selection, continues to be uninvestigated. Our study offers an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Responding to this, we introduce a novel supervised MLLM-retriever MSIER that employs a neural network to select examples that enhance multimodal in-context learning efficiency. This approach is validated through extensive testing across three distinct tasks, demonstrating the method's effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method's training and pinpoint factors contributing to our model's success. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data.
- In-context examples selection for machine translation.
- Flamingo: a visual language model for few-shot learning.
- How do in-context examples affect compositional generalization?
- Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390.
- Language models are few-shot learners.
- Understanding and improving in-context learning on vision-language models.
- Murag: Multimodal retrieval-augmented generator for open question answering over images and text.
- Sinc: Self-supervised in-context learning for vision-language tasks.
- Meta-in-context learning in large language models.
- A survey on in-context learning.
- Towards robust prompts on vision-language models.
- Scaling up visual and vision-language representation learning with noisy text supervision.
- The hateful memes challenge: Detecting hate speech in multimodal memes.
- Otter: A multi-modal model with in-context instruction tuning.
- Microsoft coco: Common objects in context.
- What makes good in-context examples for gpt-3333?
- Roberta: A robustly optimized bert pretraining approach.
- Ok-vqa: A visual question answering benchmark requiring external knowledge.
- Metaicl: Learning to learn in context.
- Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP.
- Kosmos-2: Grounding multimodal large language models to the world.
- Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.
- Retrieval-augmented image captioning. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3666–3681, Dubrovnik, Croatia. Association for Computational Linguistics.
- Learning to retrieve prompts for in-context learning.
- Laion-5b: An open large-scale dataset for training next generation image-text models.
- Cider: Consensus-based image description evaluation.
- Label words are anchors: An information flow perspective for understanding in-context learning.
- Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering.
- An empirical study of gpt-3 for few-shot knowledge-based vqa.
- Retrieval-augmented multimodal language modeling.
- Compositional exemplars for in-context learning.
- A survey on multimodal large language models.
- What makes good examples for visual in-context learning?
- Mmicl: Empowering vision-language model with multi-modal in-context learning.
- Multimodal c4: An open, billion-scale corpus of images interleaved with text.