MIVC: Multiple Instance Visual Component for Visual-Language Models

Abstract: Vision-LLMs have been widely explored across a wide range of tasks and achieve satisfactory performance. However, it's under-explored how to consolidate entity understanding through a varying number of images and to align it with the pre-trained LLMs for generative tasks. In this paper, we propose MIVC, a general multiple instance visual component to bridge the gap between various image inputs with off-the-shelf vision-LLMs by aggregating visual representations in a permutation-invariant fashion through a neural network. We show that MIVC could be plugged into the visual-LLMs to improve the model performance consistently on visual question answering, classification and captioning tasks on a public available e-commerce dataset with multiple images per product. Furthermore, we show that the component provides insight into the contribution of each image to the downstream tasks.