MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (2410.08182v1)

Abstract: Existing multimodal retrieval benchmarks primarily focus on evaluating whether models can retrieve and utilize external textual knowledge for question answering. However, there are scenarios where retrieving visual information is either more beneficial or easier to access than textual data. In this paper, we introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench, in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints. MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios. With MRAG-Bench, we conduct an evaluation of 10 open-source and 4 proprietary large vision-LLMs (LVLMs). Our results show that all LVLMs exhibit greater improvements when augmented with images compared to textual knowledge, confirming that MRAG-Bench is vision-centric. Additionally, we conduct extensive analysis with MRAG-Bench, which offers valuable insights into retrieval-augmented LVLMs. Notably, the top-performing model, GPT-4o, faces challenges in effectively leveraging retrieved knowledge, achieving only a 5.82% improvement with ground-truth information, in contrast to a 33.16% improvement observed in human participants. These findings highlight the importance of MRAG-Bench in encouraging the community to enhance LVLMs' ability to utilize retrieved visual knowledge more effectively.

MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

The paper "MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models" introduces a new benchmark designed to evaluate the capabilities of large vision-LLMs (LVLMs) in utilizing visually augmented knowledge. The benchmark, MRAG-Bench, aims to address scenarios where visual information retrieval is more pertinent than textual data, thereby providing a systematic assessment of vision-centric knowledge retrieval.

Core Contributions

MRAG-Bench focuses on scenarios where visual retrieval is advantageous due to the inherent benefits or accessibility of visual data compared to text. The benchmark consists of 16,130 images and 1,353 multiple-choice questions spanning nine distinct real-world scenarios. These scenarios are meticulously categorized into two main aspects:

1. Perspective Understanding: Evaluates model performance when visual entities are presented from different viewpoints or when only partial images are available. It includes four scenarios: Angle, Partial, Scope, and Occlusion.
2. Transformative Understanding: Assesses the model's ability to comprehend visual transformations, such as biological changes or physical deformations. It includes four scenarios: Temporal, Deformation, Incomplete, and Biological.

MRAG-Bench provides the first benchmark concentrating on vision-centric multimodal retrieval-augmented generation (RAG), emphasizing the retrieval of visual knowledge over text retrieval. This provides novel insights into how LVLMs can leverage externally sourced visual data to improve their reasoning and generation capabilities.

Experimental Findings

The authors evaluated 14 advanced LVLMs, including both open-source and proprietary models. The evaluation highlights significant insights:

• All models demonstrated improved performance when augmented with visually retrieved knowledge compared to textual retrieval.
• The top-performing model, GPT-4o, achieved only a 5.82% performance improvement when supplemented with ground-truth knowledge, compared to a 33.16% improvement observed in human evaluations.
• Open-source models struggled to effectively differentiate between high-quality and noisy retrieved examples, whereas proprietary models exhibited an emerging ability to discern usable visual knowledge from noise.

Analysis and Insights

The analysis revealed several critical findings:

• Visual knowledge provided greater performance enhancements for LVLMs than textual knowledge, suggesting an inherent advantage of visual RAG in scenarios defined by MRAG-Bench.
• The model's performance correlated positively with the accuracy of the retrieval mechanism, emphasizing the importance of effective multimodal retrievers.
• Although the benchmark challenges LVLMs to process an average of 20.4 ground-truth examples, the performance generally plateaued with around 10 visual examples, indicating an area for potential optimization in multimodal integration strategies.

Theoretical and Practical Implications

The findings underscore the necessity of developing LVLMs equipped to handle complex vision-centric tasks through effective integration of externally retrieved visual data. This has theoretical implications in expanding the architectural design of LVLMs for improved contextual reasoning and practical implications in deploying these models in real-world applications where visual content is abundant and accessible.

Future Directions

The paper opens several avenues for future research:

• Refinement of LVLM architectures to better utilize diverse and noisy visual datasets.
• Exploration of adaptive strategies for determining the optimal quantity of visual examples needed for effective knowledge integration.
• Expansion into broader multimodal contexts, incorporating not just images but potentially video or 3D graphics to further enrich model interactions.

In conclusion, MRAG-Bench is a significant contribution to the field of AI, presenting rigorous evaluation metrics for understanding and improving the integration of visual knowledge in LVLMs. The benchmark encourages further exploration in retrieval-augmented multimodal reasoning, ultimately advancing the capabilities of AI systems in handling vision-intensive tasks.