Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (2411.02937v5)

Abstract: Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal LLMs (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.

• The paper presents OmniSearch, a self-adaptive planning agent that dynamically decomposes complex multimodal questions, outperforming conventional mRAG methods.
• It addresses limitations of static retrieval queries by introducing a dynamic framework that adapts to evolving, multimodal information.
• Experimental results on the Dyn-VQA dataset highlight significant improvements in handling complex multi-hop and multi-modal questions over fixed retrieval strategies.

Insights on "Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-Adaptive Planning Agent"

The paper presents a paper on the challenges faced by current Multimodal Retrieval Augmented Generation (mRAG) systems, particularly those utilizing Multimodal LLMs (MLLMs) for visual question answering (VQA). It identifies critical issues with existing mRAG methods, proposes solutions, and evaluates the proposed approach against a new dataset.

The authors identify two significant shortcomings of conventional mRAG methodologies: Non-adaptive Retrieval Queries and Overloaded Retrieval Queries. These shortcomings limit the ability to dynamically respond to unique knowledge requirements presented by real-world questions, leading to the need for a more flexible approach to multimodal question answering.

To address these concerns, the authors introduce the Dyn-VQA dataset, a collection of 1,452 questions that require dynamic retrieval strategies. This dataset reflects more realistic scenarios compared to existing VQA datasets, focusing on three primary categories of dynamic questions: those with rapidly changing answers, those requiring multi-modal knowledge, and multi-hop questions. Unlike existing VQA datasets, where most questions can be resolved using static, text-based knowledge through a predefined two-step retrieval process, Dyn-VQA challenges models to dynamically adapt to complex, real-world information that evolves and spans multiple modalities.

The authors propose OmniSearch, the first self-adaptive planning agent designed to enhance mRAG by decomposing complex multimodal questions into sub-questions and dynamically planning retrieval actions. This innovation is inspired by human problem-solving processes and aims to dynamically adjust retrieval strategies in real-time, making it a plug-and-play component compatible with various MLLMs. OmniSearch uses retrieval tools to query knowledge dynamically from the web and images, thus providing more accurate and context-specific responses.

Experiments demonstrate the effectiveness of OmniSearch, as it outperforms existing mRAG methods in handling the complex and dynamic nature of Dyn-VQA questions. While conventional methods struggle with the rigidity of fixed retrieval processes, OmniSearch's ability to flexibly navigate and adapt to diverse retrieval scenarios shows a marked improvement in performance. The experiments reveal that even cutting-edge MLLMs, when combined with traditional mRAG techniques, fall short compared to the adaptive strategies OmniSearch employs.

The paper offers several contributions to the field of mRAG and VQA. It emphasizes the practical challenges that dynamic real-world knowledge brings to AI systems, underscores OmniSearch's role in extending current mRAG capabilities, and provides a new benchmark dataset that highlights the gaps in current systems' abilities to deal with evolving, multimodal information. Moreover, this paper suggests broader implications for the development of more robust AI systems capable of learning and adapting retrieval strategies, hinting at the potential for future advancements in AI capable of processing ever-changing information landscapes.

For future research, further exploration into refining retrieval strategies, enhancing multimodal integration, and extending the adaptability of mRAG systems, as well as improving the precision of multimodal retrieval techniques, appears to be a promising direction. These endeavors could significantly impact how AI handles real-world, dynamic information challenges and contribute to creating more versatile and reliable AI applications.