MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines (2409.12959v2)

Abstract: The advent of LLMs has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine. Project Page: https://mmsearch.github.io

An Expert Overview of MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

The paper "MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines" addresses a significant gap in the field of large multimodal models (LMMs) and their potential application in AI search engines. While LLMs have demonstrated impressive capabilities in textual data analysis, their multimodal extensions have yet to be adequately explored in the domain of search engines that handle both text and image queries. This paper rigorously evaluates these capabilities and proposes a systematic pipeline, MMSearch-Engine, along with a comprehensive benchmark, MMSearch, designed to assess the performance of various LMMs in multimodal search tasks.

Core Contributions

1. MMSearch-Engine: The authors introduce MMSearch-Engine, a detailed pipeline designed to empower LMMs with multimodal search capabilities. This pipeline maximizes the utilization of LMMs by integrating both visual and textual website content. The process entails three sequential stages: requery, rerank, and summarization. This structured approach enables the thorough evaluation of each step in the search process, providing insights into the specific areas where LMMs excel or fall short.
2. MMSearch Benchmark: The MMSearch benchmark is a carefully curated dataset designed to evaluate the performance of LMMs in multimodal search scenarios. The dataset includes 300 instances spanning 14 subfields, ensuring diverse and challenging queries. Notably, the dataset is curated to ensure no overlap with the training data of current LMMs, thereby testing the true search capabilities of these models rather than their mere memorization of information.

Evaluation Strategy

The paper adopts a step-wise evaluation strategy to dissect the performance of LMMs in multimodal search tasks:

• End-to-end Score (S_e2e) evaluates the final output of the search process.
• Requery Score (S_req) assesses the effectiveness of the model in reformulating the user query into a more search-engine-friendly format.
• Rerank Score (S_rer) evaluates the model's ability to select the most relevant website from a set of retrieved results.
• Summarization Score (S_sum) measures the model's proficiency in extracting and summarizing the correct answer from the selected website content.

This nuanced approach allows for a granular analysis of model performance, highlighting specific areas that need improvement.

Experimental Results

The paper conducts extensive experiments with both closed-source and open-source LMMs, including GPT-4o, Claude 3.5 Sonnet, and several state-of-the-art open-source models. Notably, the experiments reveal that:

• GPT-4o outperforms other models, achieving the best overall score and demonstrating superior zero-shot multimodal search capabilities.
• Open-source LMMs still lag behind their closed-source counterparts, indicating significant room for improvement in the open-source community.
• Perplexity Pro, a commercial AI search engine, is outperformed by MMSearch-Engine equipped with top LMMs, highlighting the effectiveness of the proposed pipeline.

Error Analysis

The authors provide a detailed error analysis, categorizing errors in the requery and summarization tasks. Key findings include:

• LMMs struggle with requery tasks, often failing to fully understand the specific requirements of querying a search engine.
• Summarization errors often stem from difficulties in aggregating information from both text and images, indicating a need for better multimodal comprehension and integration mechanisms.

Implications and Future Directions

The findings of this paper have several practical and theoretical implications:

• Practical Implications: The MMSearch-Engine provides a robust framework for developing and evaluating multimodal AI search engines, which can significantly enhance the user experience in real-world search scenarios by effectively handling both text and image queries.
• Theoretical Implications: The detailed error analysis and evaluation strategy provide valuable insights into the specific capabilities and limitations of current LMMs, guiding future research towards addressing these challenges.

Speculations on Future Developments

Looking forward, several avenues for future research and development emerge from this work:

• Enhancing Requery Capabilities: Future research could focus on improving the ability of LMMs to interpret and reformulate user queries, especially those involving both text and image inputs.
• Improving Multimodal Integration: Developing better mechanisms for integrating information from multiple modalities will be crucial for improving summarization performance.
• Scaling Test-Time Computation: Exploring the balance between model size and test-time computation, as highlighted in this paper, could lead to more efficient and effective multimodal search models.

Conclusion

In summary, this paper provides a comprehensive and rigorous evaluation of LMMs in the context of multimodal search engines. The proposed MMSearch-Engine and the MMSearch benchmark offer valuable tools for the research community, paving the way for future advancements in this promising area of AI.