---
title: 'MERRIN: Multimodal Search Benchmark'
url: https://www.emergentmind.com/topics/merrin
type: topic
---

# MERRIN: Multimodal Search Benchmark

Searching arXiv for the MERRIN paper and closely related benchmark/evaluation references.
{"query":"MERRIN benchmark multimodal evidence retrieval reasoning noisy web environments arXiv", "max_results": 10}
Searching for BrowseComp, which MERRIN follows for LLM-as-judge evaluation.
{"query":"BrowseComp arXiv browsecomp evaluation LLM-as-judge", "max_results": 10}
Searching for SealQA and ChartMuseum, which are listed as partial sources for adapted items in MERRIN.
{"query":"SealQA arXiv ChartMuseum arXiv multimodal QA benchmark", "max_results": 10}
MERRIN, short for **Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments**, is a **human-annotated benchmark for search-augmented agents** designed to evaluate whether an AI system can infer the needed modality, retrieve relevant multimodal evidence, and reason over noisy, heterogeneous, and often conflicting web sources. Its defining premise is that realistic web search is both underspecified and multi-hop: queries are presented in plain natural language without explicit modality cues, and correct resolution requires engagement with non-text evidence rather than text-only shortcuts. MERRIN is positioned as a benchmark for open-web search settings in which text, image, video, and table sources interact under conflict, incompleteness, and retrieval noise [2604.13418].

## 1. Motivation and problem formulation

MERRIN was introduced in response to three limitations in prior multimodal search benchmarks. First, earlier benchmarks often include **explicit modality cues** in the question, effectively telling the model whether to inspect an image, video, or other medium. Second, they tend to emphasize **text and image**, while leaving **video and audio** underexplored. Third, they do not adequately model the **noise, conflict, and incompleteness** characteristic of real web search results [2604.13418].

The benchmark therefore targets a more difficult and operationally relevant regime. Questions are intentionally written in **plain natural language** with **no explicit modality cues**, so modality selection is itself part of the task. Every question is manually verified to require **non-text evidence**, and each has a **single short, unambiguous answer**. This design makes the benchmark a test of retrieval, modality inference, and reasoning simultaneously, rather than a conventional fact lookup task.

A plausible implication is that MERRIN is less a benchmark of static multimodal QA than of **search-conditioned evidential decision making**. The benchmark does not merely ask whether a model can read a modality once retrieved; it asks whether the model can navigate the open web, identify the useful medium, and ground a final answer under source-level ambiguity.

## 2. Task structure, modalities, and annotation schema

MERRIN organizes its questions along two annotation axes. The first is **reasoning type**. A question may require **multi-hop reasoning**, meaning that the answer depends on combining information across multiple steps or sources; **multimodal conflict resolution**, meaning that the agent must reconcile conflicting evidence returned by the web; or both. The second axis is the **role of the non-text modality**. Non-text evidence may be the **answer source**, where the final answer is directly contained in that modality, or a **reasoning component**, where it supplies an intermediate fact needed to derive the answer. Some questions instantiate both roles [2604.13418].

The benchmark covers four source types: **Text**, **Image**, **Video**, and **Table**. At the same time, the benchmark emphasizes that **video and audio are especially underexplored** in prior work, and that in MERRIN, video sources incorporate **both visual and audio modalities**. This yields a slightly broader operational modality space than a simple four-way source taxonomy might suggest.

MERRIN also identifies three core failure modes. A **retrieval error** occurs when the agent chooses the correct modality but the wrong source. A **modality error** occurs when the agent uses the wrong modality, often defaulting to text. A **reasoning error** occurs when the right source is retrieved but the model grounds or combines the evidence incorrectly. This taxonomy is significant because it separates failures of search policy from failures of evidence use. It also undercuts the common simplification that multimodal benchmarking can be reduced to a single end-to-end accuracy number.

## 3. Dataset construction and quality control

MERRIN contains **162 questions** in total. Of these, **120** were created from scratch, **37** were adapted from **SealQA**, and **5** were adapted from **ChartMuseum**. For adapted items, the original QA pair was used as one hop and then extended with additional evidence to form a new multi-hop question. Annotators record the **ground-truth answer**, a **step-by-step reasoning explanation**, source **URLs**, **source type** for each resource, whether non-text evidence is the **answer source** or a **reasoning component**, whether the question is **multi-hop** and/or involves **multimodal conflict**, and the **origin** of the question [2604.13418].

The quality-control pipeline is unusually strict. A second annotator checks answer correctness, clarity, difficulty, and whether non-text evidence is truly required. More importantly, non-text necessity is verified through a **two-pass protocol**: a **Standard text-only search pass** and an **Adversarial search pass**, where the known answer is inserted into the query to test whether a text-only shortcut exists. A question is retained only if at least one sub-question survives both passes without becoming solvable from text alone. Rejection was common: about **39.5%** of candidates were rejected in the first round, and **45.3%** of those rejected were revised and accepted later.

The dataset statistics indicate both modality diversity and substantial reasoning complexity. The distribution is **31.4% text**, **35.9% image**, **28.8% video/audio**, with the remainder in tables and other appendix-level distribution details. **73.5%** of questions require **both multi-hop reasoning and multimodal conflict resolution**. The average number of gold resources is about **2.0** per question. MERRIN also includes temporal annotations in the form of **Effective Year** and **Freshness**, the latter categorized as never-, slow-, or fast-changing.

This construction methodology suggests that MERRIN is designed to minimize benchmark artifacts, particularly hidden text-only shortcuts. That design choice is central to its role as a stress test for genuine multimodal search competence.

## 4. Evaluation protocol and benchmark settings

MERRIN is evaluated using **answer accuracy**, defined as whether the predicted answer matches the gold answer. The benchmark uses an **LLM-as-judge** following BrowseComp-style evaluation. Because answers are designed to be precise and unambiguous, this procedure is described as usually close to exact match after normalization; the judging prompt extracts the final answer and accepts it if it matches the gold answer or falls within a small numerical tolerance [2604.13418].

The benchmark does not define a conventional train/dev/test partition, because it is presented as a **benchmark**, not a training dataset. Instead, it reports results for the full **162-question benchmark**, a **50-question subset** for human evaluation, and a **50-example subset** for two-step failure analysis.

The experimental design compares **10 models** across **3 search settings**. The models include closed-source systems—**GPT-5.4-nano**, **GPT-5.4-mini**, **Gemini 3 Flash**, **Gemini 3 Pro**, **Gemini 3.1 Lite**, **Gemini 3.1 Pro**, **Gemini Deep Research Agent**—and open-weight systems—**Qwen3-4B**, **Qwen3-30B**, and **Qwen3-235B**. The three search settings are:

- **No Search**: no tools, only parametric knowledge.
- **Native Search**: model-provided tools; for GPT, **web_search**; for Gemini, **Google Search + URL Context**; not applicable to Qwen models.
- **Agentic Multimodal Search**: a custom **smolagents** agent with **visit_webpage** and **watch_video** tools.

A key operational distinction is that **Native Search often cannot process video/audio well**, whereas the agentic configuration is explicitly designed to handle text, images, and video more flexibly. This makes MERRIN not only a benchmark of reasoning quality but also a benchmark of tool-interface adequacy.

## 5. Quantitative results and empirical behavior

MERRIN is reported as highly challenging. The **average accuracy across all runs** is **22.3%**, and the **best-performing agent** reaches only **40.1%**. When aggregated by search setting, the average is around **17.3%** for **No Search**, **23.1%** for **Native Search**, and **33.7%** for **Agentic Multimodal Search**. The best individual result is **Gemini 3.1 Pro + Agentic Multimodal Search** at **40.1%**. **Gemini Deep Research Agent with Native Search** reaches **33.3%**. GPT-based systems generally lag Gemini systems under native search, and open-weight Qwen models improve less from search than closed models [2604.13418].

| Setting or system | Accuracy |
|---|---:|
| Average across all runs | 22.3% |
| No Search average | 17.3% |
| Native Search average | 23.1% |
| Agentic Multimodal Search average | 33.7% |
| Gemini 3.1 Pro + Agentic Multimodal Search | 40.1% |
| Gemini Deep Research Agent + Native Search | 33.3% |

The reported results also challenge a common assumption that more search activity necessarily improves outcomes. The paper states that the number of search queries or pages visited **does not correlate strongly** with accuracy. Stronger systems often over-search rather than search better. The benchmark therefore distinguishes between raw search effort and effective evidence acquisition.

An additional intervention supports the importance of video. When a **video tool** is added to Native Search, performance improves by an average of **5.7%** across four Gemini models. This suggests that the benchmark’s multimodal difficulty is not merely nominal; video access materially changes the attainable performance ceiling.

## 6. Failure modes, human comparison, and benchmark significance

The failure analysis identifies three salient patterns. First, agents show an **over-reliance on text**. For the best agent, retrieved evidence is **87.7% text**, **6.8% image**, and **5.5% video/audio combined**, even though the benchmark itself is much more balanced across modalities. Second, stronger agents often exhibit **over-exploration**. **Gemini Deep Research** times out on **33.1%** of questions, and **Gemini Pro Native Search** hits **“Too_Many_Tool_Calls”** on **12.7%**. Third, agents show **inefficient source selection**: human URL precision is **38.1%**, whereas the agentic system’s URL precision is **1.8%** [2604.13418].

The paper also argues that **reasoning is the bigger bottleneck**. For **Gemini 3.1 Pro**, the move from **Agentic Multimodal Search** at **40.1%** to the final **gold sources prompting** upper bound at **47.7%** yields only modest gains. This indicates that removing search noise helps, but not enough to close the performance gap; reasoning over evidence remains the primary limitation.

Human comparison reinforces that conclusion. On a **50-question** subset, **human accuracy** is **71.4%**, compared with **40.1%** for **Agentic Multimodal Search (Gemini 3.1 Pro)** and **30.9%** for **Native Search (Gemini 3.1 Pro)**. Humans also use fewer resources: **2.9 searches** on average versus **9.1** for the agentic system, and **2.9 webpage visits** versus **3.5**. Human URL selection is markedly better, with **38.1%** precision, **48.9%** recall, and **42.8%** F1. Extra time benefits humans substantially more: they improve from **59.2% at 5 minutes** to **71.4% overall**, whereas agents improve only from **29.6% to 30.9%** in native mode and from **34.0% to 40.1%** in agentic mode.

Human error patterns are also diagnostically different. Mistakes are often **small extraction errors** rather than failures of source discovery: **wrong count** accounts for **43%**, **right source, wrong detail** for **29%**, **partial/imprecise answer** for **14%**, and **other** for **14%**. This suggests that humans generally solve the search and modality-selection problem more effectively, while remaining fallible at fine-grained extraction.

Taken together, these results establish MERRIN as a benchmark centered on three intertwined capacities: **modality inference**, **source selection under web noise**, and **evidence-grounded multi-hop reasoning**. A plausible implication is that progress on MERRIN will require not only better multimodal encoders or better search tools in isolation, but tighter coupling between search policy, modality-aware evidence acquisition, and robust reasoning under conflict.

Source: https://www.emergentmind.com/topics/merrin