MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments

Abstract: Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents' ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.

• The paper introduces MERRIN, a benchmark that rigorously evaluates LLM agents on multimodal evidence retrieval and reasoning in noisy web conditions.
• The paper demonstrates that leveraging non-text evidence, such as video and audio, improves accuracy by up to 5.7%, emphasizing modality fusion.
• The paper finds that reasoning errors, rather than retrieval shortcomings, are the primary barrier to achieving human-level performance.

MERRIN: Evaluation of Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments

Introduction and Motivation

The MERRIN benchmark ("MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments" (2604.13418)) represents a significant advance in the rigorous evaluation of search-augmented agents operating under realistic, web-scale, noisy, and modality-diverse conditions. Grounded in the observation that user information needs often require processing of heterogeneous evidence across text, image, video, and audio modalities, MERRIN is constructed to probe agentic weaknesses in retrieval, modality identification, and cross-modal reasoning when confronted with conflicting, incomplete, or distractor-laden real-world sources. Unlike prior benchmarks, MERRIN targets:

• Absence of modality cues in queries, forcing agents to infer which modalities are necessary rather than relying on artificial hints.
• Mandatory use of non-text evidence, preventing exploitation of shortcut behaviors that over-emphasize text.
• Open-web search with complex, noisy, and conflicting evidence, reflecting genuine web conditions.

  Figure 1: Overview of the MERRIN pipeline; agents must (i) infer appropriate modalities, (ii) select the correct sources, and (iii) perform robust multi-hop reasoning. Key failure modes are color-coded: reasoning, modality, and retrieval errors.

Benchmark Design and Dataset Composition

Dataset Structure

MERRIN comprises 162 questions, with the majority constructed from scratch and a minority adapted and extended from SeedQA and ChartMuseum. Inclusion criteria enforce:

• Natural language queries devoid of explicit modality mentions,
• Manual verification that solutions genuinely require non-textual evidence,
• At most one unambiguous, short, and verifiable answer per question.

Questions stretch across two orthogonal axes: reasoning type (multi-hop and/or multimodal conflict resolution) and multimodal role (evidential answer source and/or a required part of a reasoning chain).

Modalities

The dataset's ground-truth sources are split among text, image, video, audio, and table, with a substantial proportion of questions requiring video or image reasoning in addition to text.

Figure 2: Distribution of gold sources by modality type, illustrating MERRIN’s broad coverage beyond text and images by including significant video and audio resources.

Noise and Conflict

To simulate real search scenarios, queries yield not only relevant sources but also distractors: incomplete, conflicting, or misleading evidence typical of the web. Manual annotation ensures that text-only shortcuts are unavailable and that conflicting signals are organically induced through genuine web retrieval, not synthetically constructed.

Experimental Protocol and Systems Evaluated

Three primary agent search settings are benchmarked:

• No Search: Agent operates solely on parametric knowledge.
• Native Search: Use of each LLM's built-in search, limited modality support.
• Agentic Multimodal Search: Access to dedicated extraction modules for full multi-modality (video, image, audio), implemented atop the smolagents framework.

Ten models are evaluated, including large closed-source LLMs (various Gemini and GPT-5.4 configurations) and open-weight models (Qwen3-4B/30B/235B).

Quantitative Results and Agentic Behavior

Overall Performance

MERRIN is exceptionally challenging. Average accuracy is 22.3% across all agent runs; best-case performance is 40.1% (Gemini-3.1-Pro in Agentic Multimodal Search). No configuration approaches human accuracy (71.4%).

Impact of Multimodality and Tool Availability

Addition of a video-processing tool leads to a 5.7% mean absolute accuracy gain across Gemini variants, underscoring the importance of broad modality support for robust multi-hop and cross-modal reasoning.

Figure 3: Adding video tool capability boosts Native Search agent accuracy, indicating the substantial information content locked in non-text modalities.

Agent Search Behavior vs. Human Search

Despite investing more computational resources, agents rely disproportionately on text (up to 87%) and generate more queries (e.g., 9.1 queries versus 2.9 for humans), but their source selection is imprecise (low URL overlap precision).

Figure 4: Analysis of search effort and URL overlap: agentic models expend more queries and visit more URLs, but humans maintain vastly higher effective precision.

Reasoning Versus Retrieval Bottlenecks

Experiments isolating search versus reasoning limitations show that providing gold evidence only modestly improves accuracy (from 40.1% to 47.7%). Thus, reasoning over complex, noisy, and multi-hop evidence is the primary limiting factor; search improvements alone will not close the performance gap.

Error Analysis and Failure Modes

Agents display strong modality selection and reasoning errors in noisy conditions:

• Excessive reliance on textual evidence even when non-text evidence is required.
• Over-exploration: more capable agents issue too many queries/tool calls, get distracted by tangential or conflicting evidence, and sometimes fail to answer within practical limits.
• Multi-step questions: error propagation from initial retrieval/selection amplifies subsequent reasoning mistakes, with early-stage failure dominating incorrect responses.

Human Evaluation

Human annotators are far more effective at high-recall, high-precision evidence selection and at integrating diverse modalities. Most human errors are minor (e.g., off-by-one counts, wrong time in a video), not fundamental misunderstandings or retrieval failures. Humans also leverage additional search time productively, unlike LLM agents that plateau early and do not convert more time into commensurate performance gains.

Theoretical Implications and Future Directions

MERRIN reveals that current LLM-based agentic architectures are not only bottlenecked by retrieval in noisy environments but are especially limited in robust multimodal reasoning—particularly, resolving conflict and integrating multi-step evidence chains. The observed artifacts (e.g., over-exploration, text-centric search, inefficient modality routing) call for research in:

• More effective modality selection and fusion mechanisms,
• Improved reasoning architectures for grounding, alignment, and cross-evidence synthesis,
• Strategies for mitigating distraction and over-exploration under real-world noise constraints.

Additionally, it motivates future benchmarks with even broader source coverage, temporal dynamics (Figure 5), and task diversity.

Figure 5: Effective year distribution for MERRIN, demonstrating attention to temporal freshness and recency in question/evidence selection.

Conclusion

MERRIN provides a technically rigorous, diagnosis-oriented evaluation setting for search-augmented agents operating with open-web, noisy, and broad modality evidence. Quantitative and qualitative analyses across a broad suite of closed and open LLMs demonstrate that robust multi-hop, multimodal, and conflict-aware reasoning is a major unsolved challenge. Substantial headroom remains for both methodology and architecture research aimed at bridging the agentic gap between current LLM architectures and human performance in unconstrained, information-rich, multimodal web environments.