---
title: 'InfoMosaic-Bench: Multi-Source LLM Evaluation'
url: https://www.emergentmind.com/topics/infomosaic-bench
type: topic
---

# InfoMosaic-Bench: Multi-Source LLM Evaluation

InfoMosaic-Bench is a benchmark for evaluating the capacity of tool-augmented large language model (LLM) agents to perform non-trivial multi-source information seeking by coordinating general-purpose web search with domain-specific Model Context Protocol (MCP) tools. InfoMosaic-Bench spans six representative domains—Medicine/Biology, Finance, Maps, Video, Web, and Multi-Domain Integration—and is designed to expose the limitations of web search alone, rigorously measure tool orchestration abilities, and require agent reasoning over chained, heterogeneous APIs. All benchmark tasks are constructed to be reliably grounded in verified tool outputs and are systematically pruned to eliminate shortcut tasks solvable by trivial web lookup, thereby guaranteeing both reliability and non-triviality [2510.02271].

## 1. Benchmark Construction and Motivation

The principal motivation for InfoMosaic-Bench is to address two fundamental weaknesses in LLM-based information seeking: (1) the unreliability and noise inherent in open web content, and (2) the necessity for precise, domain-specific knowledge not accessible from the general web. With the introduction of MCP, agents are theoretically capable of leveraging thousands of specialized APIs; InfoMosaic-Bench tests the empirical reality of this ability by targeting complex scenarios requiring multiple, coordinated tool calls and cross-source dependency reasoning.

Benchmark tasks are formulated such that a single API invocation is insufficient: agents must chain together data from distinct sources within or across domains, and cannot succeed through simple web search or superficial retrieval.

## 2. InfoMosaic-Flow Synthesis Pipeline

InfoMosaic-Flow is a two-stage, organizer–worker workflow ensuring that constructed benchmark tasks are both non-trivial and grounded in actual tool outputs:

### 2.1 Task Formalization

Let $\mathcal{D}$ denote a domain and $\mathcal{Q}_{\mathcal{D}}$ its query set. Each task instance is denoted by
$$
\tau = (q, \mathcal{T}_\text{avail}, K, GT)
$$
where $q$ is the query, $\mathcal{T}_\text{avail}$ is the set of available MCP tools, $K$ the permitted tool-call budget, and $GT$ the ground-truth answer. An agent’s interaction with tools forms a history $H = (h_1, h_2, ..., h_T)$, with an evaluation function $E(H, \tau) \rightarrow \{0,1\}$ scoring solutions.

### 2.2 Organizer–Worker Stages

- **Stage 1: Information Seeking**
  - *Scenario Proposing*: An "organizer" drafts complex scenarios, seeded from structured databases (e.g., Wikipedia, Baidu Baike).
  - *Domain Fact Gathering*: Organizers emit executor calls for subtasks; "workers" interact with domain MCP tools to yield verified facts (e.g., clinical trial data, finance tables).
  - *Integration*: Organizer combines multi-tool facts as conjunctive queries (“Find $X$ satisfying Condition 1 AND Condition 2 AND ...”).
  
- **Stage 2: Iterative Refinement**
  - *Verifier* (endowed only with web_search) attempts to answer via single-source lookups. If a web-only shortcut is discovered, the problem is rewritten until isolated web solutions are eliminated but a unique solution exists via multi-source reasoning. This process systematically enforces the benchmark’s non-triviality.

## 3. Domain Coverage and Task Examples

InfoMosaic-Bench targets six domains, each paired with purpose-fit MCP tools. All task examples and tool configurations come explicitly from the benchmark design [2510.02271]:

| Domain           | MCP Tools                                                       | Example Task (summary)                                                |
|------------------|-----------------------------------------------------------------|-----------------------------------------------------------------------|
| Medicine/Biology | PubMed, ClinicalTrials.gov, MyVariant/MyGene/MyChem.info        | European tech company, Q1 2025 net loss, stock beta < 1; find ticker. |
| Finance          | FMP MCP server (financials, price, betas)                       | Public tech co., HQ in Austin, Q2 ’25 negative net income, beta > 5.  |
| Maps             | AMap, Google Maps MCP (geocoding, routing, POI)                 | Dalian toponym matching complex multi-modal transit/time constraints.  |
| Video            | YouTube MCP (search, metadata, comments)                        | Sep. 2025 upload, specified duration/subscribers/comments on YouTube.  |
| Web              | web_search, web_parse                                           | Historic sea voyage; identify a commander via multi-evidence query.   |
| Multi-Domain     | All above                                                       | US biotech: revenue, EPS, share price trend, CRISPR focus; find ticker.|

Each task is multi-condition, requires integrating at least two sources, and has no web-only solution after pruning.

## 4. Evaluation Protocol and Metrics

Each agent–task pair is evaluated via two principal metrics:

- **Accuracy**:
  $$
  \mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{E(H_i, \tau_i) = 1\}
  $$
  with $N = 621$ total tasks.
  
- **Pass Rate** (PR): Fraction of correctly solved subconditions (“testcases”) per task:
  $$
  \mathrm{PR} = \frac{\sum_{\text{subconditions}} \mathbf{1}\{\text{correct}\}}{\text{total subconditions}}
  $$

Quality control includes both automatic (tool-call filtering, answer–evidence consistency, coherence filtering) and manual (expert revision, factual alignment, user studies; Cohen’s $\kappa = 0.92$) procedures.

## 5. Experimental Setup

Fourteen state-of-the-art LLM agents were evaluated:

- **Closed-source**: GPT-5, o3, Grok-4, Claude-4.0-Sonnet, Qwen2.5-Max, Gemini-2.5-Flash, o4-mini.
- **Open-source**: GLM-4.5, Qwen3-235B-A22B, Qwen3-32B, DeepSeek-V3, Kimi-K2, Qwen3-Coder, Llama-4-Scout.

Agents rely on a ReAct-based framework (OpenAI Function Calling API, Python Sandbox for safe code execution, limit $K=20$ tool calls). Three tool-access settings are compared:
1. **Web-Only:** web_search.
2. **Domain-Only:** MCP tools per single domain.
3. **Multi-Domain:** all 77 MCP tools plus web_search.

## 6. Quantitative Results and Failure Analysis

### 6.1 Accuracy and Domain Effects

- **GPT-5 (web-only):** 38.2% Acc, 67.5% PR (best closed-source).
- Closed-source agents outperform open-source by 15–20% Acc.
- Substantial domain variability (Medical 53.1%, Video 36.0%, Web 29.0%).
- Aggregated accuracy improves with increased tool calls, plateauing at ≈8, after which redundancy degrades performance.
- Domain tools offer selective, inconsistent gains (Map and Video domains see improvements; Medical, Finance, and Multi-Domain often degrade).

### 6.2 Failure Modes

Primary failure classes for GPT-5 (web-only):

| Failure Mode                | Fraction of Total Failures |
|-----------------------------|---------------------------|
| Retrieval Miss              | 39.6%                     |
| Overgeneralization/Halluc.  | 28.2%                     |
| Tool Misuse/Selection       | 22.4%                     |
| Other (reasoning/context)   | ≈10%                      |

A total of 22.4% of failures are attributable to tool selection or misuse, highlighting persistent challenges in agent tool orchestration.

## 7. Findings, Limitations, and Research Directions

Major findings:

1. Web search alone is insufficient; even GPT-5 achieves only 38.2% accuracy in non-trivial scenarios.
2. Domain tools provide selective, inconsistent gains and may degrade performance outside precisely structured contexts.
3. Tool orchestration and input–output chaining remain a major bottleneck; 22.4% of failures are due to incorrect tool usage.
4. Non-trivial multi-tool reasoning remains unsolved: SOTA models are unable to reliably plan, parameterize, and chain tool calls for grounded information integration.

Recommended research directions include the development of more robust tool-reasoning curricula, planner–executor agent architectures with improved feedback, extension of task synthesis to new modalities, and integration of tool usage diagnostics in LLM fine-tuning [2510.02271].

Source: https://www.emergentmind.com/topics/infomosaic-bench