InfoMosaic-Flow: Agentic Data Synthesis

• InfoMosaic-Flow is a two-stage agentic data-synthesis pipeline that integrates multi-source evidence from general web search and specialized APIs to construct non-trivial tasks.
• It employs an Organizer to decompose tasks, an Executor to invoke domain-specific tools, and a Verifier to iteratively refine queries against trivial solutions.
• The pipeline underpins the InfoMosaic-Bench benchmark, rigorously evaluating LLM agents on complex, cross-domain tasks in fields like finance, medicine, and geospatial analysis.

InfoMosaic-Flow is a two-stage agentic data-synthesis pipeline developed to generate complex multi-source information-seeking tasks that require true integration of general web search and specialized domain tools, with all tasks grounded in verified tool outputs and filtered for non-triviality. The pipeline is a cornerstone of the InfoMosaic-Bench benchmark, devised to rigorously evaluate tool-augmented LLM agents on non-trivial tasks spanning domains such as medicine, finance, maps, video, and multi-domain integration (Du et al., 2 Oct 2025).

1. Core Motivation and Objectives

Modern LLM agents predominantly rely on open-web search as their primary information resource. While this mode of information seeking is convenient, it is fundamentally limited by noise, incompleteness, and the absence of many domain-specific facts (e.g., proprietary financials, specialized medical trial results, geospatial schedules). With the emergence of the Model Context Protocol (MCP)—which exposes hundreds of specialized APIs ("tools") across various fields—a new challenge is posed: can agentic pipelines synthesize non-trivial tasks that necessitate genuine integration of generalist and domain-specific resources, such that no single information source suffices?

InfoMosaic-Flow is designed to systematically generate complex tasks whose solutions are unambiguously contingent on information drawn from multiple sources. The objectives are:

• To ground every task in verified outputs from domain-specific tools, guaranteeing evidence-based construction.
• To enforce cross-source dependencies so that correctly answering requires reasoning over distinct data sources.
• To prune “shortcut” cases (solvable via trivial lookups or a single web query), thereby raising the bar for multi-step, tool-augmented reasoning.

2. Pipeline Structure and Stepwise Execution

InfoMosaic-Flow operates in two principal stages: information seeking and iterative refinement.

Stage 1: Task Definition and Grounding

An agentic division-of-labor is employed:

• Organizer ("planner" LLM): Generates scenarios from seed data (e.g., Wikipedia, ClinicalTrials.gov), decomposes into discrete constraints/subtasks (e.g., company HQ location, financials, beta coefficient).
• Executor ("worker" LLM + MCP tools): Selects and invokes domain-specific tools to resolve subtasks, returning structured evidence (e.g., numerical values, coordinates).
• Integration: The Organizer updates its plan with incoming structured evidence and issues further subtasks. Once a task involves at least a threshold number of tool calls ($T_{\text{min}}$), the collected constraints and evidence are bundled into an initial QA pair.

A minimum-calls filter discards any instances with fewer than $T_{\text{min}}$ tool invocations, ensuring sufficient complexity and interdependency.

Stage 2: Enforcement of Multi-Source Requirements and Shortcut Filtering

• Verifier: An LLM restricted to web search (the web_search tool) attempts to answer the complete question or any atomic condition using the web alone.
• Refiner: Decomposes the QA pair into atomic conditions $\{c_1, ..., c_k\}$, checks their individual web solvability, and "fuzzes" any that are trivially web-solvable (by, for example, adding temporal qualifiers, synonyms, or combining conditions).
• Iteration: This loop continues until neither the overall question nor any sub-condition is answerable via web search alone, enforcing cross-source dependency.

The result is a question-evidence-answer tuple where the solution requires integrating multiple pieces of evidence that cannot be obtained from a single lookup, web search, or atomic query.

3. Pseudocode and Key Formalisms

The InfoMosaic-Flow pipeline is algorithmically specified as follows:

$\{c_1, ..., c_k\}$0

Mathematical notation central to the pipeline includes:

• Task: $\tau = (q, T_{\text{avail}}, K, GT)$, where $T_{\text{avail}}$ is the available toolset and $K$ is the number of tool calls.
• Dependency graph $G = (V, E)$: $V = \{c_1,\dots,c_k\}$; $E$ signifies must-combine dependencies.
• Filtering criteria:
  • Web-solvable filter: $E_\text{web}(q) = 1 \implies$ discard.
  • Atomic condition filter: $T_{\text{min}}$0 discard and fuzz $T_{\text{min}}$1.
• Informal shortcut loss for task $T_{\text{min}}$2: $T_{\text{min}}$3, minimized when $T_{\text{min}}$4.

4. Running Example and Domain Application

In a finance domain instantiation, a seed such as “Pick a US-listed tech company in Austin, Texas” evolves as follows:

1. The Organizer queries for companies headquartered in Austin.
2. The Executor leverages maps_text_search, receives possible HQs, then queries get_income_statement for Q1 and Q2 earnings, and get_quote_change for stock metrics.
3. The Organizer specifies further constraints (e.g., YTD stock change, revenue bounds).
4. These subtasks collectively produce a question requiring simultaneous satisfaction of multiple numerical and categorical constraints:
   • “Identify the publicly traded tech company in Austin, TX with Q1 2025 net income > 0, Q2 2025 net < 0, YTD change 10–20%, beta > 5, Q2 revenue $70–80 M. What is its ticker?”
5. In Stage 2, the Verifier's inability to answer this question or any individual condition via web search confirms its multi-source, tool-requiring status.

5. Hyperparameters, Toolsets, and Domain Configurations

Principal hyperparameters and operational settings include:

• Maximum tool calls $T_{\text{min}}$5 (to avoid runaway processes).
• Minimum tool calls $T_{\text{min}}$6 (to enforce task non-triviality—“richness”).
• Each domain (medicine, finance, etc.) employs its own curated subset from a broader suite of 77 MCP tools (see cited work’s appendix).
• The Refiner employs domain-agnostic fuzzing heuristics: adding time qualifiers, merging constraints, or employing synonyms to frustrate direct web resolution.
• Verifier employs up to 5 web queries per attempt.

A summary table of foundational settings:

Parameter	Value/Setting	Purpose
$T_{\text{min}}$7	20	Upper bound for tool call iterations
$T_{\text{min}}$8	3	Lower bound to enforce multi-source logic
Tool domains	6 (e.g. medicine, finance)	Domain coverage for diversity
Tool inventory	77 tools	Total, as curated in the benchmark
Fuzzing heuristics	Temporal, merge	For defeating web-based shortcutting

6. Scalability and Reliability Properties

The pipeline is designed for both scalability and reliability:

• Scalability: Segregating high-level planning (Organizer) from low-level tool invocation (Executor) enables executor calls to be parallelized or distributed across multiple MCP servers. This modularity ensures that new tools can be integrated without redesigning the Organizer logic.
• Automation: Stage 2’s web-only Verifier and iterative fuzzing loop completely automate shortcut filtering, bypassing the need for manual construction or sample inspection on a large scale.
• Reliability: Each synthesized task is traceable to actual tool responses, guaranteeing evidence–answer consistency. Filters enforce answer coherence and exclude contradictory or under-specified queries. Human annotators provide final screening, achieving a Cohen’s $T_{\text{min}}$9 agreement metric.
• Ablation studies underline the necessity of both pipeline stages: omitting the Executor collapses tool use and task diversity, while omitting fuzzing leaves ~45% of cases trivial.

7. Context, Benchmark Use, and Research Impact

InfoMosaic-Flow is not an evaluation protocol but a meta-synthesis method furnishing InfoMosaic-Bench, a benchmark comprising 621 questions distributed across six major knowledge domains. Every question demands multi-source synthesis over web and specialist APIs, constructed through rigorous filtering to assure difficulty and eliminate triviality. This design directly supports empirical investigation into the current limits of LLM agent tool-usage capabilities, demonstrating (for example) that even models such as GPT-5 succeed in only 38.2% of tasks when restricted to web search, and that 22.4% of failures are attributable to incorrect tool selection or usage (Du et al., 2 Oct 2025). A plausible implication is that InfoMosaic-Flow’s methodology establishes a strong lower bound on the complexity of tool-augmented information-seeking tasks for benchmarking next-generation LLM agents.