M3-Bench: A Multimodal MCP Benchmark

• M3-Bench is a benchmark designed to evaluate multimodal, multi-hop, and multi-threaded tool-using agents under the Model Context Protocol (MCP) by assessing both semantic fidelity and workflow structure.
• It introduces a similarity-driven trajectory evaluation pipeline that uses tool-restricted, similarity-bucketed Hungarian matching to align predicted and reference tool calls.
• The benchmark comprises 28 curated tasks, 231 tools, and 208 trajectories, rigorously testing agents’ abilities in visual grounding, state persistence, and step-by-step tool invocation.

M$^3$-Bench is a benchmark for multimodal tool-using agents operating under the Model Context Protocol (MCP), introduced in "M3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark" (Zhou et al., 21 Nov 2025). It is presented as the first benchmark specifically designed to evaluate multimodal tool use under MCP, with emphasis on workflows in which images and text jointly determine tool selection, argument construction, cross-step causal dependencies, within-step parallelism, and persistence of intermediate resources across steps. Its central methodological contribution is a similarity-driven trajectory evaluation pipeline that aligns predicted and reference tool calls through tool-restricted, similarity-bucketed Hungarian matching, thereby separating call-level semantic fidelity from workflow-level structural consistency (Zhou et al., 21 Nov 2025).

1. Scope and defining concepts

M$^3$-Bench is organized around three properties that motivate its name: Multi-modal, Multi-hop, and Multi-threaded. "Multi-modal" denotes tasks in which visual grounding and textual reasoning are jointly required before tool invocation. "Multi-hop" denotes workflows with at least two steps and explicit inter-step dependencies, so that later tool calls consume artifacts or information produced earlier. "Multi-threaded" denotes steps containing multiple order-independent tool calls that can be executed in parallel under shared state (Zhou et al., 21 Nov 2025).

This formulation distinguishes the benchmark from MCP evaluations that reduce tool use to flat retrieval or single-call API selection. In the M$^3$-Bench setting, an agent may need to identify entities from an image, use them to parameterize tools, preserve created files, and coordinate multi-step execution rather than merely emit a plausible final answer. The benchmark therefore evaluates the trajectory of MCP interaction rather than only end-task success (Zhou et al., 21 Nov 2025).

The paper formalizes a reference trajectory as a sequence of steps,

$$\mathcal{T}^\star=\{S^\star_{1},\ldots,S^\star_{L^\star}\},$$

with each step represented as a set of tool calls,

$$S^\star_{\ell}=\{c^\star_{\ell,1},\ldots,c^\star_{\ell,n^\star_\ell}\},$$

and each call written as $c=(\mathrm{tool}(c),\mathrm{args}(c))$. This step-set representation is essential: causal order is enforced across dependent steps, while within-step order can remain unconstrained when calls are independent (Zhou et al., 21 Nov 2025).

2. Benchmark composition and task design

The benchmark spans 28 MCP servers, 231 tools, 28 curated multimodal MCP tasks, 208 evaluation trajectories, 644 reference steps, and 1337 MCP tool calls in those reference trajectories (Zhou et al., 21 Nov 2025).

Component	Scale
MCP servers	28
Tools	231
Curated multimodal MCP tasks	28
Evaluation trajectories	208
Reference steps	644
MCP tool calls in references	1337

The server ecosystem is deliberately curated to minimize functional overlap and instability. The covered domains include Academic / Knowledge, Science / Space, Geography / Travel, E-commerce / Finance, Office Automation, Weather / Air Quality, and Health (Zhou et al., 21 Nov 2025). This curation is intended to keep the benchmark reproducible while still exposing agents to heterogeneous tool graphs.

Each task combines an image with a natural-language instruction. Representative tasks include Items Price, Attraction, Movies, Vision Research, Camping, Hazard, Biology Cut, QR Attraction, Generate PowerPoint, Travel Comparison, Heatmap, Stock Growth Estimation, and NixOS Diagnostic (Zhou et al., 21 Nov 2025). These tasks differ substantially in failure mode: some are dominated by OCR or object recognition, some by cross-domain retrieval chains, and some by resource persistence and document-generation workflows.

The task design explicitly includes persistent artifacts. Later steps may consume files or images created earlier, such as cropped animal images, downloaded linked media, annotated hazard images, or generated presentation files. This makes state persistence part of the benchmarked capability rather than an implementation detail (Zhou et al., 21 Nov 2025).

3. Reference trajectory construction

Reference trajectories are not simple manual annotations. M$^3$-Bench constructs them through an Executor–Judge pipeline with human verification (Zhou et al., 21 Nov 2025).

The Executor receives the task prompt, the image, and the MCP tool registry with tool docstrings. It first performs stepwise tool planning, then emits strictly valid JSON tool calls under controlled execution, and finally decides whether enough evidence has been collected to produce the final natural-language reply. The format constrains the interaction to explicit MCP calls while allowing multiple calls in a single step, thereby supporting multi-threaded execution. Tool outputs, including generated file paths and images, persist in a temporary workspace and can be consumed by later steps (Zhou et al., 21 Nov 2025).

The Judge reviews the initial prompt, image, and full Executor dialogue, rejecting trajectories with causal misordering, factual errors, or hallucinated continuation after incorrect tool outputs. When the trajectory is acceptable, the Judge reconstructs an "optimal trajectory" by removing failed or redundant calls, merging operations that could run concurrently into a single step, and preserving only the useful tool name and arguments. That reconstructed trajectory becomes the benchmark reference (Zhou et al., 21 Nov 2025).

Human verifiers then audit scope, logical consistency, and trajectory quality. The paper reports a pairwise preference study in which participants preferred the human-optimized best trajectory over original MLLM-produced trajectories in 98.16\% of cases, corresponding to 589/600 comparisons (Zhou et al., 21 Nov 2025). A separate calibration study over 40 call pairs selected the similarity thresholds

$$(\tau_{\mathrm{strong}}, \tau_{\mathrm{weak}}) = (0.8, 0.6),$$

as best matching human judgments of strong, medium, and weak similarity (Zhou et al., 21 Nov 2025).

4. Alignment procedure and metric suite

M$^3$-Bench is designed to avoid brittle exact matching and to reduce dependence on opaque LLM judges at the call-alignment level. Each tool call is serialized by a canonical function $\phi(\cdot)$, embedded with a sentence encoder $^3$0, and compared through cosine similarity (Zhou et al., 21 Nov 2025):

$^3$1

Alignment is restricted to same-tool pairs through tool-name buckets, preventing cross-tool hallucinations from receiving credit. Similarities below $^3$2 are heavily penalized, and within each tool bucket the final correspondence is obtained via Hungarian matching. The result is an auditable one-to-one matched set $^3$3 between predicted and reference calls (Zhou et al., 21 Nov 2025).

On top of that alignment, the benchmark reports three metric families.

First, detection metrics quantify call coverage and overproduction:

$^3$4

These separate omission of required calls from gratuitous or hallucinated tool use (Zhou et al., 21 Nov 2025).

Second, trajectory alignment metrics evaluate semantic and structural quality. ArgSim measures semantic fidelity of matched arguments among confident matches. StepCoh measures whether calls from one reference step are unnecessarily scattered across multiple predicted steps, preserving within-step grouping and parallel structure. MergePur measures whether distinct causal steps are wrongly merged. OrdCons measures inversions in cross-step order among matched calls. To prevent structural metrics from being inflated by systems that match only a few easy calls, M$^3$5-Bench also reports recall-covered structural scores in which per-sample structural metrics are weighted by recall (Zhou et al., 21 Nov 2025).

Third, LLM judge metrics assess end-task performance. A four-LLM judge ensemble reports Task Completion and Information Grounding, with trimmed-mean aggregation after removing the per-instance minimum and maximum judge scores. Task Completion evaluates whether the task was actually solved; Information Grounding evaluates whether the final output is supported by tool-derived evidence rather than unsupported world knowledge (Zhou et al., 21 Nov 2025).

The benchmark also replays predicted trajectories through an MCP interpreter and classifies each call outcome as Illegal calling format, Unknown Tool Invocation, Invalid Invocation Arguments, Successful (Resource Not Found), or Successful call. This replay-based layer makes it possible to distinguish reasoning failures from syntax or schema failures (Zhou et al., 21 Nov 2025).

5. Empirical findings

M$^3$6-Bench evaluates representative state-of-the-art MLLMs, including GLM 4.5v, Qwen2.5-VL-72B, InternVL 3.5, Gemini 2.5 Flash Lite, Gemini 2.5 Flash, Gemini 2.5 Pro, Claude 4.5 Haiku, Claude 4.5 Sonnet, GPT-5 Nano, GPT-5 Mini, GPT-5, Llama-4-Scout-17B-16E, Grok-4 Fast, and Grok 4 (0709) (Zhou et al., 21 Nov 2025).

Model	Average score
GPT-5	0.482
Gemini 2.5 Pro	0.423
Grok 4 (0709)	0.411
GPT-5 Mini	0.395
Gemini 2.5 Flash	0.388
Claude 4.5 Sonnet	0.333

The headline result is that even the best systems remain far from saturation. GPT-5 achieves the highest average score at 0.482, with Recall 0.627, Precision 0.264, ArgSim 0.583, StepCoh 0.502, OrdCons 0.290, MergePur 0.453, Task Completion 0.671, and Information Grounding 0.657. Gemini 2.5 Pro shows a more balanced precision-recall profile, with Recall 0.501 and Precision 0.449, but lower end-task completion. Grok 4 (0709) attains especially strong judge-based end-task metrics, including Task Completion 0.664 and Information Grounding 0.638 (Zhou et al., 21 Nov 2025).

The benchmark reveals persistent weaknesses in argument fidelity and structure consistency. Smaller or latency-oriented models often exhibit high precision but low recall; for example, Claude 4.5 Haiku records Precision 0.480 but Recall 0.193, indicating short, locally correct, under-complete trajectories. Larger models improve recall by attempting more calls, but often over-call tools, as illustrated by GPT-5 with Recall 0.627 and Precision 0.264 (Zhou et al., 21 Nov 2025).

Replay-based failure analysis identifies recurring error classes. Some models fail at the formatting layer by not producing valid MCP JSON. Others hallucinate tool names, invoking nonexistent tools. A third pattern is correct tool selection with invalid arguments, often leading to HTTP 400 or analogous execution failures. A fourth pattern is syntactically valid invocation of missing backend resources. A fifth pattern is poor image understanding that corrupts the downstream chain; the paper’s case study on Travel Comparison shows how incorrect landmark identification poisons later geocoding and AQI retrieval (Zhou et al., 21 Nov 2025).

Performance is also highly task-dependent. Top systems perform comparatively well on Barcode Lookup—for example, Gemini 2.5 Pro 0.828, GPT-5 0.762, and Grok 4 0.770—and on workflow-heavy tasks such as Biology Cut (GPT-5 0.851), Generate PowerPoint (Gemini 2.5 Pro 0.727, GPT-5 0.705), Travel Comparison (Gemini 2.5 Flash 0.718, GPT-5 0.688), Heatmap (GPT-5 0.662), and Stock Growth Estimation (Grok 4 0.735) (Zhou et al., 21 Nov 2025). Harder tasks include Drug, MedExcel, Vision Research, Fruit Cal, NixOS Diagnostic, and some long-horizon planning tasks such as Camping, suggesting that cross-domain reasoning, exact argument construction, and conditional branching remain major bottlenecks (Zhou et al., 21 Nov 2025).

6. Benchmark landscape, significance, and disambiguation

Within MCP-oriented evaluation, M$^3$7-Bench occupies a distinct niche. The paper’s comparison table states that it is the only listed benchmark jointly supporting an MCP ecosystem, information grounding, fuzzy task description, multi-hop + threaded workflows, multimodality, and a similarity metric (Zhou et al., 21 Nov 2025). A closely related benchmark is "MSC-Bench: A Rigorous Benchmark for Multi-Server Tool Orchestration" (Dong et al., 22 Oct 2025), which focuses on hierarchical multi-server orchestration, functional overlap, end-to-end retrieval and planning, and robustness in an MCP ecosystem, but does not center multimodal visual grounding or similarity-driven tool-call alignment in the manner of M$^3$8-Bench (Dong et al., 22 Oct 2025).

The name also requires disambiguation. Several unrelated benchmarks use similar "M3" or "M$^3$9" nomenclature: "M$^3$0oralBench: A MultiModal Moral Benchmark for LVLMs" (Yan et al., 2024), "M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenes" (Zhang et al., 2024), "M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts" (Li et al., 2023), "M$^3$1Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions" (Huang et al., 5 Jun 2026), and "M3-BENCH: Process-Aware Evaluation of LLM Agents Social Behaviors in Mixed-Motive Games" (Xie et al., 13 Jan 2026). These works address moral evaluation, robotics, 3D instruction following, multimodal conversational memory, and social-behavior evaluation, respectively, rather than MCP tool use.

The significance of M$^3$2-Bench lies in its reframing of multimodal agent evaluation around the MCP trajectory. It evaluates whether an agent can transform visual evidence into valid tool parameters, schedule dependent and parallel calls, preserve intermediate artifacts, and produce a final answer grounded in tool outputs. The benchmark’s metric design is equally central: by distinguishing detection, argument similarity, step coherence, merge purity, order consistency, and judge-based end-task grounding, it supports diagnosis of failure mode rather than only leaderboard ranking (Zhou et al., 21 Nov 2025).

The paper also makes clear the main limitations. M$^3$3-Bench covers a small curated task set of 28 tasks and 208 trajectories rather than an exhaustive MCP ecosystem. Its server pool is intentionally compact and curated, which improves reproducibility but may underrepresent open-ecosystem messiness. Reference trajectories are standardized "optimal" trajectories, so some valid alternative workflows may not be fully reflected by a single reference even with fuzzy matching. Task Completion and Information Grounding still depend on an LLM judge ensemble, and similarity-based alignment depends on calibrated thresholds. A plausible implication is that the benchmark is best understood as a high-control diagnostic suite rather than a complete model of open-world MCP deployment (Zhou et al., 21 Nov 2025).

The benchmark is publicly released through an anonymous repository at https://github.com/EtaYang10th/Open-M3-Bench (Zhou et al., 21 Nov 2025).