---
title: 'M3-Bench: A Multimodal MCP Benchmark'
url: https://www.emergentmind.com/topics/m-3-bench
type: topic
---

# M3-Bench: A Multimodal MCP Benchmark

M$^3$-Bench is a benchmark for multimodal tool-using agents operating under the Model Context Protocol (MCP), introduced in "M3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark" [2511.17729]. It is presented as the first benchmark specifically designed to evaluate multimodal tool use under MCP, with emphasis on workflows in which images and text jointly determine tool selection, argument construction, cross-step causal dependencies, within-step parallelism, and persistence of intermediate resources across steps. Its central methodological contribution is a similarity-driven trajectory evaluation pipeline that aligns predicted and reference tool calls through tool-restricted, similarity-bucketed Hungarian matching, thereby separating call-level semantic fidelity from workflow-level structural consistency [2511.17729].

## 1. Scope and defining concepts

M$^3$-Bench is organized around three properties that motivate its name: **Multi-modal**, **Multi-hop**, and **Multi-threaded**. "Multi-modal" denotes tasks in which visual grounding and textual reasoning are jointly required before tool invocation. "Multi-hop" denotes workflows with at least two steps and explicit inter-step dependencies, so that later tool calls consume artifacts or information produced earlier. "Multi-threaded" denotes steps containing multiple order-independent tool calls that can be executed in parallel under shared state [2511.17729].

This formulation distinguishes the benchmark from MCP evaluations that reduce tool use to flat retrieval or single-call API selection. In the M$^3$-Bench setting, an agent may need to identify entities from an image, use them to parameterize tools, preserve created files, and coordinate multi-step execution rather than merely emit a plausible final answer. The benchmark therefore evaluates the **trajectory** of MCP interaction rather than only end-task success [2511.17729].

The paper formalizes a reference trajectory as a sequence of steps,
$$
\mathcal{T}^\star=\{S^\star_{1},\ldots,S^\star_{L^\star}\},
$$
with each step represented as a set of tool calls,
$$
S^\star_{\ell}=\{c^\star_{\ell,1},\ldots,c^\star_{\ell,n^\star_\ell}\},
$$
and each call written as $c=(\mathrm{tool}(c),\mathrm{args}(c))$. This step-set representation is essential: causal order is enforced across dependent steps, while within-step order can remain unconstrained when calls are independent [2511.17729].

## 2. Benchmark composition and task design

The benchmark spans **28 MCP servers**, **231 tools**, **28 curated multimodal MCP tasks**, **208 evaluation trajectories**, **644 reference steps**, and **1337 MCP tool calls** in those reference trajectories [2511.17729].

| Component | Scale |
|---|---:|
| MCP servers | 28 |
| Tools | 231 |
| Curated multimodal MCP tasks | 28 |
| Evaluation trajectories | 208 |
| Reference steps | 644 |
| MCP tool calls in references | 1337 |

The server ecosystem is deliberately curated to minimize functional overlap and instability. The covered domains include **Academic / Knowledge**, **Science / Space**, **Geography / Travel**, **E-commerce / Finance**, **Office Automation**, **Weather / Air Quality**, and **Health** [2511.17729]. This curation is intended to keep the benchmark reproducible while still exposing agents to heterogeneous tool graphs.

Each task combines an image with a natural-language instruction. Representative tasks include **Items Price**, **Attraction**, **Movies**, **Vision Research**, **Camping**, **Hazard**, **Biology Cut**, **QR Attraction**, **Generate PowerPoint**, **Travel Comparison**, **Heatmap**, **Stock Growth Estimation**, and **NixOS Diagnostic** [2511.17729]. These tasks differ substantially in failure mode: some are dominated by OCR or object recognition, some by cross-domain retrieval chains, and some by resource persistence and document-generation workflows.

The task design explicitly includes persistent artifacts. Later steps may consume files or images created earlier, such as cropped animal images, downloaded linked media, annotated hazard images, or generated presentation files. This makes state persistence part of the benchmarked capability rather than an implementation detail [2511.17729].

## 3. Reference trajectory construction

Reference trajectories are not simple manual annotations. M$^3$-Bench constructs them through an **Executor–Judge pipeline** with human verification [2511.17729].

The **Executor** receives the task prompt, the image, and the MCP tool registry with tool docstrings. It first performs stepwise tool planning, then emits strictly valid JSON tool calls under controlled execution, and finally decides whether enough evidence has been collected to produce the final natural-language reply. The format constrains the interaction to explicit MCP calls while allowing multiple calls in a single step, thereby supporting multi-threaded execution. Tool outputs, including generated file paths and images, persist in a temporary workspace and can be consumed by later steps [2511.17729].

The **Judge** reviews the initial prompt, image, and full Executor dialogue, rejecting trajectories with causal misordering, factual errors, or hallucinated continuation after incorrect tool outputs. When the trajectory is acceptable, the Judge reconstructs an "optimal trajectory" by removing failed or redundant calls, merging operations that could run concurrently into a single step, and preserving only the useful tool name and arguments. That reconstructed trajectory becomes the benchmark reference [2511.17729].

Human verifiers then audit scope, logical consistency, and trajectory quality. The paper reports a pairwise preference study in which participants preferred the human-optimized best trajectory over original MLLM-produced trajectories in **98.16\%** of cases, corresponding to **589/600** comparisons [2511.17729]. A separate calibration study over 40 call pairs selected the similarity thresholds
$$
(\tau_{\mathrm{strong}}, \tau_{\mathrm{weak}}) = (0.8, 0.6),
$$
as best matching human judgments of strong, medium, and weak similarity [2511.17729].

## 4. Alignment procedure and metric suite

M$^3$-Bench is designed to avoid brittle exact matching and to reduce dependence on opaque LLM judges at the call-alignment level. Each tool call is serialized by a canonical function $\phi(\cdot)$, embedded with a sentence encoder $\mathcal{E}$, and compared through cosine similarity [2511.17729]:
$$
S_{ij}=\frac{\langle \mathbf{x}^\star_i,\widehat{\mathbf{x}}_j\rangle}{\lVert \mathbf{x}^\star_i\rVert_2\,\lVert \widehat{\mathbf{x}}_j\rVert_2},
\qquad
\mathbf{x}^\star_i=\mathcal{E}(\phi(c^\star_i)),
\qquad
\widehat{\mathbf{x}}_j=\mathcal{E}(\phi(\hat{c}_j)).
$$

Alignment is restricted to same-tool pairs through tool-name buckets, preventing cross-tool hallucinations from receiving credit. Similarities below $\tau_{\mathrm{weak}}$ are heavily penalized, and within each tool bucket the final correspondence is obtained via Hungarian matching. The result is an auditable one-to-one matched set $M$ between predicted and reference calls [2511.17729].

On top of that alignment, the benchmark reports three metric families.

First, **detection metrics** quantify call coverage and overproduction:
$$
\mathrm{Recall}=\frac{\sum_m |M_m|}{\sum_m N_{\mathrm{gt},m}},
\qquad
\mathrm{Precision}=\frac{\sum_m |M_m|}{\sum_m N_{\mathrm{pred},m}}.
$$
These separate omission of required calls from gratuitous or hallucinated tool use [2511.17729].

Second, **trajectory alignment metrics** evaluate semantic and structural quality. **ArgSim** measures semantic fidelity of matched arguments among confident matches. **StepCoh** measures whether calls from one reference step are unnecessarily scattered across multiple predicted steps, preserving within-step grouping and parallel structure. **MergePur** measures whether distinct causal steps are wrongly merged. **OrdCons** measures inversions in cross-step order among matched calls. To prevent structural metrics from being inflated by systems that match only a few easy calls, M$^3$-Bench also reports recall-covered structural scores in which per-sample structural metrics are weighted by recall [2511.17729].

Third, **LLM judge metrics** assess end-task performance. A four-LLM judge ensemble reports **Task Completion** and **Information Grounding**, with trimmed-mean aggregation after removing the per-instance minimum and maximum judge scores. Task Completion evaluates whether the task was actually solved; Information Grounding evaluates whether the final output is supported by tool-derived evidence rather than unsupported world knowledge [2511.17729].

The benchmark also replays predicted trajectories through an MCP interpreter and classifies each call outcome as **Illegal calling format**, **Unknown Tool Invocation**, **Invalid Invocation Arguments**, **Successful (Resource Not Found)**, or **Successful call**. This replay-based layer makes it possible to distinguish reasoning failures from syntax or schema failures [2511.17729].

## 5. Empirical findings

M$^3$-Bench evaluates representative state-of-the-art MLLMs, including **GLM 4.5v**, **Qwen2.5-VL-72B**, **InternVL 3.5**, **Gemini 2.5 Flash Lite**, **Gemini 2.5 Flash**, **Gemini 2.5 Pro**, **Claude 4.5 Haiku**, **Claude 4.5 Sonnet**, **GPT-5 Nano**, **GPT-5 Mini**, **GPT-5**, **Llama-4-Scout-17B-16E**, **Grok-4 Fast**, and **Grok 4 (0709)** [2511.17729].

| Model | Average score |
|---|---:|
| GPT-5 | 0.482 |
| Gemini 2.5 Pro | 0.423 |
| Grok 4 (0709) | 0.411 |
| GPT-5 Mini | 0.395 |
| Gemini 2.5 Flash | 0.388 |
| Claude 4.5 Sonnet | 0.333 |

The headline result is that even the best systems remain far from saturation. **GPT-5** achieves the highest average score at **0.482**, with **Recall 0.627**, **Precision 0.264**, **ArgSim 0.583**, **StepCoh 0.502**, **OrdCons 0.290**, **MergePur 0.453**, **Task Completion 0.671**, and **Information Grounding 0.657**. **Gemini 2.5 Pro** shows a more balanced precision-recall profile, with **Recall 0.501** and **Precision 0.449**, but lower end-task completion. **Grok 4 (0709)** attains especially strong judge-based end-task metrics, including **Task Completion 0.664** and **Information Grounding 0.638** [2511.17729].

The benchmark reveals persistent weaknesses in **argument fidelity** and **structure consistency**. Smaller or latency-oriented models often exhibit high precision but low recall; for example, **Claude 4.5 Haiku** records **Precision 0.480** but **Recall 0.193**, indicating short, locally correct, under-complete trajectories. Larger models improve recall by attempting more calls, but often over-call tools, as illustrated by **GPT-5** with **Recall 0.627** and **Precision 0.264** [2511.17729].

Replay-based failure analysis identifies recurring error classes. Some models fail at the formatting layer by not producing valid MCP JSON. Others hallucinate tool names, invoking nonexistent tools. A third pattern is correct tool selection with invalid arguments, often leading to HTTP 400 or analogous execution failures. A fourth pattern is syntactically valid invocation of missing backend resources. A fifth pattern is poor image understanding that corrupts the downstream chain; the paper’s case study on **Travel Comparison** shows how incorrect landmark identification poisons later geocoding and AQI retrieval [2511.17729].

Performance is also highly task-dependent. Top systems perform comparatively well on **Barcode Lookup**—for example, **Gemini 2.5 Pro 0.828**, **GPT-5 0.762**, and **Grok 4 0.770**—and on workflow-heavy tasks such as **Biology Cut** (**GPT-5 0.851**), **Generate PowerPoint** (**Gemini 2.5 Pro 0.727**, **GPT-5 0.705**), **Travel Comparison** (**Gemini 2.5 Flash 0.718**, **GPT-5 0.688**), **Heatmap** (**GPT-5 0.662**), and **Stock Growth Estimation** (**Grok 4 0.735**) [2511.17729]. Harder tasks include **Drug**, **MedExcel**, **Vision Research**, **Fruit Cal**, **NixOS Diagnostic**, and some long-horizon planning tasks such as **Camping**, suggesting that cross-domain reasoning, exact argument construction, and conditional branching remain major bottlenecks [2511.17729].

## 6. Benchmark landscape, significance, and disambiguation

Within MCP-oriented evaluation, M$^3$-Bench occupies a distinct niche. The paper’s comparison table states that it is the only listed benchmark jointly supporting an **MCP ecosystem**, **information grounding**, **fuzzy task description**, **multi-hop + threaded workflows**, **multimodality**, and a **similarity metric** [2511.17729]. A closely related benchmark is "MSC-Bench: A Rigorous Benchmark for Multi-Server Tool Orchestration" [2510.19423], which focuses on hierarchical multi-server orchestration, functional overlap, end-to-end retrieval and planning, and robustness in an MCP ecosystem, but does not center multimodal visual grounding or similarity-driven tool-call alignment in the manner of M$^3$-Bench [2510.19423].

The name also requires disambiguation. Several unrelated benchmarks use similar "M3" or "M$^3$" nomenclature: "M$^3$oralBench: A MultiModal Moral Benchmark for LVLMs" [2412.20718], "M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenes" [2410.06678], "M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts" [2312.10763], "M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions" [2606.07402], and "M3-BENCH: Process-Aware Evaluation of LLM Agents Social Behaviors in Mixed-Motive Games" [2601.08462]. These works address moral evaluation, robotics, 3D instruction following, multimodal conversational memory, and social-behavior evaluation, respectively, rather than MCP tool use.

The significance of M$^3$-Bench lies in its reframing of multimodal agent evaluation around the **MCP trajectory**. It evaluates whether an agent can transform visual evidence into valid tool parameters, schedule dependent and parallel calls, preserve intermediate artifacts, and produce a final answer grounded in tool outputs. The benchmark’s metric design is equally central: by distinguishing detection, argument similarity, step coherence, merge purity, order consistency, and judge-based end-task grounding, it supports diagnosis of failure mode rather than only leaderboard ranking [2511.17729].

The paper also makes clear the main limitations. M$^3$-Bench covers a **small curated task set** of 28 tasks and 208 trajectories rather than an exhaustive MCP ecosystem. Its server pool is intentionally compact and curated, which improves reproducibility but may underrepresent open-ecosystem messiness. Reference trajectories are standardized "optimal" trajectories, so some valid alternative workflows may not be fully reflected by a single reference even with fuzzy matching. Task Completion and Information Grounding still depend on an LLM judge ensemble, and similarity-based alignment depends on calibrated thresholds. A plausible implication is that the benchmark is best understood as a high-control diagnostic suite rather than a complete model of open-world MCP deployment [2511.17729].

The benchmark is publicly released through an anonymous repository at **https://github.com/EtaYang10th/Open-M3-Bench** [2511.17729].

Source: https://www.emergentmind.com/topics/m-3-bench