Papers

Topics

Authors

Recent

View all

AI Research Assistant

Well-researched responses based on relevant abstracts and paper content.

Custom Instructions Pro

Preferences or requirements that you'd like Emergent Mind to consider when generating responses.

Gemini 2.5 Flash

Gemini 2.5 Flash 71 tok/s

Gemini 2.5 Pro 50 tok/s Pro

GPT-5 Medium 21 tok/s Pro

GPT-5 High 19 tok/s Pro

GPT-4o 91 tok/s Pro

Kimi K2 164 tok/s Pro

GPT OSS 120B 449 tok/s Pro

Claude Sonnet 4 36 tok/s Pro

2000 character limit reached

MCPToolBench++: LLM Tool-Use Benchmark

Updated 9 September 2025

MCPToolBench++ is a comprehensive benchmark framework that evaluates LLMs' ability to invoke standardized MCP tools in both single-step and multi-hop tasks.
It leverages a diverse dataset of 1,500 queries spanning 40+ real-world tool categories to simulate realistic tool usage scenarios.
Robust metrics such as AST, DAG accuracy, Pass@K, and Tool Call Success Rate quantify both structural planning and execution accuracy.

MCPToolBench++ is a large-scale, multi-domain benchmark framework designed to rigorously evaluate LLMs and AI agents on their ability to invoke and use tools standardized under the Model Context Protocol (MCP). It addresses key challenges in benchmarking tool use for LLMs—including the scarcity of unified datasets, variability in tool interface schemas and response formats, and the token overhead required to represent complex toolchains in-context—by building upon a marketplace of over 4,000 MCP servers and 40+ real-world tool categories. The framework systematically measures agent capabilities in both single-step and multi-step tool use across diverse domains and provides a set of robust, protocol-compliant evaluation metrics tailored to the operational realities and complexities of MCP-based tool invocation (Fan et al., 11 Aug 2025).

1. Objectives and Scope

MCPToolBench++ is devised to fill critical gaps identified in prior tool-use benchmarks:

Unified Evaluation: It provides a protocol-driven, end-to-end testbed for assessing tool use capabilities across major functional areas—including web search, map routing, finance, payment, browser automation, and filesystem interaction.
Real-World Grounding: The benchmark integrates real MCP tool deployments and configurations from public marketplaces and open-source repositories, ensuring that both tool schemas and response payloads reflect real-world operational diversity and reliability variation.
Contextual Complexity: Unlike purely synthetic or domain-isolated benchmarks, MCPToolBench++ includes multi-category, multi-hop tool usage tasks, capturing realistic operational chains and dependencies.
Token-Scoped Design: To cope with LLM context length constraints, the benchmark employs subsampling via an explicit Tool Dispatcher, reducing the number of tool schemas presented per run while retaining sufficient diversity and coverage.

The framework's core dataset consists of approximately 1,500 question-answer pairs sampled over the full tool ecosystem, with queries spanning single-step (atomic tool calls) and multi-step (directed acyclic graph–structured call chains) scenarios.

2. Dataset Structure and Tool Categorization

MCPToolBench++ curates metadata, configuration files (e.g., mcp_config.json, tool_schema.json), and canonical schemas from a wide array of MCP marketplaces and associated GitHub communities. This corpus is organized into:

40+ Tool Categories: Domains include search, browser operations, finance, payment, geospatial (maps), file systems, and more.
Task Taxonomy: The dataset distinguishes between single-step tool invocations (e.g., get_weather, maps_directions) sampled per category and multi-step tool calls that may require composing up to ten sequential and/or parallel operations, with multi-hop chains manually validated for correctness and coverage.

Sampling is designed to maximize coverage:

Intra-category sampling avoids duplicates via sampling without replacement.
Cross-category composition in multi-step chains ensures realistic workflows and diverse tool combinations.

The pipeline generalizes across languages and platforms, supporting multilingual queries and schema documentation.

3. Benchmark Design and Evaluation Protocols

The evaluation framework combines static plan validation with dynamic, real-world tool execution checks:

AST Accuracy: Using the Abstract Syntax Tree (AST) metric, it validates structural correspondence between the agent's predicted tool invocation plan (tool name, parameter coverage, types) and the canonical ground truth—capturing errors in tool selection and argument construction.
AST DAG Accuracy: For multi-step and parallelized tasks, predicted and reference tool call plans are represented as DAGs. The metric computes correctness by matching terminal and dependent nodes, using binary labels.
Pass@K Accuracy: This dynamic metric evaluates end-to-end execution success—measuring whether the produced tool call sequence, when executed, yields correct outputs (valid parameters, non-empty and accurate responses).
Tool Call Success Rate: This directly measures operation status codes and payloads, benchmarking agents' ability to handle heterogeneity and error codes across MCP implementations.

Each tool call is subjected to multiple execution trials to ensure robustness given the inherent variability of external APIs.

The formal computational complexity for processing all tool schemas is $O(M \times N_t \times T_{tool})$ , where $M$ is the number of MCP servers, $N_t$ the average tools per server, and $T_{tool}$ the token length per schema. After sampling via the Tool Dispatcher, this is reduced to $O(M \times N_k \times T_{tool})$ with $N_k \approx 10$ (Fan et al., 11 Aug 2025).

4. Model Benchmarking and Error Analysis

MCPToolBench++ enables detailed comparative evaluation across state-of-the-art LLMs (GPT-4, Claude, Qwen, Kimi, among others) with respect to tool use acumen:

Category Leadership: Performance varies by domain. For example, Qwen3-coder leads in AST accuracy for Browser and Map tools, while Qwen2.5-max excels in File System and Finance. Execution-level (Pass@K) leaders are domain-dependent (GPT-4 in Map and Finance; Claude-3.7-Sonnet in Search).
Failure surface: There is a notable divergence between syntactic correctness (AST) and execution success (Pass@K), exposing practical issues such as:
- Parameterization errors
- API-level incompatibilities and error returns
- Tool selection ambiguity (where multiple similar tools exist)
- Partial completions in DAG-based multi-step plans
- Context window limitations leading to omitted required schemas

Reliability is further affected by MCP service provider stability and interface fidelity.

Metric	Evaluation Focus	Observed Challenges
AST Accuracy	Plan structure and argument syntax	Surplus/missing tool calls, mismatched params
AST DAG Accuracy	Multi-step/dag chain matching	Partial ordering errors, parallel step mismatches
Pass@K Accuracy	End-to-end execution correctness	Parameteric errors, API errors
Tool Call Success Rate	API-level result verification	Status code errors, empty or malformed results

5. Technical Innovations and Pipeline Automation

The MCPToolBench++ pipeline encompasses automated schema ingestion, stratified tool/query sampling, LLM-driven synthetic query generation, and post-filtering via semantic and grammatical reasonableness checks. Notable aspects include:

Query Generation: LLM-based template filling combines parameter slot dictionaries, code-driven data population, and subsequent rewriting for grammatical/naturalness assurances.
Multilinguality: The data pipeline is capable of multi-language task formulation, generalizing to various linguistic and cultural contexts.
Reduction of Overhead: Through dynamic tool dispatching and narrowing schema presentation, the benchmark circumvents context length constraints while preserving functional diversity (Fan et al., 11 Aug 2025).
Validity Filtering: Post-processing eliminates logically inconsistent or low-quality queries, resulting in a high-quality, diagnostically rich dataset.

6. Comparative Context and Integration with Other MCP Benchmarks

MCPToolBench++ integrates lessons and advances from related MCP benchmarking and ecosystem research:

Relation to MCP-RADAR (Gao et al., 22 May 2025): Both frameworks deploy multi-dimensional profiling, but MCPToolBench++ uniquely emphasizes large-scale, heterogeneous, and dynamic tool environments and the structural/planning facets of tool-oriented reasoning.
Extension over MCP-Bench (Wang et al., 28 Aug 2025): While MCP-Bench targets fuzzy, naturally underspecified instructions and multi-hop planning, MCPToolBench++ systematically covers both single-step and complex DAG workflows, emphasizing end-to-end invocation accuracy and error analysis across a broader range of real-world MCP artifacts.
Data Foundation from MCPCorpus (Lin et al., 30 Jun 2025): Tool and server metadata are harmonized with MCPCorpus, benefiting from its standardized, up-to-date snapshot of the rapidly evolving MCP tool landscape and its associated analytics.
Alignment with MCPWorld (Yan et al., 9 Jun 2025) and LiveMCPBench (Mo et al., 3 Aug 2025): MCPToolBench++ complements these frameworks by providing direct, protocol-level analysis of tool invocation chains, with a focus on structural plan fidelity and large-scale operational coverage.

7. Limitations and Future Directions

MCPToolBench++ exposes substantial remaining challenges:

Reliability Gaps: Even with structurally correct predictions, actual execution is hampered by parameter mismatches, API idiosyncrasies, and incomplete tool coverage across providers.
Context Window Constraints: As task complexity increases (more tool steps, longer schema descriptions), LLMs exhibit degraded planning fidelity due to prompt size limitations, despite dispatcher optimizations.
Need for Advanced Evaluation: Future work is directed toward further granularity of evaluation metrics, deeper error analysis, and integration of human-in-the-loop assessment to benchmark nuanced agentic behavior on ambiguous or contextually complex queries.
Adaptivity and Scalability: Next-generation iterations will aim to dynamically adapt tool dispatching, enhance multilingual task synthesis, and increase coverage by leveraging continuously updated datasets such as MCPCorpus.

A plausible implication is that as the MCP ecosystem proliferates, maintaining standardized and protocol-compliant benchmarks like MCPToolBench++, with dynamic integration of new tool and server schemas, will be central to advancing both LLM agentic reasoning and the practical robustness of real-world tool-oriented AI.