MCPSecBench: MCP Security Benchmark

• MCPSecBench is a systematic security benchmark for MCP-based systems, featuring a unified threat model and 17 attack types to assess protocol vulnerabilities.
• It provides a modular test harness that simulates end-to-end MCP call flows, enabling quantitative comparisons across platforms like Claude, OpenAI, and Cursor.
• The framework reveals that prompt-interface attacks dominate overall vulnerabilities, while tool-invocation and orchestration attacks exhibit varying success rates.

Searching arXiv for MCPSecBench and related MCP security benchmarks. MCPSecBench is a systematic security benchmark and playground for testing Model Context Protocols (MCPs), introduced as the first systematic, community-driven framework for assessing end-to-end security in MCP-based LLM systems (Yang et al., 17 Aug 2025). It targets the security properties of modern “agent-style” LLM services that use MCPs as well-defined interfaces for tool use, function calling, inter-message state, and structured orchestration. In that setting, added structure enlarges the attack surface: a malicious user or compromised sub-component can hijack request intent, inject rogue instructions, subvert tool invocations, or tamper with orchestration. MCPSecBench addresses the absence of a standard evaluation method by providing a unified threat model, a library of 17 attack types across four primary attack surfaces, a modular test harness, and quantitative metrics for cross-provider benchmarking. Its reported experiments show that over 85% of the identified attacks successfully compromise at least one platform, with core vulnerabilities affecting Claude, OpenAI, and Cursor, while prompt-based and tool-centric attacks vary substantially across hosts and models (Yang et al., 17 Aug 2025).

1. Scope, motivation, and benchmark role

MCPSecBench is motivated by the transition from single-prompt LLM systems to MCP-based pipelines in which models execute sequences such as “summarize this document,” “run a search,” and “write a report.” In these workflows, MCP acts as a universal, open standard for connecting AI agents with data sources and external tools. The benchmark is therefore framed around security evaluation of the entire MCP call path rather than around isolated prompt attacks.

The benchmark’s role is explicitly end-to-end. It is designed to cover all phases of an MCP call and to compare hardening strategies across different implementations. The framework provides a unified threat model covering all phases of an MCP call, a library of 17 concrete attack types grouped into four high-level surfaces, a modular test harness for driving these attacks against any MCP provider, and a set of quantitative metrics including attack success rate, cross-provider volatility, and per-attack breakdowns (Yang et al., 17 Aug 2025).

A common misconception is to treat MCP security as equivalent to prompt injection. MCPSecBench rejects that reduction by formalizing prompt interfaces, tool invocations, communication/state handling, and orchestration/workflow control as distinct security surfaces. This suggests that MCP security is a protocol-layer problem as much as a prompting problem.

2. Threat taxonomy

MCPSecBench organizes adversarial behavior into four primary surfaces, with 17 attack types intended to capture realistic malicious goals. Each attack is instantiated with a parametrized payload, a success criterion, and a harm rating (Yang et al., 17 Aug 2025).

Surface	Attack types
Prompt Interface Attacks	Instruction Injection, Jailbreak, Context Leakage, Backdoor Activation
Tool-Invocation Attacks	Argument Tampering, Call Hijacking, Path Traversal, Arbitrary Code Execution
Communication and State Attacks	Metadata Poisoning, Replay, Man-in-the-Middle, Session Fixation
Orchestration and Workflow Attacks	Chain-of-Thought Hijacking, Multi-Agent Collusion, Privilege Escalation, Data Exfiltration, Stateful Poisoning

Within Surface A. Prompt Interface Attacks, MCPSecBench defines Instruction Injection (prompt-in) as appending or prepending malicious commands to the user’s prompt; Jailbreak (jailbreak) as crafting a second-person “you must” style prompt to override safety guardrails; Context Leakage (leak) as embedding out-of-scope private data to be echoed back; and Backdoor Activation (backdoor) as inserting code words that trigger hidden functionality.

Within Surface B. Tool-Invocation Attacks, Argument Tampering (arg-tamper) supplies a tool call with malicious parameters; Call Hijacking (hijack) intercepts the declared function name to reroute execution; Path Traversal (path-trv) passes file paths such as ../../etc/passwd into file-tool APIs; and Arbitrary Code Execution (ace) uses code-evaluation tools such as python_exec to run system commands.

Within Surface C. Communication and State Attacks, Metadata Poisoning (meta-poison) injects malformed headers or JSON fields to confuse the protocol; Replay (replay) resubmits previously witnessed messages to violate idempotency; Man-in-the-Middle (mitm) alters the payload in transit, simulated by the client wrapper; and Session Fixation (sess-fix) forces the server to reuse an old context ID, breaking access controls.

Within Surface D. Orchestration and Workflow Attacks, Chain-of-Thought Hijacking (cot-hijack) inserts flawed reasoning steps to bias downstream tools; Multi-Agent Collusion (collude) coordinates two or more LLMs to leak or escalate privileges; Privilege Escalation (priv-esc) persuades the protocol to expose admin-level operations; Data Exfiltration (exfil) carves out sensitive tokens or API keys via successive calls; and Stateful Poisoning (state-poison) corrupts the running context so that future calls misbehave.

The taxonomy is notable for spanning both message-level and workflow-level compromise. A plausible implication is that MCPSecBench treats MCP security failures as compositional failures in protocol execution, not merely failures of local content moderation.

3. Architecture and execution model

MCPSecBench is implemented as a four-module pipeline consisting of a Prompt Dataset Module, MCP Server Wrappers, an MCP Client Harness, and an Attack Script Library (Yang et al., 17 Aug 2025).

The Prompt Dataset Module contains a library of benign “seed” use cases, including document summarization, data analysis, and customer support dialogue, together with adversarial templates that inject attack payloads into each seed. The MCP Server Wrappers are thin adapters around provider APIs for Claude, OpenAI, and Cursor, and they normalize call/response formats to a common JSON schema:

$$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$9

The MCP Client Harness orchestrates the full benchmark loop: it selects a seed prompt, applies an attack template, dispatches the request to a server wrapper, collects the output, records timestamps, logs full transcripts, and computes success or failure. The Attack Script Library consists of Python classes implementing the 17 attack types, with each attack script defining two methods: generate_payload(seed) → (prompt,tool_args) and evaluate(response) → Bool.

The interaction sequence is defined as a fixed seven-step process: load the next seed prompt; choose an attack script; use script.generate_payload to merge the malicious snippet into the seed; send the combined message from ClientHarness to the ServerWrapper; call the real MCP endpoint from the server wrapper; retrieve the response and hand it to script.evaluate; and log success while updating metrics (Yang et al., 17 Aug 2025).

This architecture distinguishes MCPSecBench from purely static corpora. It functions both as a benchmark and as a “playground” because it couples prompts, protocol wrappers, tool interfaces, and attack implementations in a single execution environment.

4. Evaluation methodology and metrics

MCPSecBench evaluates three leading MCP providers—Anthropic Claude, OpenAI GPT-4 with function-call interface, and Cursor.ai—across all 17 attacks and 50 distinct seed contexts. For each $(\text{provider}, \text{attack}, \text{seed})$ triple, it performs 20 repeated trials to account for nondeterminism (Yang et al., 17 Aug 2025).

Its primary metric is the attack success rate:

$$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$

To summarize across multiple attacks, the benchmark defines a provider-level vulnerability score:

$V = \frac{1}{|A|}\sum_{a\in A} S_a$

where $A$ is the set of chosen attack types. It also computes per-attack 95% Wilson confidence intervals and pairwise differences $\Delta_{i,j} = S_i - S_j$ between providers (Yang et al., 17 Aug 2025).

The methodology is explicitly comparative. Because the benchmark normalizes inputs and outputs through server wrappers and applies the same repeated-trial protocol across providers, it is intended to measure both absolute vulnerability and cross-provider volatility. This suggests that MCPSecBench is structured not only to detect failures, but also to localize whether a weakness is pervasive across MCP implementations or contingent on particular hosts, models, or interface decisions.

5. Experimental findings

The benchmark reports three broad phenomena in its experiments (Yang et al., 17 Aug 2025). First, prompt-based attacks dominate. Across all three platforms, the average success rate for Surface A attacks exceeded 70%, with example values for instruction injection of $S_{\text{prompt\_injection}} = 0.78$ for OpenAI, $0.64$ for Claude, and $0.71$ for Cursor.

Second, tool-centric attacks are harder but still feasible. Surface B attacks had a mean $S \approx 0.15$. Within that category, Argument Tampering reached $S=0.22$ on OpenAI but only $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$0 on Claude.

Third, the results exhibit platform-specific vulnerabilities. OpenAI function calls were reported as most vulnerable to Path Traversal with $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$1, which the paper describes as likely due to lenient JSON schema validation. Claude was relatively robust to Arbitrary Code Execution with $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$2 but showed high Metadata Poisoning success at $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$3, suggesting looser header checks. Cursor had the lowest mean vulnerability score, $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$4, yet remained notably susceptible to Chain-of-Thought Hijacking with $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$5.

The benchmark also reports cross-provider contrasts through pairwise differences. For example, the OpenAI-versus-Claude difference on Jailbreak attacks is $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$6 with $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$7 (Yang et al., 17 Aug 2025). In descriptive terms, the paper’s “Table 1” reports attack success rates by provider and attack category, while “Figure 1” presents a heatmap of $$S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.$$8.

These findings support two narrower conclusions already embedded in the benchmark design. One is that some vulnerabilities are effectively universal across major MCP hosts. The other is that attack prevalence is highly surface-dependent: prompt-interface weaknesses are common, whereas tool-invocation and protocol-state weaknesses are more heterogeneous.

6. Extensibility, research use, and relation to later MCP benchmarks

MCPSecBench is designed to be fully extensible. It supports custom MCP clients through an abstract Client interface with send_message, receive_message, and call_tool; new servers/protocols through additional ServerWrapper subclasses, including on-prem models or emerging gRPC-style interfaces; alternate transports such as WebSocket or local IPC with minimal changes; and attack library growth, where any new attack derived from the base AttackScript class is auto-discovered and versioned (Yang et al., 17 Aug 2025).

The paper identifies three open challenges for future work: automated fuzzing of protocol schemas, modeling adaptive attackers who can chain multiple attack types, and integrating defensive red-teaming or real-time anomaly detection. These open problems are consistent with MCPSecBench’s framing of MCP security as a systems problem spanning prompt content, tool schemas, message integrity, and state evolution.

Within the later MCP-security literature, MCPSecBench can be situated as an early benchmark emphasizing provider-level end-to-end security across four attack surfaces. Subsequent work broadens the evaluation axis in different directions. MSB (MCP Security Bench) emphasizes real malicious tools executed via MCP, evaluates nine LLM agents across 10 domains and 2,000 attack instances, and introduces Net Resilient Performance (NRP) to balance utility and security (Zhang et al., 14 Oct 2025). MCP-SafetyBench extends toward realistic multi-turn, multi-server workflows built on real MCP servers, supports five domains, uses a taxonomy of 20 attack types spanning server, host, and user sides, and reports that vulnerabilities escalate as task horizons and server interactions grow (Zong et al., 17 Dec 2025).

A plausible implication is that MCPSecBench established a baseline vocabulary and evaluation structure for MCP security, while later benchmarks increased environmental realism, attack diversity, and multi-server task complexity. In that sense, MCPSecBench occupies a foundational position in the benchmark lineage: it standardizes the evaluation of MCP security and enables rigorous testing across all MCP layers (Yang et al., 17 Aug 2025).