---
title: 'MCPSecBench: MCP Security Benchmark'
url: https://www.emergentmind.com/topics/mcpsecbench
type: topic
---

# MCPSecBench: MCP Security Benchmark

Searching arXiv for MCPSecBench and related MCP security benchmarks.
MCPSecBench is a systematic security benchmark and playground for testing Model Context Protocols (MCPs), introduced as the first systematic, community-driven framework for assessing end-to-end security in MCP-based large language model systems [2508.13220]. It targets the security properties of modern “agent-style” LLM services that use MCPs as well-defined interfaces for tool use, function calling, inter-message state, and structured orchestration. In that setting, added structure enlarges the attack surface: a malicious user or compromised sub-component can hijack request intent, inject rogue instructions, subvert tool invocations, or tamper with orchestration. MCPSecBench addresses the absence of a standard evaluation method by providing a unified threat model, a library of 17 attack types across four primary attack surfaces, a modular test harness, and quantitative metrics for cross-provider benchmarking. Its reported experiments show that over 85% of the identified attacks successfully compromise at least one platform, with core vulnerabilities affecting Claude, OpenAI, and Cursor, while prompt-based and tool-centric attacks vary substantially across hosts and models [2508.13220].

## 1. Scope, motivation, and benchmark role

MCPSecBench is motivated by the transition from single-prompt LLM systems to MCP-based pipelines in which models execute sequences such as “summarize this document,” “run a search,” and “write a report.” In these workflows, MCP acts as a universal, open standard for connecting AI agents with data sources and external tools. The benchmark is therefore framed around security evaluation of the entire MCP call path rather than around isolated prompt attacks.

The benchmark’s role is explicitly end-to-end. It is designed to cover all phases of an MCP call and to compare hardening strategies across different implementations. The framework provides a unified threat model covering all phases of an MCP call, a library of 17 concrete attack types grouped into four high-level surfaces, a modular test harness for driving these attacks against any MCP provider, and a set of quantitative metrics including attack success rate, cross-provider volatility, and per-attack breakdowns [2508.13220].

A common misconception is to treat MCP security as equivalent to prompt injection. MCPSecBench rejects that reduction by formalizing prompt interfaces, tool invocations, communication/state handling, and orchestration/workflow control as distinct security surfaces. This suggests that MCP security is a protocol-layer problem as much as a prompting problem.

## 2. Threat taxonomy

MCPSecBench organizes adversarial behavior into four primary surfaces, with 17 attack types intended to capture realistic malicious goals. Each attack is instantiated with a parametrized payload, a success criterion, and a harm rating [2508.13220].

| Surface | Attack types |
|---|---|
| Prompt Interface Attacks | Instruction Injection, Jailbreak, Context Leakage, Backdoor Activation |
| Tool-Invocation Attacks | Argument Tampering, Call Hijacking, Path Traversal, Arbitrary Code Execution |
| Communication and State Attacks | Metadata Poisoning, Replay, Man-in-the-Middle, Session Fixation |
| Orchestration and Workflow Attacks | Chain-of-Thought Hijacking, Multi-Agent Collusion, Privilege Escalation, Data Exfiltration, Stateful Poisoning |

Within **Surface A. Prompt Interface Attacks**, MCPSecBench defines **Instruction Injection (prompt-in)** as appending or prepending malicious commands to the user’s prompt; **Jailbreak (jailbreak)** as crafting a second-person “you must” style prompt to override safety guardrails; **Context Leakage (leak)** as embedding out-of-scope private data to be echoed back; and **Backdoor Activation (backdoor)** as inserting code words that trigger hidden functionality.

Within **Surface B. Tool-Invocation Attacks**, **Argument Tampering (arg-tamper)** supplies a tool call with malicious parameters; **Call Hijacking (hijack)** intercepts the declared function name to reroute execution; **Path Traversal (path-trv)** passes file paths such as `../../etc/passwd` into file-tool APIs; and **Arbitrary Code Execution (ace)** uses code-evaluation tools such as `python_exec` to run system commands.

Within **Surface C. Communication and State Attacks**, **Metadata Poisoning (meta-poison)** injects malformed headers or JSON fields to confuse the protocol; **Replay (replay)** resubmits previously witnessed messages to violate idempotency; **Man-in-the-Middle (mitm)** alters the payload in transit, simulated by the client wrapper; and **Session Fixation (sess-fix)** forces the server to reuse an old context ID, breaking access controls.

Within **Surface D. Orchestration and Workflow Attacks**, **Chain-of-Thought Hijacking (cot-hijack)** inserts flawed reasoning steps to bias downstream tools; **Multi-Agent Collusion (collude)** coordinates two or more LLMs to leak or escalate privileges; **Privilege Escalation (priv-esc)** persuades the protocol to expose admin-level operations; **Data Exfiltration (exfil)** carves out sensitive tokens or API keys via successive calls; and **Stateful Poisoning (state-poison)** corrupts the running context so that future calls misbehave.

The taxonomy is notable for spanning both message-level and workflow-level compromise. A plausible implication is that MCPSecBench treats MCP security failures as compositional failures in protocol execution, not merely failures of local content moderation.

## 3. Architecture and execution model

MCPSecBench is implemented as a four-module pipeline consisting of a **Prompt Dataset Module**, **MCP Server Wrappers**, an **MCP Client Harness**, and an **Attack Script Library** [2508.13220].

The **Prompt Dataset Module** contains a library of benign “seed” use cases, including document summarization, data analysis, and customer support dialogue, together with adversarial templates that inject attack payloads into each seed. The **MCP Server Wrappers** are thin adapters around provider APIs for Claude, OpenAI, and Cursor, and they normalize call/response formats to a common JSON schema:

```json
{ "context_id": "...", "messages": [{ "role": "...", "content": "..." }], "tool_calls": ["..."] }
```

The **MCP Client Harness** orchestrates the full benchmark loop: it selects a seed prompt, applies an attack template, dispatches the request to a server wrapper, collects the output, records timestamps, logs full transcripts, and computes success or failure. The **Attack Script Library** consists of Python classes implementing the 17 attack types, with each attack script defining two methods: `generate_payload(seed) → (prompt,tool_args)` and `evaluate(response) → Bool`.

The interaction sequence is defined as a fixed seven-step process: load the next seed prompt; choose an attack script; use `script.generate_payload` to merge the malicious snippet into the seed; send the combined message from `ClientHarness` to the `ServerWrapper`; call the real MCP endpoint from the server wrapper; retrieve the response and hand it to `script.evaluate`; and log success while updating metrics [2508.13220].

This architecture distinguishes MCPSecBench from purely static corpora. It functions both as a benchmark and as a “playground” because it couples prompts, protocol wrappers, tool interfaces, and attack implementations in a single execution environment.

## 4. Evaluation methodology and metrics

MCPSecBench evaluates three leading MCP providers—Anthropic Claude, OpenAI GPT-4 with function-call interface, and Cursor.ai—across all 17 attacks and 50 distinct seed contexts. For each $(\text{provider}, \text{attack}, \text{seed})$ triple, it performs 20 repeated trials to account for nondeterminism [2508.13220].

Its primary metric is the **attack success rate**:

$$
S = \frac{\text{number of successful attacks}}{\text{total number of attempts}}.
$$

To summarize across multiple attacks, the benchmark defines a provider-level **vulnerability score**:

$$
V = \frac{1}{|A|}\sum_{a\in A} S_a
$$

where $A$ is the set of chosen attack types. It also computes per-attack 95% Wilson confidence intervals and pairwise differences $\Delta_{i,j} = S_i - S_j$ between providers [2508.13220].

The methodology is explicitly comparative. Because the benchmark normalizes inputs and outputs through server wrappers and applies the same repeated-trial protocol across providers, it is intended to measure both absolute vulnerability and cross-provider volatility. This suggests that MCPSecBench is structured not only to detect failures, but also to localize whether a weakness is pervasive across MCP implementations or contingent on particular hosts, models, or interface decisions.

## 5. Experimental findings

The benchmark reports three broad phenomena in its experiments [2508.13220]. First, **prompt-based attacks dominate**. Across all three platforms, the average success rate for Surface A attacks exceeded 70%, with example values for instruction injection of $S_{\text{prompt\_injection}} = 0.78$ for OpenAI, $0.64$ for Claude, and $0.71$ for Cursor.

Second, **tool-centric attacks are harder but still feasible**. Surface B attacks had a mean $S \approx 0.15$. Within that category, **Argument Tampering** reached $S=0.22$ on OpenAI but only $S=0.08$ on Claude.

Third, the results exhibit **platform-specific vulnerabilities**. OpenAI function calls were reported as most vulnerable to **Path Traversal** with $S=0.31$, which the paper describes as likely due to lenient JSON schema validation. Claude was relatively robust to **Arbitrary Code Execution** with $S=0.04$ but showed high **Metadata Poisoning** success at $S=0.47$, suggesting looser header checks. Cursor had the lowest mean vulnerability score, $V=0.39$, yet remained notably susceptible to **Chain-of-Thought Hijacking** with $S=0.58$.

The benchmark also reports cross-provider contrasts through pairwise differences. For example, the OpenAI-versus-Claude difference on **Jailbreak** attacks is $\Delta = 0.14$ with $p < 0.01$ [2508.13220]. In descriptive terms, the paper’s “Table 1” reports attack success rates by provider and attack category, while “Figure 2” presents a heatmap of $\Delta_{i,j}$.

These findings support two narrower conclusions already embedded in the benchmark design. One is that some vulnerabilities are effectively universal across major MCP hosts. The other is that attack prevalence is highly surface-dependent: prompt-interface weaknesses are common, whereas tool-invocation and protocol-state weaknesses are more heterogeneous.

## 6. Extensibility, research use, and relation to later MCP benchmarks

MCPSecBench is designed to be fully extensible. It supports **custom MCP clients** through an abstract `Client` interface with `send_message`, `receive_message`, and `call_tool`; **new servers/protocols** through additional `ServerWrapper` subclasses, including on-prem models or emerging gRPC-style interfaces; **alternate transports** such as WebSocket or local IPC with minimal changes; and **attack library growth**, where any new attack derived from the base `AttackScript` class is auto-discovered and versioned [2508.13220].

The paper identifies three open challenges for future work: automated fuzzing of protocol schemas, modeling adaptive attackers who can chain multiple attack types, and integrating defensive red-teaming or real-time anomaly detection. These open problems are consistent with MCPSecBench’s framing of MCP security as a systems problem spanning prompt content, tool schemas, message integrity, and state evolution.

Within the later MCP-security literature, MCPSecBench can be situated as an early benchmark emphasizing provider-level end-to-end security across four attack surfaces. Subsequent work broadens the evaluation axis in different directions. **MSB (MCP Security Bench)** emphasizes real malicious tools executed via MCP, evaluates nine LLM agents across 10 domains and 2,000 attack instances, and introduces **Net Resilient Performance (NRP)** to balance utility and security [2510.15994]. **MCP-SafetyBench** extends toward realistic multi-turn, multi-server workflows built on real MCP servers, supports five domains, uses a taxonomy of 20 attack types spanning server, host, and user sides, and reports that vulnerabilities escalate as task horizons and server interactions grow [2512.15163].

A plausible implication is that MCPSecBench established a baseline vocabulary and evaluation structure for MCP security, while later benchmarks increased environmental realism, attack diversity, and multi-server task complexity. In that sense, MCPSecBench occupies a foundational position in the benchmark lineage: it standardizes the evaluation of MCP security and enables rigorous testing across all MCP layers [2508.13220].

Source: https://www.emergentmind.com/topics/mcpsecbench