---
title: 'MCP-AttackBench: Benchmark for MCP Security'
url: https://www.emergentmind.com/topics/mcp-attackbench
type: topic
---

# MCP-AttackBench: Benchmark for MCP Security

MCP-AttackBench is a benchmark dataset for security evaluation in Model Context Protocol (MCP) interactions, introduced to support training and evaluation of defenses such as MCP-Guard. It is designed not to collect generic unsafe prompts, but to represent attacks as they appear inside LLM–tool workflows, where threats can be embedded in tool descriptions, metadata, invocation context, or protocol-carried content rather than only in plain user prompts. The benchmark is presented as comprising **70,448 samples**, sourced from **public datasets**, **real-world MCP metadata**, and **GPT-4-generated augmentation**, and is intended to cover MCP-specific attack vectors including prompt injection, data exfiltration, command/shell injection, SQL injection, tool-name spoofing, shadow hijack, cross-origin attacks, puppet attacks, `<IMPORTANT>`-tag attacks, and jailbreak instruction attacks [2508.10991].

## 1. Concept and scope

MCP-AttackBench is motivated by the claim that MCP expands the attack surface of LLM systems because attack content may appear in **tool descriptions**, **server metadata**, **tool names**, **embedded tags**, **references to sensitive files or external origins**, and **protocol-formatted JSON fields**. In this view, MCP security differs from ordinary prompt-safety filtering because attacks are often **semantic, protocol-aware, and multi-channel** rather than confined to a user message [2508.10991].

The benchmark is therefore positioned as MCP-specific rather than prompt-only. Its coverage spans both classic payload-based attacks and tool-ecosystem manipulations specific to MCP. The categories explicitly named are prompt injection, data exfiltration, command/shell injection, SQL injection, tool-name spoofing, shadow hijack, cross-origin attacks, puppet attacks, `<IMPORTANT>`-tag attacks, and jailbreak instruction attacks. This suggests a broad notion of MCP attack surfaces that includes instruction override, data theft, tool misuse, protocol-context attacks, and payload attacks [2508.10991].

A plausible implication is that MCP-AttackBench occupies a distinct niche within MCP evaluation: whereas capability-oriented benchmarks study whether agents can use large tool ecosystems, MCP-AttackBench studies whether MCP-oriented content should be treated as malicious or benign. That contrast is visible in adjacent work such as LiveMCPBench, which is described as a scalability-and-realism benchmark for MCP agents rather than a security benchmark [2508.01780].

## 2. Dataset composition and attack categories

The paper gives the total benchmark size as **70,448 samples** and reports the following category counts in Table 2 [2508.10991].

| Attack category | Samples |
|---|---:|
| Puppet Attack | 100 |
| Shadow Hijack Attack | 300 |
| Cross Origin Attack | 628 |
| SQL Injection Attack | 128 |
| Tool-name Spoofing | 88 |
| `<IMPORTANT>` Tag | 40 |
| Command Injection Attack | 519 |
| Data-exfiltration attack | 147 |
| Prompt Injection Attack | 326 |
| Jailbreak Instruction Attack | 68,172 |
| Total | 70,448 |

This composition is highly imbalanced, with **Jailbreak Instruction Attack: 68,172** dominating the benchmark. The paper explicitly notes that the benchmark is overwhelmingly dominated by jailbreak instruction attacks, so some categories are tiny by comparison [2508.10991].

The benchmark’s stated source material combines several named resources. It builds on **GenTelBench / GenTel-safe** for jailbreak instruction attacks, uses the **GitHub Command Injection Payload List** for command injection examples, the **Kaggle SQL Injection Dataset** for SQL injection strings, and **MCP-Tools corpus / MCP-Tools metadata** from **MCP-Zero** for tool-aware variants and benign samples. GPT-4 is used for synthetic generation of jailbreak instruction attacks, prompt injection attacks, and data exfiltration attacks [2508.10991].

The category-specific construction details are also partially specified. The paper says it extracts **38 MCP-suitable templates** from GenTelBench, uses GPT-4 to generate **1,837 instruction-style attacks** across **28 subclasses**, and after **semantic and conflict checks** obtains **68,172 validated template–instruction pairs**. It reports **519 command injection** examples, **128 SQL injection** strings, **326 prompt injection** attacks, **147 data exfiltration** attacks, **88 tool-name spoofing** cases, **40 “IMPORTANT” tag attacks**, and **1,028** tool-aware variants comprising **100 Puppet**, **300 Shadow Hijack**, and **628 Cross Origin** cases [2508.10991].

## 3. Construction pipeline, MCP formatting, and quality control

The benchmark is described as being transformed into **MCP format**, but the transformation is only partially specified. The paper states that “validated samples are **embedded into MCP tool JSON fields** and labeled with attack metadata to simulate diverse threats,” and that MCP-AttackBench simulates attack vectors “**specifically tailored to the MCP format**.” It also notes that the Stage 3 system prompt asks an LLM to analyze a **tool description**, which indicates that at least one benchmark view is tool-description-centric [2508.10991].

The exact MCP serialization, however, is left underspecified. The paper does not fully specify the JSON schema, field names, whether items include separate `system`, `user`, and `tool` channels, or any canonical input formatting string. The benchmark items can therefore be described confidently as MCP-style tool artifacts with attack metadata, but not reconstructed exactly from the paper alone [2508.10991].

The quality-control pipeline is more concrete. The authors state that they **filter out 15% low-quality items** using: **semantic deduplication** with **E5 embeddings** and **cosine similarity > 0.95**, **manual review** with **\(\kappa > 0.8\)**, **LLM pre-filtering**, and **outlier detection**. For jailbreak construction, they also mention **semantic checks** and **conflict checks**. These procedures suggest an effort to remove near-duplicates and low-quality synthetic items, although the paper does not provide more detailed operational definitions for every filter stage [2508.10991].

The label structure is also partially specified. The main supervised task is **binary classification**, with **1 = malicious** and **0 = benign**. At the same time, the benchmark includes category labels and attack metadata, since samples are “labeled with attack metadata” and organized into category counts. The paper does not define whether the stored schema is multiclass, multilabel, or hierarchical, and it does not provide severity levels or span-level annotations [2508.10991].

## 4. Experimental role in MCP-Guard

MCP-AttackBench is tightly coupled to MCP-Guard because it serves both as the **training corpus** for the Stage 2 learnable detector and as the **evaluation benchmark** for the full defense pipeline. The main trained component is the Stage 2 **“MCP-Guard Learnable Detector”**, based on a **multilingual E5 text embedding model** that is fine-tuned using MCP-AttackBench’s training dataset [2508.10991].

The paper distinguishes between the full benchmark and the subset actually used in experiments. While the full benchmark is reported as **70,448** attack samples, the experiments evaluate on a curated subset of **2,153 adversarial examples** spanning **11 categories** across **6 threat families**, together with **3,105 benign tool descriptions** from the MCP-Tools corpus. These are split **80% / 20% stratified at the sub-type level**, preserving class balance. The wording around “training/validation” and “full test set” is not fully reconciled in the paper, so the exact train/validation/test protocol remains partially underspecified [2508.10991].

Within this setup, the paper reports results for Stage 1 pattern-based detectors, the Stage 2 learnable detector, Stage 3 LLM arbitration, the full MCP-Guard pipeline, and competing baselines such as MCP-Scan, SafeMCP, and MCP-Shield. For Stage 2, the paper reports **Accuracy: 96.01%**, **Recall: 93.52%**, and **F1: 95.06%**, and states that full fine-tuning improved Stage 2 accuracy from **65.37%** to **96.01%**. The exact precision value is inconsistent between table and prose, which the paper itself leaves unresolved in the quoted material [2508.10991].

The benchmark therefore functions as both a training substrate for semantic attack detection and an empirical basis for comparing end-to-end MCP security pipelines. A plausible implication is that benchmark design choices directly shape the reported strength of learned detectors, especially because the benchmark is centered on tool-description-style inputs rather than only fully interactive MCP traces.

## 5. Relation to other MCP security benchmarks

MCP-AttackBench belongs to a broader emerging literature on MCP security benchmarking, but its emphasis differs from several adjacent benchmarks. MCPSecBench is described as a systematic security benchmark and playground that identifies **17 attack types across 4 primary attack surfaces** and evaluates attacks across three major MCP providers, standardizing evaluation across MCP layers [2508.13220]. By contrast, MCP-AttackBench is presented as a large MCP-oriented attack corpus used primarily for detector training and benchmarking within MCP-Guard [2508.10991].

MCPTox studies **Tool Poisoning Attack**, where malicious instructions are embedded within a tool’s metadata without execution, and builds **1312 malicious test cases** on **45 live, real-world MCP servers** and **353 authentic tools** [2508.14925]. “When the Manual Lies” studies **Tool Description Poisoning** through the **MCP-TDP Security Benchmark**, with **32 realistic, real-world test cases** spanning **6 distinct risk categories** [2605.24069]. MSB, or **MCP Security Bench**, presents an end-to-end evaluation suite with a taxonomy of **12 attacks**, **2,000 attack instances**, **10 domains**, and **400+ tools**, focusing on resistance across planning, invocation, and response handling [2510.15994].

These benchmarks are related but not interchangeable. MCP-AttackBench is especially centered on attack examples embedded into MCP tool-centric textual contexts, with binary malicious-versus-benign classification as the main task [2508.10991]. MCPTox and MCP-TDP instead isolate metadata-poisoning attacks in realistic tool ecosystems [2508.14925; 2605.24069], while MSB evaluates end-to-end agent robustness under executable MCP attacks [2510.15994]. This suggests that “MCP-AttackBench” is best understood as one component of a larger MCP security benchmark landscape rather than a complete substitute for all MCP attack evaluations.

## 6. Limitations, ambiguities, and future directions

The paper leaves several benchmark details underspecified. It does not provide the exact MCP JSON schema, the exact GPT-4 generation prompts and parameters, the exact mapping from the **70,448** full benchmark to the **2,153-adversarial** evaluation subset, the precise benign-sample integration procedure, or the exact E5 checkpoint and classifier-head implementation. It also does not report confidence intervals, multiple-seed variance, threshold-search protocol for Stage 2, cross-dataset generalization tests, or out-of-distribution split design [2508.10991].

Several ambiguities are explicit in the paper’s own reporting. The benign data count of **3,105 benign tool descriptions** is discussed separately from the **70,448** total, and the wording suggests that the full benchmark count may refer only to attack-category samples. The paper does not fully resolve whether benign samples are included in that total. Likewise, the curated subset is described as spanning **11 categories across 6 threat families**, but the paper does not fully reconcile that family structure with the 10-category breakdown in Table 2 [2508.10991].

The benchmark is also heavily imbalanced, with jailbreak instruction attacks dominating the sample pool. A plausible implication is that any model trained directly on the full dataset may be influenced strongly by the statistical weight of jailbreak-style content unless additional balancing or stratified evaluation is applied. The paper itself suggests future benchmark extensions by proposing **more dynamic attack scenarios** and **field trials in production systems** to test long-term robustness and adaptability [2508.10991].

Within the broader MCP literature, this leaves a clear research gap between large static or semi-static attack corpora and more execution-grounded, trace-grounded, or environment-grounded benchmarks. That gap is reflected in adjacent work on trace-based auditing, metadata poisoning, and end-to-end attack execution [2508.14925; 2510.15994; 2604.21477]. MCP-AttackBench remains important because it provides a large MCP-oriented attack dataset, but the paper itself makes clear that full independent reproduction is not yet possible from the published description alone [2508.10991].

Source: https://www.emergentmind.com/topics/mcp-attackbench