MCP-AttackBench: Benchmark for MCP Security

• MCP-AttackBench is a large MCP-specific dataset comprising 70,448 samples from public data, real-world metadata, and GPT-4 augmented content targeting LLM–tool interactions.
• The benchmark covers varied attack types including prompt injection, SQL and command injections, with jailbreak instruction attacks dominating the sample pool.
• It underpins the training and evaluation of defenses like MCP-Guard, using rigorous quality controls such as semantic deduplication and manual review for reliable performance metrics.

MCP-AttackBench is a benchmark dataset for security evaluation in Model Context Protocol (MCP) interactions, introduced to support training and evaluation of defenses such as MCP-Guard. It is designed not to collect generic unsafe prompts, but to represent attacks as they appear inside LLM–tool workflows, where threats can be embedded in tool descriptions, metadata, invocation context, or protocol-carried content rather than only in plain user prompts. The benchmark is presented as comprising 70,448 samples, sourced from public datasets, real-world MCP metadata, and GPT-4-generated augmentation, and is intended to cover MCP-specific attack vectors including prompt injection, data exfiltration, command/shell injection, SQL injection, tool-name spoofing, shadow hijack, cross-origin attacks, puppet attacks, <IMPORTANT>-tag attacks, and jailbreak instruction attacks (Xing et al., 14 Aug 2025).

1. Concept and scope

MCP-AttackBench is motivated by the claim that MCP expands the attack surface of LLM systems because attack content may appear in tool descriptions, server metadata, tool names, embedded tags, references to sensitive files or external origins, and protocol-formatted JSON fields. In this view, MCP security differs from ordinary prompt-safety filtering because attacks are often semantic, protocol-aware, and multi-channel rather than confined to a user message (Xing et al., 14 Aug 2025).

The benchmark is therefore positioned as MCP-specific rather than prompt-only. Its coverage spans both classic payload-based attacks and tool-ecosystem manipulations specific to MCP. The categories explicitly named are prompt injection, data exfiltration, command/shell injection, SQL injection, tool-name spoofing, shadow hijack, cross-origin attacks, puppet attacks, <IMPORTANT>-tag attacks, and jailbreak instruction attacks. This suggests a broad notion of MCP attack surfaces that includes instruction override, data theft, tool misuse, protocol-context attacks, and payload attacks (Xing et al., 14 Aug 2025).

A plausible implication is that MCP-AttackBench occupies a distinct niche within MCP evaluation: whereas capability-oriented benchmarks study whether agents can use large tool ecosystems, MCP-AttackBench studies whether MCP-oriented content should be treated as malicious or benign. That contrast is visible in adjacent work such as LiveMCPBench, which is described as a scalability-and-realism benchmark for MCP agents rather than a security benchmark (Mo et al., 3 Aug 2025).

2. Dataset composition and attack categories

The paper gives the total benchmark size as 70,448 samples and reports the following category counts in Table 2 (Xing et al., 14 Aug 2025).

Attack category	Samples
Puppet Attack	100
Shadow Hijack Attack	300
Cross Origin Attack	628
SQL Injection Attack	128
Tool-name Spoofing	88
<IMPORTANT> Tag	40
Command Injection Attack	519
Data-exfiltration attack	147
Prompt Injection Attack	326
Jailbreak Instruction Attack	68,172
Total	70,448

This composition is highly imbalanced, with Jailbreak Instruction Attack: 68,172 dominating the benchmark. The paper explicitly notes that the benchmark is overwhelmingly dominated by jailbreak instruction attacks, so some categories are tiny by comparison (Xing et al., 14 Aug 2025).

The benchmark’s stated source material combines several named resources. It builds on GenTelBench / GenTel-safe for jailbreak instruction attacks, uses the GitHub Command Injection Payload List for command injection examples, the Kaggle SQL Injection Dataset for SQL injection strings, and MCP-Tools corpus / MCP-Tools metadata from MCP-Zero for tool-aware variants and benign samples. GPT-4 is used for synthetic generation of jailbreak instruction attacks, prompt injection attacks, and data exfiltration attacks (Xing et al., 14 Aug 2025).

The category-specific construction details are also partially specified. The paper says it extracts 38 MCP-suitable templates from GenTelBench, uses GPT-4 to generate 1,837 instruction-style attacks across 28 subclasses, and after semantic and conflict checks obtains 68,172 validated template–instruction pairs. It reports 519 command injection examples, 128 SQL injection strings, 326 prompt injection attacks, 147 data exfiltration attacks, 88 tool-name spoofing cases, 40 “IMPORTANT” tag attacks, and 1,028 tool-aware variants comprising 100 Puppet, 300 Shadow Hijack, and 628 Cross Origin cases (Xing et al., 14 Aug 2025).

3. Construction pipeline, MCP formatting, and quality control

The benchmark is described as being transformed into MCP format, but the transformation is only partially specified. The paper states that “validated samples are embedded into MCP tool JSON fields and labeled with attack metadata to simulate diverse threats,” and that MCP-AttackBench simulates attack vectors “specifically tailored to the MCP format.” It also notes that the Stage 3 system prompt asks an LLM to analyze a tool description, which indicates that at least one benchmark view is tool-description-centric (Xing et al., 14 Aug 2025).

The exact MCP serialization, however, is left underspecified. The paper does not fully specify the JSON schema, field names, whether items include separate system, user, and tool channels, or any canonical input formatting string. The benchmark items can therefore be described confidently as MCP-style tool artifacts with attack metadata, but not reconstructed exactly from the paper alone (Xing et al., 14 Aug 2025).

The quality-control pipeline is more concrete. The authors state that they filter out 15% low-quality items using: semantic deduplication with E5 embeddings and cosine similarity > 0.95, manual review with $\kappa > 0.8$, LLM pre-filtering, and outlier detection. For jailbreak construction, they also mention semantic checks and conflict checks. These procedures suggest an effort to remove near-duplicates and low-quality synthetic items, although the paper does not provide more detailed operational definitions for every filter stage (Xing et al., 14 Aug 2025).

The label structure is also partially specified. The main supervised task is binary classification, with 1 = malicious and 0 = benign. At the same time, the benchmark includes category labels and attack metadata, since samples are “labeled with attack metadata” and organized into category counts. The paper does not define whether the stored schema is multiclass, multilabel, or hierarchical, and it does not provide severity levels or span-level annotations (Xing et al., 14 Aug 2025).

4. Experimental role in MCP-Guard

MCP-AttackBench is tightly coupled to MCP-Guard because it serves both as the training corpus for the Stage 2 learnable detector and as the evaluation benchmark for the full defense pipeline. The main trained component is the Stage 2 “MCP-Guard Learnable Detector”, based on a multilingual E5 text embedding model that is fine-tuned using MCP-AttackBench’s training dataset (Xing et al., 14 Aug 2025).

The paper distinguishes between the full benchmark and the subset actually used in experiments. While the full benchmark is reported as 70,448 attack samples, the experiments evaluate on a curated subset of 2,153 adversarial examples spanning 11 categories across 6 threat families, together with 3,105 benign tool descriptions from the MCP-Tools corpus. These are split 80% / 20% stratified at the sub-type level, preserving class balance. The wording around “training/validation” and “full test set” is not fully reconciled in the paper, so the exact train/validation/test protocol remains partially underspecified (Xing et al., 14 Aug 2025).

Within this setup, the paper reports results for Stage 1 pattern-based detectors, the Stage 2 learnable detector, Stage 3 LLM arbitration, the full MCP-Guard pipeline, and competing baselines such as MCP-Scan, SafeMCP, and MCP-Shield. For Stage 2, the paper reports Accuracy: 96.01%, Recall: 93.52%, and F1: 95.06%, and states that full fine-tuning improved Stage 2 accuracy from 65.37% to 96.01%. The exact precision value is inconsistent between table and prose, which the paper itself leaves unresolved in the quoted material (Xing et al., 14 Aug 2025).

The benchmark therefore functions as both a training substrate for semantic attack detection and an empirical basis for comparing end-to-end MCP security pipelines. A plausible implication is that benchmark design choices directly shape the reported strength of learned detectors, especially because the benchmark is centered on tool-description-style inputs rather than only fully interactive MCP traces.

5. Relation to other MCP security benchmarks

MCP-AttackBench belongs to a broader emerging literature on MCP security benchmarking, but its emphasis differs from several adjacent benchmarks. MCPSecBench is described as a systematic security benchmark and playground that identifies 17 attack types across 4 primary attack surfaces and evaluates attacks across three major MCP providers, standardizing evaluation across MCP layers (Yang et al., 17 Aug 2025). By contrast, MCP-AttackBench is presented as a large MCP-oriented attack corpus used primarily for detector training and benchmarking within MCP-Guard (Xing et al., 14 Aug 2025).

MCPTox studies Tool Poisoning Attack, where malicious instructions are embedded within a tool’s metadata without execution, and builds 1312 malicious test cases on 45 live, real-world MCP servers and 353 authentic tools (Wang et al., 19 Aug 2025). “When the Manual Lies” studies Tool Description Poisoning through the MCP-TDP Security Benchmark, with 32 realistic, real-world test cases spanning 6 distinct risk categories (Liu et al., 22 May 2026). MSB, or MCP Security Bench, presents an end-to-end evaluation suite with a taxonomy of 12 attacks, 2,000 attack instances, 10 domains, and 400+ tools, focusing on resistance across planning, invocation, and response handling (Zhang et al., 14 Oct 2025).

These benchmarks are related but not interchangeable. MCP-AttackBench is especially centered on attack examples embedded into MCP tool-centric textual contexts, with binary malicious-versus-benign classification as the main task (Xing et al., 14 Aug 2025). MCPTox and MCP-TDP instead isolate metadata-poisoning attacks in realistic tool ecosystems (Wang et al., 19 Aug 2025, Liu et al., 22 May 2026), while MSB evaluates end-to-end agent robustness under executable MCP attacks (Zhang et al., 14 Oct 2025). This suggests that “MCP-AttackBench” is best understood as one component of a larger MCP security benchmark landscape rather than a complete substitute for all MCP attack evaluations.

6. Limitations, ambiguities, and future directions

The paper leaves several benchmark details underspecified. It does not provide the exact MCP JSON schema, the exact GPT-4 generation prompts and parameters, the exact mapping from the 70,448 full benchmark to the 2,153-adversarial evaluation subset, the precise benign-sample integration procedure, or the exact E5 checkpoint and classifier-head implementation. It also does not report confidence intervals, multiple-seed variance, threshold-search protocol for Stage 2, cross-dataset generalization tests, or out-of-distribution split design (Xing et al., 14 Aug 2025).

Several ambiguities are explicit in the paper’s own reporting. The benign data count of 3,105 benign tool descriptions is discussed separately from the 70,448 total, and the wording suggests that the full benchmark count may refer only to attack-category samples. The paper does not fully resolve whether benign samples are included in that total. Likewise, the curated subset is described as spanning 11 categories across 6 threat families, but the paper does not fully reconcile that family structure with the 10-category breakdown in Table 2 (Xing et al., 14 Aug 2025).

The benchmark is also heavily imbalanced, with jailbreak instruction attacks dominating the sample pool. A plausible implication is that any model trained directly on the full dataset may be influenced strongly by the statistical weight of jailbreak-style content unless additional balancing or stratified evaluation is applied. The paper itself suggests future benchmark extensions by proposing more dynamic attack scenarios and field trials in production systems to test long-term robustness and adaptability (Xing et al., 14 Aug 2025).

Within the broader MCP literature, this leaves a clear research gap between large static or semi-static attack corpora and more execution-grounded, trace-grounded, or environment-grounded benchmarks. That gap is reflected in adjacent work on trace-based auditing, metadata poisoning, and end-to-end attack execution (Wang et al., 19 Aug 2025, Zhang et al., 14 Oct 2025, Hao et al., 23 Apr 2026). MCP-AttackBench remains important because it provides a large MCP-oriented attack dataset, but the paper itself makes clear that full independent reproduction is not yet possible from the published description alone (Xing et al., 14 Aug 2025).