---
title: Prompt Injection Datasets
url: https://www.emergentmind.com/topics/prompt-injection-datasets
type: topic
---

# Prompt Injection Datasets

Prompt injection datasets provide the empirical foundation for evaluating, benchmarking, and strengthening defenses against the manipulation of large language models (LLMs) via adversarial prompts. These corpora encompass both direct and indirect injection attacks, test cases for detection and over-defense assessment, variant generation for robustness benchmarking, and multimodal manipulations relevant to web agent and application security. The construction, diversity, labeling protocols, and analytic methodologies vary considerably between datasets, reflecting the evolving complexity of threat models, defense architectures, and research priorities.

## 1. Dataset Taxonomy and Construction Principles

Prompt injection datasets can be stratified by attack modality, attack source, diversity of content, and annotation protocols:

- **Direct vs. Indirect Attacks**: Many early corpora center on direct attacks, where an adversary operates as the user. Recent benchmarks, notably LLMail-Inject [2506.09956], foreground indirect attacks by embedding adversarial instructions into external data streams, such as emails, retrieved documents, or search results. Datasets such as that built by [2502.16580] systematize direct comparison between clean and injection-modified documents using SQuAD and TriviaQA-derived samples.
- **Manual vs. Automated Generation**: The Maatphor framework [2312.11513] automates prompt variant generation using LLM-guided mutation with a feedback loop, enabling structured coverage over multiple attack styles and increasing diversity beyond hand-crafted samples. Universal gradient-based methods [2403.04957] propose formal loss minimization and enable synthetic injection string generation that generalizes far beyond training examples.
- **Attack Type Representation**: Datasets systematically span override, completion, combined, context manipulation, data exfiltration, cross-context contamination, and obfuscation, as seen in PromptShield [2501.15145], Securing AI Agents [2511.15759], and WAInjectBench [2510.01354].
- **Scale and Scope**: LLMail-Inject is the largest currently, with 208,095 unique attack prompts, whereas curated evaluation sets such as NotInject [2410.22770] focus on a smaller number of carefully filtered benign samples for over-defense diagnosis.

## 2. Schema, Annotation, and Labeling Protocols

Datasets vary in labeling granularity and annotation pipelines:

- **Raw Submission Metadata**: LLMail-Inject stores highly detailed JSON records for every attack submission, including team identifiers, timestamps, scenario, objectives (per email), and LLM outputs. Defense efficacy is tagged at the individual submission level.
- **Binary Classification**: HackAPrompt-sourced datasets [2512.12583, 2501.15145] and the SQuAD/TriviaQA-based indirect injection benchmark [2502.16580] use Boolean malicious/benign labels, sometimes with attack-type metadata.
- **Multilabel Evaluation/Annotations**: Datasets supporting defense benchmarking may include labels for attack success, defense bypass, benign/false-positive (as in NotInject [2410.22770]), LLM-annotator labels (attack vs. unclear vs. clean), and sub-level completion flags.
- **Injection Variant Metadata**: Maatphor and gradient-based universal datasets store iteration indices, variant lineage, template position, success rates, required/forbidden phrases, and methods used for evaluation (string match, similarity).
- **Multimodal Segmentation**: WAInjectBench [2510.01354] organizes both text segments and image samples, annotating for explicit instructions and observer modality (embedded image, screenshot).

## 3. Domain Coverage, Attack Taxonomies, and Defense Benchmarks

Datasets catalog attacks in multiple operational domains and threat scenarios:

- **Email, Web, and Application Workflows**: LLMail-Inject simulates an LLM-based email assistant that interacts with real external emails, driving agents to invoke tool calls under attacker influence.
- **Retrieval-Augmented Generation (RAG) Agents**: Securing AI Agents [2511.15759] delineates five categories (direct injection, context manipulation, override, data exfiltration, cross-context contamination) across customer support, finance, and similar domains.
- **Web Agents and Multimodal Threats**: WAInjectBench structures attacks by attacker goal, capability (HTML injection, image perturbation), and knowledge level, spanning both text (comment, email, interface) and image-based (adversarial perturbation, popups) routes.
- **Benign/Over-Defense Evaluation**: NotInject [2410.22770] tests defense models’ false-positive rates by embedding high-risk trigger words into strictly non-malicious sentences, crossing four topical categories (queries, techniques, virtual creation, multilingual).

## 4. Evaluation Metrics and Empirical Benchmarks

Quantitative metrics underpin dataset utility:

- **Attack Success Rate (ASR)**: Fraction of attack submissions yielding desired adversarial side-effects (unauthorized tool call, injected output string). LLMail-Inject reports ASR ~ 0.8% for Phase 1, declining with defense sophistication.
- **Tool Invocation Rate**: In LLMail-Inject, proportion of trials generating an external action (send_email invocation), used as a sub-level benchmark.
- **Team Success Rate (TSR)**: Measures collective success per defense/scenario; LLMail-Inject reports important inter-defense and inter-LLM TSR deltas (e.g., LLM Judge ≈ 0.32; TaskTracker ≈ 0.44).
- **Defense Recall/Precision/F₁/ROC**: PromptShield, InjecGuard, and related works report standard classification metrics, with particular focus on low-FPR thresholds, crucial for practical deployments with benign-heavy traffic. NotInject provides over-defense accuracy; InjecGuard achieves 87.3% in that regime [2410.22770].
- **Variant Success**: Maatphor computes per-variant success rates under repeated trials, quantifies diversity by n-gram overlap, and measures convergence speed (best variant ≥60% in ≤40 iterations).
- **Removal and Mitigation Efficacy**: The indirect injection defense study [2502.16580] reports both detection TPR/FPR and removal rates for segmentation/extraction models, achieving ≥95–100% removal for certain attack positions and types.

## 5. Access, Licensing, and Practical Utility

Open science principles prevail:

- **Repository and Licensing**: Most datasets are released under permissive licenses (MIT, Apache 2.0), with URLs provided in each paper. LLMail-Inject and WAInjectBench host extensive metadata and code on HuggingFace and GitHub.
- **Data Format**: JSON Lines (.jsonl) or NDJSON is standard, with clearly documented field schemas. Large-scale structured sets support both traditional ML and neural model training pipelines.
- **Benchmark Guidance**: Authors recommend against training production models directly on attack corpora to prevent leakage of adversarial techniques; LLMail-Inject, for instance, warns about overfitting to exploit strategies.
- **Evaluation Use Cases**: Datasets serve to calibrate new detectors, stress-test prompt guardrails, develop end-to-end defense pipelines, assess adaptation and transferability between attack styles, and systematically benchmark model vulnerabilities.

## 6. Research Significance, Findings, and Implications

Prompt injection datasets have driven key advances:

- **Adaptive Adversary Coverage**: LLMail-Inject’s adaptive challenge exposed limitations of “static” defenses and revealed a strong recall boost for ensemble methods (from 0.6 to ≈0.998 on attacks triggering send_email).
- **Variant and Universality**: Maatphor and universal-gradient datasets established the necessity of systematically generated variants; reliance on hand-crafted prompts underestimates attack viability. Automated variants reach ASR levels—static (0.81), semi-dynamic (0.39), dynamic (0.50)—untouchable by manual baselines [2403.04957].
- **Over-Defense Diagnosis**: NotInject’s high trigger word concentration illustrated how prior guardrails falter, dropping benign classification accuracy to near random guessing levels (60%); retrained detectors do markedly better [2410.22770].
- **Modalities and Adaptation**: WAInjectBench demonstrated that image- and text-based attacks require concurrent multimodal detectors; context-agnostic models fail against imperceptible or obfuscated adversarial inputs.
- **Indirect Attack/Defense Cycle**: Segmentation and extraction-based mitigation [2502.16580] push detection research into post-processing and active removal, subsequently driving down attack persistence (ASR).

A plausible implication is that robust, scalable prompt injection defenses are contingent on high-fidelity, diverse, and systematically labeled datasets—especially those representing adaptive adversaries, variants, indirect channels, and over-defense traps—across both unimodal and multimodal domains. Continued open dissemination and method transparency are essential for further research progress and practical safeguard deployment.

Source: https://www.emergentmind.com/topics/prompt-injection-datasets