Prompt Injection Attacks (PIAs)

• Prompt injection attacks (PIAs) are adversarial techniques that insert malicious instructions into LLM inputs to override intended tasks.
• They exploit LLMs' inability to differentiate between trusted and injected prompts, leading to actions like data exfiltration and directive overrides.
• Empirical studies show high attack success rates in vulnerable models, driving research into robust detection and mitigation strategies.

A prompt injection attack (PIA) is the adversarial insertion of a crafted instruction or data fragment into the input context presented to a LLM or an LLM-integrated application, such that the model executes the attacker’s instruction instead of its intended, developer- or user-specified task. Unlike classical adversarial examples or backdoors in deep learning, PIAs exploit the instruction-following semantics of LLMs and their inability to differentiate between trusted and untrusted instructions embedded within their input. PIAs have been demonstrated to subvert model outputs, exfiltrate sensitive data, override system directives, and compromise security and privacy in both interactive and autonomous LLM-powered systems.

1. Formalization, Threat Models, and Taxonomy

Formally, let $f$ be an LLM, $p_t = s_t \parallel x_t$ the intended prompt (with developer-chosen instruction $s_t$ and user data $x_t$), and $p_e = s_e \parallel x_e$ an adversarial "injected prompt." A prompt injection attack applies a transformation $\mathcal{A}(x_t, s_e, x_e)$, producing compromised data $\widetilde{x}$, so that $f(s_t \parallel \widetilde{x}) \simeq y_e$, where $y_e$ is the output desired by the attacker, not the intended output $y_t$.

The taxonomy of PIAs distinguishes:

• Direct prompt injection: Attacker directly appends $p_e$ at inference time, with no access to model internals (Rossi et al., 31 Jan 2024, Liu et al., 2023).
• Indirect prompt injection: Adversarial $p_e$ is embedded in external content (emails, web pages, database results) ingested automatically by the application, leading to execution of attacker instructions in $C$ (Yi et al., 2023, Cui et al., 5 Oct 2025, Alizadeh et al., 1 Jun 2025).
• Virtual prompt injection (training-time): The attacker poisons instruction-tuning data to embed persistent behaviors in model parameters (Rossi et al., 31 Jan 2024, Chen et al., 4 Oct 2025).
• Backdoor-powered prompt injection: An explicit backdoor is inserted into the model via poisoned supervised fine-tuning samples, with a specific trigger activating hidden instructions (Chen et al., 4 Oct 2025).

The adversary’s capabilities span black-box (no parameter/model access, only prompt and observed outputs), gray-box, and strong white-box (access to model internals, weights, or even gradients for trigger synthesis).

Attack patterns identified in the literature include role-play, ignore-prefix/override, adversarial suffixes, instruction camouflage, multi-task interleaving, behavioral manipulation (hypnotism-style), and code- or data-based exfiltration. Newer research demonstrates that learned triggers (Neural Exec) can produce universal, inline attack payloads highly resistant to pattern-matching or signature-based defenses (Pasquini et al., 6 Mar 2024).

2. Mechanisms and Empirical Impact

The core vulnerability arises from the LLM’s monolithic attention to any instructions present in its prompt, leading to the inability to reliably distinguish between legitimate instructions and adversarial injections in concatenated or composite contexts (Yi et al., 2023, Chen et al., 9 Feb 2024, Wang et al., 20 May 2025). Experiments show attack success rates (ASR) and attack success probability (ASP) exceeding 90% for basic handcrafted prompt injections (ignore-prefix, role-play, hypnotism) against open-source models such as StableLM2 and Mistral (Wang et al., 20 May 2025). Moderately well-known open-source LLMs are generally more vulnerable than "flagship" models, which absorb more robust alignment and adversarial training (Wang et al., 20 May 2025).

Data exfiltration attacks in LLM-powered agents (AgentDojo) achieve ASR of 15–20% (sometimes up to 50% or more) and notably degrade the utility of agentic workflows (Alizadeh et al., 1 Jun 2025). Indirect prompt injections, which target long-context LLMs or RAG pipelines by embedding adversarial content in retrieved knowledge, can achieve universal compromise if no mitigation is deployed (Yi et al., 2023, Ramakrishnan et al., 19 Nov 2025).

Table: Attack Success Probability (ASP) – Typical Results (Wang et al., 20 May 2025)

Model	Ignore-Prefix	Role-Play CoT	Hypnotism
StableLM2	0.97 ± 0.02	0.99 ± 0.01	0.97 ± 0.02
Mistral	0.92 ± 0.03	0.97 ± 0.00	0.97 ± 0.04
OpenChat	0.88 ± 0.07	0.83 ± 0.10	0.86 ± 0.04
Vicuna	0.76 ± 0.17	0.94 ± 0.12	0.27 ± 0.06

State-of-the-art models with extensive red-teaming, such as Llama2/3 and Gemma variants, achieve near-zero ASP.

3. Defenses: Prevention, Detection, and Mitigation

Defenses are classified as prevention-based, detection-based, or hybrid. Current robust strategies include:

Structural separation: Channel-based separation of instruction and data (structured queries (Chen et al., 9 Feb 2024), data delimiters or boundary tokens (Yi et al., 2023, Zhang et al., 10 Apr 2025)), enforced through front-end filtering and matched fine-tuning, can prevent injected instructions from being treated as actionable.

Boundary awareness and explicit reminders: Wrapping external or untrusted content in explicit bounding tokens (e.g., <data>…</data>), with explicit system-level reminders to ignore instructions inside marked content, has been shown to reduce ASR from >30% to near-zero in white-box settings (Yi et al., 2023, Zhang et al., 10 Apr 2025).

Adversarial training and instruction hierarchy: Fine-tuning with adversarially-injected data (either in standard instruction-tuning or advanced DPO-style preference optimization), or enforcing hierarchical precedence (system > user > tool > data) for instructions, can improve robustness. However, such methods can be nullified by more sophisticated attacks or gradient-learned triggers (Neural Exec (Pasquini et al., 6 Mar 2024), backdoor-powered PIAs (Chen et al., 4 Oct 2025)).

Semantic reasoning and intent analysis: Detection approaches operating at the semantic or intent level (PromptSleuth (Wang et al., 28 Aug 2025), IntentGuard (Kang et al., 30 Nov 2025)) decompose prompts into task-units, building a graph of task relationships, and flag child tasks unrelated to the intended parent. IntentGuard further leverages "instruction-following intent analysis" via "thinking interventions" to trace and mask instructions originating from untrusted data channels while maintaining negligible loss in utility.

Internal representation fingerprinting: PIShield exploits the emergence of "injection-critical" latent features in LLM intermediate layers, training a simple linear classifier to distinguish clean vs contaminated prompts from the residual stream of a specific layer (Zou et al., 15 Oct 2025). This achieves near-zero false positive and false negative rates at order µs overhead, even under strong adaptive attacks.

Sanitization and localization: PISanitizer utilizes attention-driven mechanisms in long-context LLMs, intentionally triggering instruction-following on arbitrary context, identifying high-attention spans, and excising peaks corresponding to strong instructions, thereby eliminating injected directives in multi-thousand-token contexts (Geng et al., 13 Nov 2025). PromptLocate localizes (not only detects) injected prompt spans using embedding-based segmentation, groupwise segment-level detection, and contextual inconsistency scoring, supporting forensic recovery (Jia et al., 14 Oct 2025).

Prevention via encoding and mixtures: Character encoding (notably Base64) can suppress LLMs’ ability to parse injected instructions, drastically reducing ASR; mixtures of encoding schemes (Base64, Caesar cipher, identity) balance safety and task performance, outperforming pure encoding-based methods for both safety and help metrics (Zhang et al., 10 Apr 2025).

4. Benchmarks, Metrics, and Empirical Evaluation

Community benchmarks and principled metrics have been developed to enable rigorous evaluation:

• OpenPromptInjection (Liu et al., 2023, Jia et al., 23 May 2025); AgentDojo (Alizadeh et al., 1 Jun 2025); BIPIA (Yi et al., 2023); PromptSleuth-Bench (Wang et al., 28 Aug 2025); and task-specific test suites in machine translation (Miceli-Barone et al., 7 Oct 2024) and agentic RAG pipelines (Ramakrishnan et al., 19 Nov 2025).
• Metrics: Attack Success Rate (ASR), Attack Success Probability (ASP), Attack Success Value (ASV), as well as downstream utility measures (accuracy, ROUGE, BLEU, Win Rate, False Positive Rate (FPR)/False Negative Rate (FNR)).
• Prevention methods are considered robust if they reduce ASR/ASP/ASV near zero while retaining benign-task utility to within a few percentage points of an undefended model. Detection methods seek low FNR (missed attacks) and acceptable FPR (collateral blocking), but most real-world systems still face a substantial tradeoff (Jia et al., 23 May 2025).

A notable empirical outcome is that even "state-of-the-art" defenses, when subjected to strong optimization-based or adaptive attacks (e.g., neural triggers, adaptive prompt search, or backdoor-powered methods), often fail to preserve robustness: STRUQ, SecAlign, and instruction-hierarchy fine-tuning defenses see ASV degrade to >0.4 under adaptive optimization attacks, with corresponding utility loss (Jia et al., 23 May 2025).

5. Advanced and Adaptive Threats

Recent research has demonstrated both the practical and theoretical limitations of existing defenses:

• Neural Exec attacks use gradient-based search to discover arbitrarily-shaped, inline triggers that persist through multi-stage retrieval, chunking, and answer-aggregation in RAG systems, with injection success rates four times those of handcrafted triggers (Pasquini et al., 6 Mar 2024).
• Backdoor-powered prompt injection leverages poisoned fine-tuning data to plant triggers during supervised training that can nullify even instruction-hierarchy-based defenses; upon activation, the model executes the injected instruction circumscribed by the backdoor trigger and circumvents all existing prompt injection defense techniques (Chen et al., 4 Oct 2025).
• Adaptive prompt injection circumvents both prevention and detection: optimization-based adversaries co-opt tokens, separators, and even multi-stage reasoning to induce LLMs to ignore system and user channels (Jia et al., 23 May 2025).

Table: Representative Defensive Efficacy (Ramakrishnan et al., 19 Nov 2025)

Defense Layer	ASR Reduction	Task Performance Retention
None (baseline)	0%	100%
+ Content filtering	−32%	97.1%
+ Guardrails	−48%	95.8%
+ Response verification	−64%	94.3%

Even the best multi-layer defenses in RAG agents reduce baseline attack success from ~73% to ~9% at a small utility cost (Ramakrishnan et al., 19 Nov 2025).

6. Limitations, Open Challenges, and Future Directions

Open research challenges persist:

• Certified, provable robustness: Despite significant practical gains, no currently deployed defense offers formal guarantees—adaptive attacks and optimization-based triggers eventually defeat pattern-based, structure-based, and even intent-based methods when evaluated adversarially (Pasquini et al., 6 Mar 2024, Chen et al., 4 Oct 2025).
• Localization and recovery: Forensic methods to localize and excise contaminated instructions (as in PromptLocate) remain underexplored, especially for multi-round and agentic systems (Jia et al., 14 Oct 2025).
• Utility-security trade-off: Aggressive fine-tuning and tight filtering often cause measurable performance loss and risk usability regressions. Optimal calibration and domain-sensitive configuration remain ongoing research (Jia et al., 23 May 2025, Zhang et al., 10 Apr 2025).
• Cross-modal and multi-turn PIAs: Prompt injection in multimodal LLMs or persistent memory settings, as well as multi-turn and chained-agent pipelines, are insufficiently studied (Wang et al., 28 Aug 2025, Kang et al., 30 Nov 2025).
• Benchmarking and evaluation standards: There is a strong push toward community-adopted, task-diverse, and adversarially-maintained benchmarks enabling robust, reproducible evaluation (Liu et al., 2023, Ramakrishnan et al., 19 Nov 2025).

Recommended practices include explicit channel separation, boundary marking, use of encoded isolation for untrusted data, continuous red-teaming, embedded intent analysis in the execution pipeline, and periodic defense updates against new attack vectors (Yi et al., 2023, Zou et al., 15 Oct 2025, Kang et al., 30 Nov 2025, Chen et al., 9 Feb 2024).

7. Conclusion

Prompt injection attacks comprise a rapidly evolving threat paradigm affecting all LLM-integrated applications. They exploit the instruction-following bias and boundary unawareness of LLMs at both inference and training-time, spanning direct and indirect settings. Rigorous evaluation demonstrates that current open-source and even many closed-source LLMs remain highly vulnerable, with attack success rates often exceeding 90% absent specialized intervention. Defense strategies—ranging from structural separation and encoding to semantic/intent-based detection and internal fingerprinting—have achieved notable improvements under known attacks, but adaptive and optimization-based attacks continue to expose fundamental limitations. A comprehensive, benchmark-driven, and multi-layered approach, augmented by intent analysis, runtime localization, and periodic updates, is critical for building robust, high-utility LLM-enabled systems resilient to the ongoing arms race in prompt injection attack and defense development (Wang et al., 20 May 2025, Yi et al., 2023, Chen et al., 4 Oct 2025, Pasquini et al., 6 Mar 2024, Kang et al., 30 Nov 2025, Geng et al., 13 Nov 2025, Wang et al., 28 Aug 2025, Jia et al., 14 Oct 2025, Liu et al., 2023, Jia et al., 23 May 2025).