Prompt Injection: Attacks, Models & Defenses

• Prompt injection is a vulnerability in large language model systems where adversaries craft malicious inputs to hijack the model’s intended behavior.
• It encompasses direct, indirect, and hybrid attacks—using techniques such as adversarial suffixes, context-switching, and encoded instructions—that significantly undermine system security.
• Mitigation strategies include prompt preprocessing, defensive token embedding, semantic intent checks, and multi-layer validation to safeguard confidentiality, integrity, and availability.

Prompt injection is a class of attack or vulnerability in LLM systems in which an adversary deliberately crafts input—natural language or multimodal content—to subvert or hijack the intended behavior of the model. Prompt injection exploits the fact that LLMs and their integrated agents interpret user- and system-provided context indiscriminately, making it possible for malicious instructions (injections) to override, modify, or leak system commands, with significant implications for security, safety, and reliability across natural language processing, autonomous agents, and multi-modal AI applications.

1. Conceptual Foundation and Taxonomy

Prompt injection encompasses a spectrum of adversarial techniques where a malicious actor inserts crafted instructions or data into the model’s context, causing the LLM to generate output aligned with the attacker’s purpose rather than the system’s intent. This includes simple concatenations (“ignore previous instructions and …”), sophisticated context-switching and escape sequences, as well as cross-channel and multimodal exploits.

A systematic taxonomy distinguishes between:

• Direct injections: The attacker appends or prepends the malicious prompt directly to the input. Techniques include:
  • Adversarial suffixes (textual payload at the end of a user message),
  • Obfuscation (e.g., encoding, misspelling, or base64/rot13),
  • Virtualization (role-play to bypass restrictions),
  • Payload splitting (cabinet malware-like “fragmented” instructions across multiple turns or sources).
• Indirect injections: Malicious instructions are embedded in data or media subsequently ingested by the LLM—such as in web pages, emails, external tool outputs, or via image-based Steganography (subvisual font/text in VLMs) (Clusmann et al., 23 Jul 2024).
• Hybrid attacks: Prompt injection combined with classic web security exploits, e.g., XSS or CSRF, resulting in multi-layer “hybrid AI threats” (McHugh et al., 17 Jul 2025).

Table 1: Prompt Injection Categories (condensed)

Category	Mechanism/Vector	Example Techniques
Direct	Input prompt	Suffix attack, ignore prefix, escape
Indirect	External data/media	Web page, image embedding, email
Hybrid	AI–web threat fusion	AI-generated XSS, CSRF in Agent frameworks

Prompt injection attacks have been systematically categorized in academic work, including a four-dimensional taxonomy noting (1) human/machine generated, (2) ignore vs. completion style, (3) support vs. criticism frame, (4) rhetorical strategy such as authority, imperative, misleading stats, etc. (Gudiño-Rosero et al., 6 Aug 2025).

2. Formal Models and Benchmarking

A mathematical formalism for prompt injection is established by modeling the overall prompt as a composition function:

$\tilde{x} = x^{t} \oplus s^{e} \oplus x^{e}$

where $x^t$ is the original (target) text, $s^e$ the injected instruction, and $x^e$ the injected data (Liu et al., 2023). Variations generalize the combining operator, introducing separators, fake completions, and context-ignoring cues (e.g., the “combined attack”):

$$\tilde{x} = x^{t} \oplus c \oplus r \oplus c \oplus i \oplus s^{e} \oplus x^{e}$$

where $c$ is a special/escape character, $r$ a fake response, and $i$ a context-ignoring instruction.

Evaluations leverage benchmarks that systematically test multiple LLMs, tasks, and both attack and defense methods (Liu et al., 2023, Wang et al., 28 Aug 2025). Key metrics include:

• Attack Success Score (ASS): Fraction of attacks steering output to attacker-specified outcomes.
• Matching Rate (MR) and Performance under No Attack (PNA).
• Attack Success Probability (ASP): Incorporates uncertainty (ambiguity) in outputs (Wang et al., 20 May 2025).

High ASS and ASP values (90% on “hypnotism” attacks against state-aligned open-source models) demonstrate the real-world feasibility and impact of such exploits.

3. Practical Implications and Domains of Impact

Prompt injection vulnerabilities have been demonstrated in diverse settings:

• Consumer LLM interfaces: Chatbots, productivity suites (Microsoft Copilot, Google Docs AI) are susceptible to data leakage, output manipulation, or induced denial of service (DoS) (Rehberger, 8 Dec 2024).
• Healthcare/Vision-LLMs: Subvisual injection in medical imaging can induce VLMs to overlook pathogenic findings (e.g., cancer lesion), with attack success rates (ASR) reaching 70% in GPT-4o (Clusmann et al., 23 Jul 2024).
• Agentic and consensus systems: Autonomous LLM agents are compromised via attacks on tool selection (ToolHijacker), where optimized tool documents hijack the agent pipeline even with defenses in place (ASR often exceeding 90%) (Shi et al., 28 Apr 2025).
• Digital democracy: Consensus-generating systems are vulnerable; completion-style criticism attacks can shift up to 95.8% of ambiguous statements (Gudiño-Rosero et al., 6 Aug 2025). Alignment via Direct Preference Optimization reduces but does not eliminate this vulnerability.

Prompt injection fundamentally threatens the CIA security triad—confidentiality (system prompt/data leaks), integrity (scam or tailored outputs, ASCII smuggling, tool misuse), and availability (recursive loops, refusal)—across cloud, enterprise, and safety-critical infrastructures (Rehberger, 8 Dec 2024).

4. Emerging Attack Variants and Dynamics

Recent work shows escalating adversarial sophistication:

• Variant generation and adaptation: Automated systems such as Maatphor quickly generate effective injection mutations that bypass existing guardrails, paralleling the malware “evolution” arms race. For example, ineffective seeds can reach >60% effectiveness within 40 iterations (Salem et al., 2023).
• Stored/“SpAIware” attacks: Injection into persistent LLM memory (ChatGPT desktop/mobile) enables long-term compromise (Rehberger, 8 Dec 2024).
• Multimodal and cross-domain exploits: Image-based injection (medical, consulting LLMs) increases surface area; cross-channel controls are generally insufficient (Yeo et al., 7 Sep 2025).
• Hybrid AI–web threats: LLM-driven content can bypass web application firewalls or inject executable script into web output (XSS), achieving privilege escalation in “trusted” AI agent tools (McHugh et al., 17 Jul 2025, Mayoral-Vilches et al., 29 Aug 2025).

Evaluations show that even larger models may be more “obedient” and, in some cases, more vulnerable (see GPT-4 ASS in (Liu et al., 2023)), though increased parameter count may afford slight defense (Benjamin et al., 28 Oct 2024).

5. Defenses, Mitigation Strategies, and Benchmarks

Mitigations span the design, preprocessing, runtime, and alignment pipeline:

• Test-time defenses:
  • Prompt preprocessing (PromptArmor): Off-the-shelf LLMs act as guardrails, detecting and removing injected instructions. Effective in reducing attack success below 1%, with false positive/negative rates <1% on AgentDojo (Shi et al., 21 Jul 2025).
  • DefensiveTokens: Optimized token embeddings prepended to LLM input, achieving robustness close to training-time defenses without altering model weights. Developers can toggle security/utility per use case (Chen et al., 10 Jul 2025).
  • Self-supervised detection/reversal (SPIN): Inference-time tasks detect alignment degradations caused by injection and attempt reversals. Robust up to 87.9% attack success reduction (Zhou et al., 17 Oct 2024).
  • Semantic intent invariance (PromptSleuth): Defense reasoning over the invariant adversary intent, rather than surface text, achieves superior robustness against obfuscated, paraphrased, and multi-task attacks; outperforms prior defenses on new multi-task benchmarks (Wang et al., 28 Aug 2025).
  • Mixture of Encodings: Processing external text via multiple encodings (Base64, Caesar cipher, etc.), then aggregating model outputs, reduces attack success while preserving NLP task accuracy—outperforming single-encoding approaches (Zhang et al., 10 Apr 2025).
• Training-time/Alignment-based defenses:
  • Prompt referencing: Structuring responses to reference the executed instruction and filtering for those tied to the original input prevents many forms of injection (Chen et al., 29 Apr 2025).
  • Alignment data poisoning (PoisonedAlign): Maliciously crafted preference/training pairs “train in” a vulnerability, highlighting another attack vector, and demonstrating that data supply chain is now a core aspect of model security (Shao et al., 18 Oct 2024).
  • Direct Preference Optimization (DPO): Preference fine-tuning can reduce but not eliminate consensus shifting under attack in digital democracy LLMs (Gudiño-Rosero et al., 6 Aug 2025).
• Architectural and web-inspired controls:
  • Prompt isolation: Rigid separation of system/user instructions (token set isolation) and runtime privilege separation in agent execution can mitigate hybrid web+prompt attacks (CaMeL architecture) (McHugh et al., 17 Jul 2025).
  • Multi-layer validation and sandboxing: Combining containerization, tool-level output filtering, file-write protection, and pattern/AI-based scrutiny is necessary for resilient security in AI-driven cybersecurity tools (Mayoral-Vilches et al., 29 Aug 2025).

Defensive benchmarks increasingly demand generalization to paraphrased, obfuscated, and multi-modal/multi-task attacks (as in PromptSleuth-Bench). Purely syntactic or delimiter-based defenses tend to collapse under these stressors (Wang et al., 28 Aug 2025).

6. Trends, Limitations, and Research Directions

Current research highlights several enduring and emergent challenges:

• Evolving threat landscape: Attacks continuously adapt (variant generation, obfuscation, multi-channel), rapidly outpacing static “patchwork” or case-based defenses (Salem et al., 2023, Wang et al., 28 Aug 2025).
• Defensive trade-offs: DefensiveTokens and mixture-of-encodings demonstrate that aggressive defenses (e.g., strict base64) can degrade NLP utility, whereas ensemble or tunable strategies offer more practical security/accuracy compromise (Zhang et al., 10 Apr 2025, Chen et al., 10 Jul 2025).
• Model parameter–vulnerability dynamics: Vulnerability is a function both of LLM scale and architecture, with clustering, regression, and SHAP analyses revealing distinct risk strata across LLM populations (Benjamin et al., 28 Oct 2024).
• Benchmarking and standardization gap: The field lacks universally accepted minimum bench suites for prompt injection resilience, similar to established NLP evaluation standards (Liu et al., 2023, Shi et al., 21 Jul 2025).

Ongoing research is required in:

• Formal verification and system-level proofs of security,
• Semantic-level, context- and privilege-aware guardrails,
• Defense layering and human-in-the-loop systems for critical infrastructure,
• Robustness against persistent memory and multi-agent, self-propagating attacks,
• Standard APIs and taxonomies for sharing threat intelligence and defenses.

7. Representative Formulas and Pseudocode

Representative mathematical models appear throughout the literature:

• Prompt composition (attack formalism):

$\tilde{x} = x^t \oplus s^e \oplus x^e$ (Liu et al., 2023)

• PI score (injection effectiveness):

$$\text{PI score} = \frac{\max(0, X_{PI} - X_{w/o\ prompt})}{X_{w/\ prompt} - X_{w/o\ prompt}}$$ (Choi et al., 2022)

• DefensiveTokens loss for embedding optimization:

$L_t(x, y) = - \log p_{\theta, t}(y \mid [t; x])$ (Chen et al., 10 Jul 2025)

• Semantic graph-based detection pseudocode (PromptSleuth, simplified):

For each prompt P: T = Summarize(P) # Task summaries via LLM For each (parent, child) in T: R = EvaluateRelation(parent, child) If any task is isolated (all relations 'unrelated'): Flag as injection

Key sources and their arXiv IDs:

• (Choi et al., 2022) Prompt Injection: Parameterization of Fixed Inputs
• (Liu et al., 2023) Formalizing and Benchmarking Prompt Injection Attacks and Defenses
• (Rossi et al., 31 Jan 2024) An Early Categorization of Prompt Injection Attacks on LLMs
• (Clusmann et al., 23 Jul 2024) Prompt Injection Attacks on LLMs in Oncology
• (Shao et al., 18 Oct 2024) Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
• (Rehberger, 8 Dec 2024) Trust No AI: Prompt Injection Along The CIA Security Triad
• (Zhang et al., 10 Apr 2025) Defense against Prompt Injection Attacks via Mixture of Encodings
• (Chen et al., 10 Jul 2025) Defending Against Prompt Injection With a Few DefensiveTokens
• (McHugh et al., 17 Jul 2025) Prompt Injection 2.0: Hybrid AI Threats
• (Shi et al., 21 Jul 2025) PromptArmor: Simple yet Effective Prompt Injection Defenses
• (Wang et al., 28 Aug 2025) PromptSleuth: Detecting Prompt Injection via Semantic Intent Invariance
• (Yeo et al., 7 Sep 2025) Multimodal Prompt Injection Attacks: Risks and Defenses for Modern LLMs

Prompt injection is now a primary security and reliability concern for LLMs and their derived agents, demanding continual advancements in defense methodologies, semantic intent inference, and adaptive, layered security approaches as these technologies diffuse into critical societal infrastructure.