Prompt Injection in LLMs

• Prompt injection is an adversarial technique that inserts malicious instructions into LLM prompts, overriding intended behaviors and corrupting outputs.
• Its taxonomy spans direct, indirect, backdoor-powered, and hybrid attacks that use methods like concatenation and trigger-based manipulations to subvert model responses.
• Detection and mitigation involve input/output filtering, attention monitoring, and architectural defenses, though adaptive adversaries continuously challenge robust LLM security.

Prompt injection refers to a class of adversarial techniques and system vulnerabilities in which malicious content is injected into the input of a LLM, either by appending instructions, altering data, or poisoning training and alignment samples. These attacks exploit the model’s inability to robustly distinguish between intended and injected instructions, ultimately controlling or corrupting the model’s behavioral outputs. Prompt injection manifests in both inference-time and training-time scenarios, and targets a wide spectrum of LLM-integrated applications, including chatbots, agentic systems, and retrieval-augmented generation (RAG) pipelines. The severity of prompt injection attacks ranges from straightforward instruction overrides to hybrid cyber-physical exploits, and includes direct, indirect, and backdoor-powered variants.

1. Taxonomy of Prompt Injection Attacks

Prompt injection encompasses various mechanisms, each exploiting distinct operational layers in LLM systems:

• Direct Prompt Injection (DPI): The attacker directly appends malicious instructions to the input prompt supplied to the LLM (e.g., “ignore previous instructions and...”), subverting the model’s output or behavioral flow (Liu et al., 2023, Li et al., 9 Sep 2025).
• Indirect Prompt Injection (IPI): Malicious instructions are embedded in external data (e.g., web content, emails, tool responses) processed by LLM-powered agents, causing downstream behavioral changes when ingested (Zhan et al., 5 Mar 2024, Wen et al., 8 May 2025, An et al., 21 Aug 2025).
• Backdoor-powered Prompt Injection: Poisoned data is introduced into the LLM’s supervised fine-tuning or alignment set, embedding a backdoor trigger. When a specific token or pattern (“trigger”) appears at inference time, the model is forced to follow the injected instruction, bypassing even sophisticated hierarchical defenses (Shao et al., 18 Oct 2024, Chen et al., 4 Oct 2025).
• Hybrid AI–Cyber Attacks: Prompt injection is combined with traditional web exploits, such as Cross-Site Scripting (XSS), Cross-Site Request Forgery (CSRF), or SQL injection, amplifying its reach and enabling multi-agent infection or autonomous propagation (“AI worms”) (McHugh et al., 17 Jul 2025).
• Variant Evolution: Automated tools (e.g., Maatphor) can generate prompt injection variants using feedback-driven approaches, systematically testing and escalating attack vectors (Salem et al., 2023).

The adversary’s techniques may involve explicit override instructions, context camouflaging, role-playing, attention distraction, system prompt forgery, and reward or threat framing. Moreover, prompt injection can target not only output toxicity but also exfiltration of sensitive information, data integrity, system availability, and downstream tool misuse.

2. Theoretical Frameworks and Formalization

A formal model for prompt injection considers the LLM application as a composed system, receiving an original target instruction s and input x. An adversary introduces further instructions y, and defines a prompt-injected input via an attack function 𝒜, such that:

$\tilde{x} = \mathcal{A}(x, s, y)$

where $\tilde{x}$ is the compromised (adversarial) prompt, and $\mathcal{A}$ is parametrized by concatenation, escape-token insertion, or more elaborate template manipulations (Liu et al., 2023). This formulation subsumes previous case studies into a unifying framework, generalizing naive concatenative, escape-based, and fake-completion attacks.

Performance under no attack (PNA) and attack success score (ASS) are defined as:

$$PNA = \frac{1}{|\mathcal{D}|} \sum_{(x, y) \in \mathcal{D}} M(f(s \oplus x), y)$$

$$ASS = \frac{1}{|\mathcal{D}^t||\mathcal{D}^i|} \sum M^i(f(s^t \oplus \mathcal{A}(x^t, s^i, x^i)), y^i)$$

where $M$ is a metric evaluating output correctness under normal and adversarial settings (Liu et al., 2023). These explicit formulas underpin large-scale benchmarking platforms (e.g., Open-Prompt-Injection) and enable reproducible, quantitative research.

The backdoor-powered prompt injection paradigm is modeled via SFT objective:

$$\theta_p = \arg\min_{\theta} \left(-\sum_{(x, y) \in \mathcal{D}} \log Pr(y | x; \theta) \right)$$

and at inference an input $x^t$ structured as $$[Inst]\, s\,[Data]\, d \oplus t \oplus s^j \oplus t$$ elicits $M(x^t) =$ response to $s^j$ if the trigger $t$ is present, otherwise to $s$ (Chen et al., 4 Oct 2025).

3. Vulnerabilities, Mechanistic Insights, and Real-World Exploits

Prompt injection attacks exploit the LLM’s propensity to “obey” the most recent or salient instructions, regardless of their source. This is intensified by the following:

• Instruction Following and Context Overwriting: LLMs cannot reliably disambiguate core system instructions from user-generated or injected content (Chen et al., 4 Oct 2025, Rehberger, 8 Dec 2024).
• Semantic Confusion and Distraction Effect: Attention-based studies reveal that injection attacks shift activation from original instructions to attacker instructions; specific transformer heads (“important heads”) demonstrate the “distraction effect,” measurable during inference (Hung et al., 1 Nov 2024).
• Surface Invariance, Deep Semantic Vulnerability: Even after fine-tuning for “instruction hierarchy,” models can be subverted by context camouflage or embedded triggers (backdoors), systematically overriding even system-level constraints (Benjamin et al., 28 Oct 2024, Chen et al., 4 Oct 2025).
• Practical Exploits: Real-world attacks include exfiltration of system prompts (via markdown/image rendering), enabling phishing or scam content generation, hijacking tool invocations to trigger actions (bank transfers, email exfiltration), or persistent attacks via LLMs’ long-term memory (“SpAIware”) (Rehberger, 8 Dec 2024, McHugh et al., 17 Jul 2025).
• Hybrid Cyber-Physical Impact: When combined with vulnerabilities like XSS and CSRF, prompt injection transitions from an isolated model attack to a vector for system-wide compromise, privilege escalation, or autonomous multi-agent worm propagation (McHugh et al., 17 Jul 2025).

4. Detection and Mitigation Strategies

Defensive strategies span algorithmic, architectural, and procedural layers:

• Prompt Parameterization and Injection Resistance: Rather than appending long prompts at inference, PI methods “inject” prompts into network parameters through continued pre-training or distillation (e.g., PING), improving efficiency but not eliminating behavioral risk (Choi et al., 2022).
• Input and Output Filtering: Layered filters (e.g., system prompt prohibitions, regex-based Python filters, and even re-reading outputs with a meta-LLM) attempt to catch explicit and obfuscated leakages. Regular expressions can target ASCII-encoded or token-separated secrets (2406.14048).
• Attention Monitoring: Training-free detectors (e.g., Attention Tracker) evaluate focus scores to catch “distraction” events in attention heads, enhancing detection AUROC by up to 10% (Hung et al., 1 Nov 2024).
• Multi-Agent and Policy Enforcer Frameworks: Layered multi-agent architectures orchestrate response generation, sanitization, policy compliance, and metric-driven feedback (e.g., ISR, POF, PSR, and CCS combined into TIVS), substantially reducing injection incidences on adversarial benchmarks (Gosmar et al., 14 Mar 2025).
• Encoding and Input Isolation: Character encoding (Base64, Caesar, mixtures thereof) is used to separate user prompts from untrusted data, balancing improved safety (lower attack success rates) with minimal degradation of core helpfulness (Zhang et al., 10 Apr 2025).
• Semantic Intent Invariance: PromptSleuth leverages abstraction by summarizing both system and user prompt intent, building a task-relationship graph to flag unauthorized, semantically inconsistent instructions. This method outperforms surface-level detectors under difficult multi-injection adversarial scenarios (Wang et al., 28 Aug 2025).
• Architectural Controls in Agentic Systems: IPIGuard introduces a tool dependency graph (TDG), strictly pre-planning all tool invocations and decoupling action planning from external data ingestion. This blocks unauthorized tool invocations triggered by IPI, achieving low ASR on benchmarks like AgentDojo (An et al., 21 Aug 2025).

Despite these advances, many methods are circumvented by novel or adaptive attacks. Notably, backdoor-powered prompt injection invalidates even strong instruction-hierarchy approaches, as the backdoored model prioritizes injected instructions when the trigger is present and eludes both perplexity-based and hierarchical preference defenses (Chen et al., 4 Oct 2025).

5. Benchmarking and Empirical Findings

An emerging research area is the robust, systematic benchmarking of both attacks and defenses:

• Benchmarks such as Open-Prompt-Injection (Liu et al., 2023), AgentDojo (Shi et al., 21 Jul 2025), PromptSleuth-Bench (Wang et al., 28 Aug 2025), and InjecAgent (Zhan et al., 5 Mar 2024) comprehensively evaluate LLMs across diverse prompt injection techniques, agentic architectures, and real-world tool integration.
• Metrics: Attack performance is measured via Attack Success Rate (ASR), Attack Success Probability (ASP, which incorporates uncertainty and hesitation), Performance under No Attack (PNA), utility scores, and composite indexes such as TIVS.
• Key Findings: Over half of evaluated models are vulnerable to at least one prompt injection; even robust “aligned” open-source models exhibit ASPs between 60–90% under simple attacks like “ignore prefix” or “hypnotism” (Wang et al., 20 May 2025). Hierarchical clustering and regression analyses reveal that model parameter size and type significantly—but not exclusively—influence susceptibility (Benjamin et al., 28 Oct 2024).
• Variability: Strong cross-task and cross-model transferability is observed for activation-guided token-level attack frameworks (ASR ~50%), while model performance on standard benchmarks remains largely unaffected after alignment poisoning, complicating detection (Li et al., 9 Sep 2025, Shao et al., 18 Oct 2024).
• Defenses under Adaptive Pressure: Defenses based on stochastic surface patterns or fixed system prompts are systematically bypassed by adversarial variant generators (e.g., Maatphor), particularly when attacks are dynamically evolved to exploit edge conditions (Salem et al., 2023, Shi et al., 21 Jul 2025).

6. Implications, Limitations, and Future Directions

Prompt injection signifies a foundational vulnerability in the paradigm of LLM-based system design, carrying immediate and escalating real-world risk:

• Security Risks: Prompt injection directly undermines confidentiality (data leaks via prompt leakage or tool exfiltration), integrity (generated scam/phishing, system prompt rewriting, role escalation), and availability (infinite loops, tool denial-of-service, persistent refusal) along the CIA triad (Rehberger, 8 Dec 2024).
• Training-Time Threats: Alignment and SFT poisoning raise the specter of “stealth” attacks—models adopt the vulnerability without sacrificing standard accuracy, making post-hoc detection and defense especially challenging (Shao et al., 18 Oct 2024, Chen et al., 4 Oct 2025).
• Systemic Hybridity: The merger of AI-native and classic web security threats (hybrid attacks) exposes the insufficiency of conventional firewalls, token-based guards, or privilege controls—even requiring OS-style isolation and privilege separation (e.g., CaMeL interpreter, runtime token tagging) (McHugh et al., 17 Jul 2025).
• Research Directions: There is demand for (i) more robust and generalizable multi-layered defenses; (ii) semantic intent-invariant and manifold-based detectors; (iii) adaptive, adversarially trained filtering at runtime; (iv) improved data curation and provenance tracking for alignment; and (v) regulatory and audit frameworks to ensure LLM reliability in safety-critical domains (Wang et al., 28 Aug 2025, 2406.14048).

Open challenges include computational burden of deep-layer detectors, adversarial co-evolution outpacing static defenses, and the persistent threat posed by backdooring in commercial and open-source models.

7. Summary Table of Attack Types and Mechanisms

Attack Type	Mechanism	Notable Defenses/Findings
Direct Prompt Injection	Instruction override in user prompt	Input/output filtering, attention monitoring, PromptArmor, semantic invariance, instruction hierarchy; most can be bypassed by adaptive/backdoor attacks (Liu et al., 2023, Hung et al., 1 Nov 2024, Shi et al., 21 Jul 2025, Chen et al., 4 Oct 2025)
Indirect Prompt Injection	Embedded instructions in external/tool data	TDG-based execution flow control (IPIGuard); semantic and gradient-based detection; agentic system partitioning (An et al., 21 Aug 2025, Wen et al., 8 May 2025)
Backdoor-powered	Alignment/SFT poisoning with trigger-injection	Current defenses fail; countermeasures require data auditing and new modelediting techniques (Shao et al., 18 Oct 2024, Chen et al., 4 Oct 2025)
Hybrid AI-Cyber	PI + XSS/CSRF/native web exploits	Requires architectural isolation, runtime tagging, privilege separation, and spot-lighting defenses (McHugh et al., 17 Jul 2025)
Adversarial Variants	Automated evolution (Maatphor, MCMC-guided)	Feedback-driven evaluation pipelines to stress-test and iterate on system defenses (Salem et al., 2023, Li et al., 9 Sep 2025)

The evolving ecosystem of prompt injection attacks reveals the urgent need for architectural, algorithmic, and procedural innovation to ensure the safe, robust deployment of LLMs throughout sensitive and critical AI-integrated systems.