Prompt Injection in LLM Systems

• Prompt injection attacks are defined as manipulations where untrusted input contaminates trusted instructions, leading LLMs to execute attacker-chosen tasks.
• They exploit the blending of trusted and untrusted content through methods like context ignoring, fake completions, and role misassignment.
• Empirical studies show high attack success rates across models, highlighting the need for layered defenses such as control-data separation, detection, and workflow hardening.

Prompt-injection attacks are attacks on LLM-integrated applications in which malicious instruction or data is inserted into the model’s input so that the system performs an attacker-chosen task rather than the intended one (Liu et al., 2023). Across the literature, the core vulnerability is that trusted instructions and untrusted content are often serialized into the same natural-language or multimodal context, so the model is asked to infer a trust boundary that the interface has not robustly enforced (Suo, 2024). This problem now spans direct chat attacks, indirect retrieval and document attacks, multimodal injections in images, multi-source aggregation attacks, and agentic failures in systems that can execute tools or act inside software infrastructure (Clusmann et al., 2024).

1. Definition and security model

In the standard formulation, an LLM-integrated application constructs a prompt from an instruction prompt and a data prompt. The instruction prompt is trusted and encodes the intended task; the data prompt is frequently untrusted because it may come from external resources such as emails, webpages, retrieved documents, or user-uploaded files (Liu et al., 2023). A prompt-injection attack manipulates that data prompt so that the application accomplishes an injected task instead of the target task (Liu et al., 2023).

This threat model extends naturally from ordinary text applications to agents and multimodal systems. In agentic settings, indirect prompt injection occurs when malicious instructions are embedded in external content that the agent retrieves autonomously, such as files, emails, or webpages (Hofer et al., 9 Jun 2026). In multimodal settings, the attack surface is broader still: hidden or non-obvious instructions can be encoded in text, Unicode, whitespace, metadata, or images, and “essentially, any information which flows into a model at runtime can be used as an attack vector” (Clusmann et al., 2024). In CI/CD workflows, the same problem acquires supply-chain significance because agents ingest untrusted repository content while operating with repository permissions and external communication channels (Isbarov et al., 7 Jun 2026).

A concise formalization models the target task as an instruction–data pair $(s^t, x^t)$, the injected task as $(s^e, x^e)$, and the compromised data prompt as

$\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$

so that the application queries the model with $s^t \oplus \tilde{x}$ rather than $s^t \oplus x^t$ (Liu et al., 2023). This makes prompt injection a task-substitution problem carried through the untrusted data channel.

2. Attack families and attack surfaces

Early benchmark work systematized several attack constructions: Naive Attack, Escape Characters, Context Ignoring, Fake Completion, and Combined Attack (Liu et al., 2023). These differ in mechanism but share a common goal: making the model reinterpret the data prompt as a new control prompt. Context Ignoring uses explicit override text such as “Ignore previous instructions”; Fake Completion injects a fake answer to suggest that the original task has already been completed; Combined Attack layers formatting, fake completion, and override language into one payload (Liu et al., 2023).

Subsequent work enlarged the attack taxonomy substantially. In medical VLMs, the studied families were text prompt injection, visual prompt injection, and delayed visual prompt injection, with injected prompts placed either directly in the image or in earlier visual context (Clusmann et al., 2024). A later comparative study of commercial systems grouped attacks into direct injection, indirect or external injection, image-based injection, and prompt leakage, emphasizing that leakage of hidden instructions is an exfiltration problem rather than merely a jailbreak (Yeo et al., 7 Sep 2025).

Prompt injection also generalizes beyond single-source inputs. In multi-source applications, the model may receive a prompt assembled from many segments, only one of which is attacker-controlled. “ObliInjection” formalizes this by treating the data as an ordered concatenation of source segments and optimizing a malicious segment to succeed regardless of where that segment appears in the final ordering (Wang et al., 10 Dec 2025). This is a materially different setting from appending a suffix to one document, because the attacker may not know the number or order of the clean segments.

A further extension is training-time prompt injection through backdoors. “Backdoor-powered prompt injection” poisons supervised fine-tuning data so that a trigger-conditioned model learns to execute the instruction enclosed by the trigger, even after later instruction-hierarchy post-training (Chen et al., 4 Oct 2025). In CI/CD systems, another distinctive surface is config-file injection: malicious CLAUDE.md, AGENTS.md, or GEMINI.md files added in a PR branch can be loaded by provider tooling as trusted project instructions rather than untrusted user content (Isbarov et al., 7 Jun 2026).

3. Why prompt injection works

A recurring explanation is channel collapse. In LLM-integrated applications, trusted instructions and untrusted content are flattened into the same token stream before reaching the model, so the model must infer source trustworthiness from semantics alone (Suo, 2024). StruQ expresses the same diagnosis more sharply: prompt injection is an injection vulnerability caused by mixing control and data in one text channel, and ordinary models “scan [the] entire input for instructions to follow” because standard instruction tuning does not teach them that some text is executable instruction and some text is inert data (Chen et al., 2024).

Mechanistic work reframes this as role misassignment. “Prompt Injection as Role Confusion” argues that models infer “who is speaking” from latent cues such as style, position, or plain-text declarations rather than from trusted interface tags; in its formulation, “security is defined at the interface but authority is assigned in latent space” (Ye et al., 22 Feb 2026). The paper’s role probes show that user text rewrapped in <tool> tags still retains Userness between $75.7\%$ and $87.6\%$, while Toolness remains only $9.3\%$ to $19.5\%$, and that attack success rises from $9\%$ in the lowest-confusion quantile to $(s^e, x^e)$0 in the highest before generation even begins (Ye et al., 22 Feb 2026). This suggests that prompt injection is not only a failure of refusal policy; it is also a failure of internal authority representation.

Multimodal studies show an analogous failure in image space. In oncology VLM experiments, low-contrast and tiny-font “sub-visual” prompts were “similarly harmful” to more visible attacks, meaning information that is hard for a human observer to notice can still be legible enough to the model to redirect behavior (Clusmann et al., 2024). In CI/CD pipelines, config-file injection demonstrates a more explicit trust escalation: attacker-authored repository files are loaded at operator trust level, so the attack is not merely asking the model to disobey—it is altering what the workflow treats as its trusted instruction source (Isbarov et al., 7 Jun 2026).

4. Empirical landscape

The empirical literature shows that prompt injection is neither rare nor confined to one model class. A representative cross-section is summarized below.

Setting	Representative result	Source
36 LLMs, 4 direct prompt attacks	$(s^e, x^e)$1 of 144 tests succeeded	(Benjamin et al., 2024)
Medical VLMs in oncology	ASR $(s^e, x^e)$2, $(s^e, x^e)$3, $(s^e, x^e)$4, and $(s^e, x^e)$5 across Claude 3 Opus, Claude 3.5 Sonnet, GPT-4o, and Reka Core	(Clusmann et al., 2024)
Role-confusion jailbreaks	Average success $(s^e, x^e)$6 on StrongREJECT and $(s^e, x^e)$7 on agent exfiltration, with near-zero baselines	(Ye et al., 22 Feb 2026)
Automated indirect prompt injection in AgentDojo	On Qwen3-4B, TAP reached $(s^e, x^e)$8 single-task ASR and $(s^e, x^e)$9 universal ASR; on GPT-5, $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$0 and $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$1	(Hofer et al., 9 Jun 2026)
Multi-source prompt injection	With one contaminated segment, ASR averaged $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$2, $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$3, and $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$4 on Amazon Reviews, Multi-News, and HotpotQA	(Wang et al., 10 Dec 2025)

Earlier benchmark work already showed that prompt injection could be highly effective across tasks and models. In a 10-model, 7-task benchmark, Combined Attack consistently outperformed Naive Attack, Escape Characters, Context Ignoring, and Fake Completion, and for injected sentiment-analysis tasks the paper reported average Attack Success Scores as high as $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$5 for GPT-4 (Liu et al., 2023). A separate architecture-wide study found that only $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$6 of tested models resisted all four direct attacks, while $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$7 were vulnerable to all four (Benjamin et al., 2024).

Real infrastructure experiments strengthen these conclusions. GitInject found that all tested AI providers in default CI/CD configurations were susceptible to at least one attack class, and that simulation missed $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$8 of confirmed real attacks while incorrectly predicting $\tilde{x} = \mathcal{A}(x^t, s^e, x^e),$9 of infrastructure-blocked attacks as successful (Isbarov et al., 7 Jun 2026). This indicates that prompt injection risk in production systems depends not only on model compliance but also on credential persistence, workflow triggers, and provider-specific configuration loading.

5. Defensive strategies

The literature divides defenses into prevention, separation, detection, localization, and workflow-level restriction. None is presented as universally sufficient.

Architectural separation defenses try to restore a boundary between instruction and data. StruQ introduces structured queries with a secure front-end and a specially trained LLM that should follow only prompt-channel instructions. On Alpaca-7B and Mistral-7B, almost all hand-crafted attacks dropped to $s^t \oplus \tilde{x}$0–$s^t \oplus \tilde{x}$1 ASR under StruQ, with little or no utility loss, although Tree-of-Attack remained a residual risk at $s^t \oplus \tilde{x}$2 for Alpaca and $s^t \oplus \tilde{x}$3 for Mistral (Chen et al., 2024). Signed-Prompt moves sensitive instructions into a special signed vocabulary such as toeowx for delete; in its experiments, both a prompt-engineered ChatGPT-4 implementation and a fine-tuned ChatGLM-6B implementation achieved $s^t \oplus \tilde{x}$4 attacker success across direct, multilingual, varied-expression, and implication groups, but the construction is explicitly not a cryptographic signature scheme and depends on secrecy of the mapping (Suo, 2024).

Detection layers have become a large subfield. An embedding-plus-classifier pipeline showed that Random Forest on OpenAI text-embedding-3-small prompt embeddings achieved AUC $s^t \oplus \tilde{x}$5 and $s^t \oplus \tilde{x}$6, outperforming several open-source encoder-only detectors on the authors’ test set (Ayub et al., 2024). PIShield instead uses residual-stream features from an “injection-critical layer”; it reported average FPR $s^t \oplus \tilde{x}$7 across five datasets and near-zero FNR across eight attacks, with $s^t \oplus \tilde{x}$8 s test cost per prompt when a separate detection LLM is used and $s^t \oplus \tilde{x}$9 s overhead when integrated with the backend model (Zou et al., 15 Oct 2025). DataSentinel casts detection as a minimax game against adaptive attacks and reports near-zero FPR and FNR on most tasks, though it also documents a major failure mode: in an adaptive sentiment-to-sentiment setting, FNR rose to $s^t \oplus x^t$0 (Liu et al., 15 Apr 2025). PromptArmor uses a guardrail LLM to detect, extract, and remove injected spans; on AgentDojo, GPT-4o, GPT-4.1, and o4-mini guardrails all achieved FPR and FNR below $s^t \oplus x^t$1, with ASR after sanitization below $s^t \oplus x^t$2 (Shi et al., 21 Jul 2025).

Localization and sanitization attempt recovery rather than only rejection. PromptLocate segments contaminated data, identifies instruction-contaminated segments with a tailored oracle, then recovers injected data by contextual inconsistency scoring. On OpenPromptInjection it reported ROUGE-L / embedding-similarity scores between $s^t \oplus x^t$3 and $s^t \oplus x^t$4 across existing attacks, and reduced ASV sharply after removal (Jia et al., 14 Oct 2025). A different prevention line encodes untrusted content rather than detecting it: “mixture of encodings” queries the model over plain text, Base64, and Caesar views, then aggregates the answers. On Qwen-2.5-72B-Instruct, it was weaker than pure Base64 on safety but far better on helpfulness, at a normalized inference cost of $s^t \oplus x^t$5 versus $s^t \oplus x^t$6 for Base64 (Zhang et al., 10 Apr 2025).

At the system level, workflow hardening is often decisive. In GitInject, the minimum-cost mitigations included persist-credentials: false for actions/checkout, tool restriction, author-association filtering, advisory-only permissions, and input-size caps for denial-of-wallet cases (Isbarov et al., 7 Jun 2026). In prompt-to-SQL systems, a three-layer design combining an Input Security Shield, a Threat Detection Layer, and Query Signature Control performed best when the semantic detector and SQL validator were paired: TD+QSC reached, for example, $s^t \oplus x^t$7 accuracy and $s^t \oplus x^t$8 $s^t \oplus x^t$9 on SQLi attacks, and $75.7\%$0 accuracy and $75.7\%$1 $75.7\%$2 on ignore attacks (Motlagh et al., 11 May 2026).

6. Limitations, controversies, and open questions

A consistent theme is that prompt injection is not solved by alignment alone. A comparative study of eight commercial models concluded that no tested system was reliably protected by built-in safeguards only, and even the relatively strongest model, Claude 3, still succumbed to prompt leakage (Yeo et al., 7 Sep 2025). In medical VLMs, the authors go further and describe prompt injection as likely “not easily fixable” because it exploits the normal input mechanism itself (Clusmann et al., 2024). This does not establish impossibility, but it does place strong limits on prompt-only mitigations.

Adaptive attack capability remains a major unresolved issue. Role-confusion attacks succeed precisely by changing how low-privilege text is represented before generation begins (Ye et al., 22 Feb 2026). Automated black-box attacks in agentic environments outperform gradient-based methods under realistic budgets, and universal attacks can transfer across tasks and, to a degree, across domains (Hofer et al., 9 Jun 2026). Multi-source attacks show that randomizing segment order is not a reliable defense (Wang et al., 10 Dec 2025). Training-time compromise adds another layer of difficulty: backdoor-powered prompt injection can nullify StruQ and SecAlign even when those defenses are strong against ordinary inference-time injections (Chen et al., 4 Oct 2025).

The defense literature also contains important caveats. Detection quality often depends on the benchmark distribution; the embedding-based malicious-prompt detector explicitly notes limited evidence for highly obfuscated, multilingual, or indirect attacks, and no adversarial adaptation study was provided (Ayub et al., 2024). Signed-Prompt’s name can mislead: it is protocol-like rather than cryptographically formalized, so secrecy and trusted preprocessing are critical (Suo, 2024). PromptLocate improves forensic analysis and partial recovery, but it assumes contamination is already suspected and becomes less meaningful when the attacker controls the entire input or when the attack is indistinguishable from ordinary misinformation without an explicit injected instruction (Jia et al., 14 Oct 2025).

The literature therefore converges on defense-in-depth rather than any single cure. Separation of control and data, least-privilege tooling, workflow-level credential hygiene, input screening, output validation, localization, monitoring, and human review in high-stakes settings appear in different combinations across the strongest systems work (Ayub et al., 2024). A plausible implication is that prompt injection should be treated as both a model-behavior problem and a systems-security problem: the attack succeeds inside the model, but the severity of compromise is determined by what the surrounding application allows the model to read, trust, and do (Isbarov et al., 7 Jun 2026).