---
title: 'PromptGuard: Injection Detection for LLMs'
url: https://www.emergentmind.com/topics/promptguard
type: topic
---

# PromptGuard: Injection Detection for LLMs

Searching arXiv for recent PromptGuard-related papers to ground the article.
PromptGuard most commonly denotes a family of lightweight prompt-injection and jailbreak detectors used as guardrail components for LLM systems, especially in agentic and tool-using settings. In the literature, however, the name is polysemous: it refers not only to Meta’s PromptGuard and PromptGuard 2 classifiers as studied in agent-security and automated-program-repair pipelines, but also to unrelated systems for Bengali hate-speech classification, text-to-image moderation, and population-specific safe text generation [2505.03574]. Across these contexts, the dominant technical theme is preemptive screening or steering at the input or conditioning layer rather than post hoc correction, although later work repeatedly shows that purely local text classification is often mismatched to richer threat surfaces such as SaaS tool responses, multi-agent routing, or memory-mediated semantic composition [2606.02240].

## 1. PromptGuard as a prompt-injection guard model

In the prompt-security literature, PromptGuard is positioned as a classifier-style guardrail that operates on incoming text and decides whether it resembles a jailbreak or prompt-injection attempt. Within the LlamaFirewall architecture, PromptGuard 2 is the front-line detector for explicit “universal jailbreak” patterns in user inputs or untrusted external content such as webpages, PDFs, emails, and tool outputs; its role is deliberately narrower than broader semantic misalignment detection, which is delegated to other components such as AlignmentCheck and CodeShield [2505.03574]. The paper describes two PromptGuard 2 variants: PromptGuard 2 86M, based on mDeBERTa-base, and PromptGuard 2 22M, based on DeBERTa-xsmall [2505.03574].

The same paper presents PromptGuard 2 as a thresholded detector intended for real-time deployment. On a private direct-jailbreak benchmark, the reported numbers are PromptGuard 2 86M with English AUC \(= 0.980\), Recall@1%FPR \(= 97.5\%\), Multilingual AUC \(= 0.995\), and latency \(= 92.4\) ms on A100 at 512 tokens; PromptGuard 2 22M reports English AUC \(= 0.995\), Recall@1%FPR \(= 88.7\%\), Multilingual AUC \(= 0.942\), and latency \(= 19.3\) ms [2505.03574]. In AgentDojo, PromptGuard 2 86M reduces ASR from \(0.1763\) to \(0.0753\) while utility changes from \(0.4773\) to \(0.4701\), which the paper describes as a 57% reduction [2505.03574].

Two later benchmark papers sharpen the limits of this framing. “When Benchmarks Lie” treats PromptGuard 2 as a production malicious-prompt classifier, specifically Meta’s Llama-Prompt-Guard-2-86M, using plain-text serialization, a 512-token context limit, chunking with max-score pooling, and default threshold \(0.5\) [2602.14161]. In that benchmark, PromptGuard 2 reports category-level detection rates of 36.7% on harmful requests, 48.5% on jailbreaks, 37.3% on indirect injection, 100.0% on extraction, 54.4% on mixed datasets, and benign FPR 0.4%, while being not evaluable on agentic attacks because it “cannot process tool schemas, as it lacks chat template support” [2602.14161]. “InjecGuard” likewise uses PromptGuard as a baseline and reports especially severe over-defense on benign trigger-bearing inputs: over-defense accuracy 0.88%, benign accuracy 26.82%, malicious accuracy 97.10%, and average 41.60, with NotInject subset scores of 2.65 for one-word and 0.00 for both two-word and three-word subsets [2410.22770].

This evidence suggests a recurrent distinction in the literature: PromptGuard is strongest when the task is narrow explicit-jailbreak detection in flat text, and substantially weaker when evaluation shifts toward subtle indirect attacks, structured agent transcripts, or benign hard negatives enriched with stereotypical trigger words [2505.03574].

## 2. Deployment roles in agent and repair pipelines

PromptGuard is repeatedly deployed as a pre-execution or pre-ingestion gate rather than as a repair model, action-policy engine, or post hoc code analyzer. In adversarial automated program repair, PromptGuard is placed at the public issue-submission boundary, where it is asked to analyze incoming bug reports for prompt injection and jailbreak attacks before the APR system begins reproduction or patch generation [2509.05372]. In that workflow, it is grouped with other pre-APR filters such as LlamaGuard variants, Granite Guardian, and custom LLM-based classifiers, while post-APR review is handled separately by GitHub CodeQL and Copilot review [2509.05372].

The protected backend in that study is SWE-agent v1.1.0 with SW-ReX v1.2.2 using Claude Sonnet (`claude-sonnet-4-20250514`) over five Python repositories: `django`, `flask`, `requests`, `pip`, and `scikit-learn` [2509.05372]. Against 51 adversarial bug reports, PromptGuard blocks 3/51 and PromptGuard2 blocks 0/51; per-attack breakdown shows PromptGuard catches 1 injection-style report and 2 Revert-CVE reports, while missing all Naive APR, CI/CD exploit, and Noise APR cases [2509.05372]. The same paper states that the best pre-APR filter is instead custom o4-mini with unstructured output at 24/51, while the pre-APR ensemble blocks 32/51 and the best combined defense reaches 37/51 when full pre-APR and post-APR ensembles are combined [2509.05372].

A related systems-level role appears in LlamaFirewall, where PromptGuard 2 monitors incoming text, especially messages with role user or tool, before dangerous content enters agent context [2505.03574]. The system-level logic is layered: PromptGuard screens external text streams, AlignmentCheck audits assistant-side reasoning, and CodeShield analyzes generated code [2505.03574]. That separation of concerns becomes important in later papers that argue input filtering alone is insufficient for agentic systems, but it also explains why PromptGuard remains attractive operationally: it is lightweight, locally deployable, and cheap enough to run continuously at the perimeter [2505.03574].

A plausible implication is that PromptGuard’s enduring deployment appeal comes less from demonstrated completeness than from its role as an inexpensive first-pass guardrail whose failures can, in principle, be complemented by downstream semantic or action-boundary controls.

## 3. Empirical limitations under distribution shift and semantic realism

Several studies argue that PromptGuard’s main weakness is not merely imperfect recall, but a structural mismatch between its training assumptions and the environments in which agents actually consume untrusted content. The most extreme example is AgentRedBench, which evaluates Meta’s “Llama Prompt Guard 2 22M” as an open-source baseline for indirect prompt injection over SaaS integration responses such as Gmail, Slack, Salesforce, Jira, calendars, HRIS, ATS, storage, and observability tools [2606.02240]. In this threat model, the attacker controls only tool-response content, and the relevant attacks are subtle underspecified-authorization scenarios such as `output_channel_url_relay`, `destination_hijack`, and `content_hijack` [2606.02240].

On the held-out cross-integration attack set at threshold 0.5, PromptGuard 2 reports \(\mathrm{TPR}=0.00\%\;[0.00,0.00]\) and \(\mathrm{FPR}=1.93\%\;[1.56,2.35]\), whereas AgentRedGuard-MiniLM at nearly matched parameter scale reports \(\mathrm{TPR}=99.75\%\;[99.51,99.95]\) and \(\mathrm{FPR}=0.37\%\;[0.21,0.56]\) [2606.02240]. On the full attack corpus of 14,846 rows, PromptGuard 2 TPR by active attack type is 0.02% for `output_channel_url_relay`, 0.16% for `destination_hijack`, and 0.03% for `content_hijack`; the paper therefore describes PromptGuard 2 as “uniformly blind to indirect injection through tool-response content” [2606.02240]. The stated explanation is a training-data mismatch: open-source guards such as PromptGuard 2 are characterized as trained on chat-style or general prompt-injection data rather than tool-response content [2606.02240].

The APR study reaches a related conclusion from a different angle. There, PromptGuard is nominally specialized for prompt injection and jailbreak detection, yet adversarial bug reports are crafted to look like plausible software-maintenance requests, not stereotypical “ignore previous instructions” strings [2509.05372]. The paper argues that the maliciousness lies in repository context and repair consequences rather than in obvious textual markers [2509.05372].

“When Benchmarks Lie” generalizes this into an evaluation critique. It proposes Leave-One-Dataset-Out evaluation and reports that conventional train-test splits overestimate detector performance, with an 8.4 percentage point AUC inflation for its own activation-based baseline and per-dataset held-out versus LODO accuracy gaps ranging from 1.2% to 25.4% [2602.14161]. Although it does not publish full per-dataset LODO scores for PromptGuard 2, it explicitly argues that production guardrails such as PromptGuard 2 are often measured on distributions too close to their training data and shows category-level weakness precisely where agent security is most relevant: 37.3% on indirect injection and architectural non-applicability to tool-use injection [2602.14161].

Together these papers support a consistent interpretation: PromptGuard performs materially better on explicit lexical jailbreak cues than on semantically plausible, domain-specific, or structured attack surfaces.

## 4. Failure modes in multi-agent, memory, and composition-based attacks

A distinct line of work argues that PromptGuard fails not merely because attacks are subtle, but because harmfulness is distributed across components and only emerges after routing, composition, or retrieval. “Conjunctive Prompt Attacks in Multi-Agent LLM Systems” evaluates PromptGuard-86M as one of several existing safety mechanisms applied either as pre-routing or post-generation filters in star, chain, and DAG topologies [2604.16543]. The attack uses a benign-looking trigger key in one user segment and a hidden adversarial template in one compromised remote agent, with activation condition
\[
\exists j \in \{1,\ldots,S\}\;\; \text{s.t.}\;\; (k \in s_j)\land(a_j=a^\star).
\]
Under this setup, PromptGuard-86M achieves F1 \(=30\) against Vanilla attacks and F1 \(=0\) under Full Optimization, the weakest result among the listed defenses in the vanilla condition and a total collapse under optimized attacks [2604.16543].

A related but more graph-theoretic argument appears in “Agents Under Siege,” where PromptGuard is used as a message-level safety mechanism on defended communication edges in a constrained multi-agent graph [2504.00218]. The attacker optimizes token flow via a minimum-cost maximum-flow objective
\[
\min \sum_{(u,v)\in\mathcal{E}} \mathrm{G}(u,v)\,f(u,v)
\]
under bandwidth and flow-conservation constraints, while also optimizing a permutation-invariant evasion loss over chunk orderings [2504.00218]. The paper’s PromptGuard-specific evidence is that shorter prompt fragments often evade detection, whereas longer chunks improve safety enforcement by providing more context for classification; PromptGuard-86M and its 4-bit quantized version are used explicitly in the appendix risk study [2504.00218]. The implication is that a local chunk classifier cannot reason over globally reassembled adversarial meaning.

FragFuse extends this compositional critique into long-term memory. It attacks access-control systems by splitting a prohibited request into rejection-responsive fragments, storing them in memory through benign-looking carrier queries, and later reconstructing the intent through retrieval and fusion [2606.15609]. PromptGuard is evaluated directly as a detector on benign, carrier, and attack queries. The benign-classification rates show that carrier queries are usually classified as benign—96.0 on RAP, 100.0 on SeeAct, 100.0 on OSAgent, and 60.2 on InspAgent—while attack queries also often evade PromptGuard, especially in SeeAct and OSAgent [2606.15609]. The paper’s broader conclusion is that single-turn text classification is structurally mismatched to attacks whose semantics are assembled across time via memory retrieval [2606.15609].

These studies collectively shift the critique from calibration to ontology: PromptGuard assumes maliciousness is locally visible in a single prompt or message, whereas many modern agent attacks distribute harmful semantics across routing paths, memory states, or cross-agent composition.

## 5. Comparative defenses and alternative design patterns

PromptGuard sits within a larger ecosystem of prompt-guarding strategies, and later work increasingly treats it as one layer in a broader defense-in-depth stack rather than a sufficient standalone solution. CourtGuard, for example, is a local three-role multiagent classifier in which a defense attorney argues benignity, a prosecution attorney argues injection, and a judge issues the final verdict [2510.19844]. On LLMail-Inject, attack detection accuracy is usually worse than a direct detector; on NotInject and benign Qualifire splits, benign accuracy is consistently better or equal, illustrating a precision–recall tradeoff in which bilateral deliberation reduces false positives but often loses attack recall [2510.19844]. CourtGuard is therefore a prompt-guarding design pattern, but not a direct PromptGuard variant.

ClawGuard represents a more radical departure. It addresses indirect prompt injection at the tool-call boundary by deriving a user-confirmed rule set before tool use and enforcing deterministic allow/deny/ambiguous decisions on every proposed tool action [2604.11790]. Its policy logic is expressed as
\[
\mathcal{R}=\mathcal{R}_{\mathrm{base}}\cup\mathcal{R}_{\mathrm{task}}
\]
with element-level and aggregate verdict functions over command, file, and network attributes [2604.11790]. Because it focuses on action authorization rather than malicious-text detection, it is conceptually complementary to PromptGuard rather than an input-side competitor [2604.11790].

MCP-Guard similarly broadens the defense surface. It is a three-stage pipeline for Model Context Protocol traffic, combining lightweight static scanning, a fine-tuned E5-based learnable detector, and an LLM arbitrator [2508.10991]. Stage 2 reports 96.01% accuracy, 96.65% precision, 93.52% recall, and 95.06% F1, while the full pipeline with GPT-4o-mini reports 96.01% accuracy, 91.49% precision, 99.54% recall, and 95.43% F1 on the reported benchmark [2508.10991]. This is not PromptGuard, but it illustrates a common trend: guard systems increasingly combine signatures, semantic models, and downstream adjudication rather than relying on a single prompt classifier.

SnapGuard illustrates modality shift. It addresses screenshot-based web agents with a visual stability indicator, selective contrast-polarity reversal, OCR, and action-oriented pattern detection [2604.25562]. When text-based PromptGuard is applied in that screenshot setting after OCR extraction, it achieves average TPR 0.07 and F1 0.13, compared with SnapGuard’s TPR 0.66 and F1 0.75 [2604.25562]. The contrast is again distributional: a text-only prompt detector degrades once the attack substrate becomes rendered visual content [2604.25562].

A reasonable synthesis is that the field is moving from generic prompt-level screening toward domain-specific, pipeline-aware, and often action-aware defenses. In that landscape, PromptGuard remains a useful lexical or classifier layer, but rarely the decisive one.

## 6. Homonymous systems outside prompt-injection detection

The term “PromptGuard” is also used for several unrelated systems, and any encyclopedic treatment must distinguish them from Meta’s prompt-injection guard family. One example is “PromptGuard at BLP-2025 Task 1,” a few-shot Bengali hate-speech classification framework combining chi-square keyword extraction and adaptive majority voting [2510.09771]. Its task is six-way label prediction over \(\{\texttt{none}, \texttt{sexism}, \texttt{abusive}, \texttt{profane}, \texttt{religious hate}, \texttt{political hate}\}\), and it reports micro-F1 \(=67.61\) on the official 10,000-test-instance evaluation, outperforming random, majority, and n-gram baselines in that shared-task setting [2510.09771]. This system is unrelated to prompt injection despite the shared name.

Another homonym is “PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models,” which trains a universal safety soft prompt \(P_*\) appended in embedding space to T2I prompts [2501.03544]. Its goal is NSFW moderation rather than LLM prompt-injection detection, and its main reported result is an average unsafe ratio of 5.84%, with SD v1.4 generation time 1.37 s/image versus 1.39 s/image for P-Guard [2501.03544]. The architecture is closer to an implicit system prompt for diffusion models than to a classifier guardrail [2501.03544].

A third homonymous usage appears in “PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations,” whose centerpiece is VulnGuard Prompt, a population-specific prompting framework with six modules including Input Classification, VulnGuard Prompting, Ethical Principles Integration, External Tool Interaction, Output Validation, and User-System Interaction [2509.08910]. That paper frames PromptGuard as a generation-time safety orchestration system rather than a detector, and its evidence base and theoretical claims are largely separate from the prompt-injection literature [2509.08910].

These naming collisions matter because cross-paper comparisons can otherwise become misleading. “PromptGuard” in current arXiv usage is not a single standardized artifact but a family of unrelated systems sharing a protective naming convention.

## 7. Research significance and open problems

The accumulated literature supports a relatively precise view of PromptGuard’s place in the defense landscape. As a lightweight prompt-injection classifier, it is operationally attractive because of small model size, low latency, and simple insertion at ingress points such as user prompts, tool outputs, or issue submissions [2505.03574]. It is demonstrably useful against explicit jailbreak-style inputs and can contribute to layered defenses in agent pipelines [2505.03574].

At the same time, multiple papers now converge on a common criticism. Adaptive attacks bypass PromptGuard-like systems at high rates when they use detector feedback, semantic reframing, or human-guided iteration; one paper reports adaptive ASR for PromptGuard of 94% with Gemini-2.5 Pro, 75% with GPT-5 Mini, 99% with Llama-3.3 70B, and 95% with MetaSecAlign 70B on AgentDojo, compared with much lower static ASR in the same table [2510.09023]. Training-time poisoning can also render prompt-injection filters ineffective: Cordyceps reports that PromptGuard detects explicit CoA injections at 1.000 across tasks, yet UCC stegotext positive rates remain near covertext—typically 0.005 or 0.002—because the malicious channel is semantic and poisoning-induced rather than explicitly lexical [2605.26595].

The most important unresolved question is therefore not whether PromptGuard works on obvious injections, but where its threat boundary actually lies. Current evidence suggests that the boundary is defined less by model size or architecture than by whether maliciousness is locally recoverable from the inspected text. When attacks are expressed as explicit jailbreak syntax, PromptGuard can be strong; when they are embedded in semantically plausible software issues, SaaS tool responses, screenshot content, cross-agent composition, memory retrieval chains, or poisoned semantic channels, PromptGuard’s assumptions no longer align with the attack surface [2509.05372].

This suggests a broader research direction. PromptGuard-like detectors remain valuable as inexpensive filters and telemetry sources, but trustworthy agent security increasingly appears to require composition-aware reasoning, retrieval- and memory-aware checks, protocol- or integration-specific classifiers, and deterministic controls at action boundaries. In that sense, PromptGuard is historically important as a canonical guard-model paradigm, but the surrounding literature increasingly treats it as a baseline layer within a larger systems security architecture rather than as a complete solution [2604.11790].

Source: https://www.emergentmind.com/topics/promptguard