PromptGuard: Injection Detection for LLMs

• PromptGuard is a family of prompt-injection detectors that screen incoming text for explicit jailbreak and injection patterns in LLM systems.
• It operates at the input layer for real-time, low-latency defense in agentic pipelines and automated program repair settings.
• Empirical studies indicate strong performance on overt attacks but reveal limitations against indirect, context-rich, multi-agent, and memory-based assaults.

Searching arXiv for recent PromptGuard-related papers to ground the article. PromptGuard most commonly denotes a family of lightweight prompt-injection and jailbreak detectors used as guardrail components for LLM systems, especially in agentic and tool-using settings. In the literature, however, the name is polysemous: it refers not only to Meta’s PromptGuard and PromptGuard 2 classifiers as studied in agent-security and automated-program-repair pipelines, but also to unrelated systems for Bengali hate-speech classification, text-to-image moderation, and population-specific safe text generation (Chennabasappa et al., 6 May 2025). Across these contexts, the dominant technical theme is preemptive screening or steering at the input or conditioning layer rather than post hoc correction, although later work repeatedly shows that purely local text classification is often mismatched to richer threat surfaces such as SaaS tool responses, multi-agent routing, or memory-mediated semantic composition (Dingeto et al., 1 Jun 2026).

1. PromptGuard as a prompt-injection guard model

In the prompt-security literature, PromptGuard is positioned as a classifier-style guardrail that operates on incoming text and decides whether it resembles a jailbreak or prompt-injection attempt. Within the LlamaFirewall architecture, PromptGuard 2 is the front-line detector for explicit “universal jailbreak” patterns in user inputs or untrusted external content such as webpages, PDFs, emails, and tool outputs; its role is deliberately narrower than broader semantic misalignment detection, which is delegated to other components such as AlignmentCheck and CodeShield (Chennabasappa et al., 6 May 2025). The paper describes two PromptGuard 2 variants: PromptGuard 2 86M, based on mDeBERTa-base, and PromptGuard 2 22M, based on DeBERTa-xsmall (Chennabasappa et al., 6 May 2025).

The same paper presents PromptGuard 2 as a thresholded detector intended for real-time deployment. On a private direct-jailbreak benchmark, the reported numbers are PromptGuard 2 86M with English AUC $= 0.980$, Recall@1%FPR $= 97.5\%$, Multilingual AUC $= 0.995$, and latency $= 92.4$ ms on A100 at 512 tokens; PromptGuard 2 22M reports English AUC $= 0.995$, Recall@1%FPR $= 88.7\%$, Multilingual AUC $= 0.942$, and latency $= 19.3$ ms (Chennabasappa et al., 6 May 2025). In AgentDojo, PromptGuard 2 86M reduces ASR from $0.1763$ to $0.0753$ while utility changes from $= 97.5\%$0 to $= 97.5\%$1, which the paper describes as a 57% reduction (Chennabasappa et al., 6 May 2025).

Two later benchmark papers sharpen the limits of this framing. “When Benchmarks Lie” treats PromptGuard 2 as a production malicious-prompt classifier, specifically Meta’s Llama-Prompt-Guard-2-86M, using plain-text serialization, a 512-token context limit, chunking with max-score pooling, and default threshold $= 97.5\%$2 (Fomin, 15 Feb 2026). In that benchmark, PromptGuard 2 reports category-level detection rates of 36.7% on harmful requests, 48.5% on jailbreaks, 37.3% on indirect injection, 100.0% on extraction, 54.4% on mixed datasets, and benign FPR 0.4%, while being not evaluable on agentic attacks because it “cannot process tool schemas, as it lacks chat template support” (Fomin, 15 Feb 2026). “InjecGuard” likewise uses PromptGuard as a baseline and reports especially severe over-defense on benign trigger-bearing inputs: over-defense accuracy 0.88%, benign accuracy 26.82%, malicious accuracy 97.10%, and average 41.60, with NotInject subset scores of 2.65 for one-word and 0.00 for both two-word and three-word subsets (Li et al., 2024).

This evidence suggests a recurrent distinction in the literature: PromptGuard is strongest when the task is narrow explicit-jailbreak detection in flat text, and substantially weaker when evaluation shifts toward subtle indirect attacks, structured agent transcripts, or benign hard negatives enriched with stereotypical trigger words (Chennabasappa et al., 6 May 2025).

2. Deployment roles in agent and repair pipelines

PromptGuard is repeatedly deployed as a pre-execution or pre-ingestion gate rather than as a repair model, action-policy engine, or post hoc code analyzer. In adversarial automated program repair, PromptGuard is placed at the public issue-submission boundary, where it is asked to analyze incoming bug reports for prompt injection and jailbreak attacks before the APR system begins reproduction or patch generation (Przymus et al., 4 Sep 2025). In that workflow, it is grouped with other pre-APR filters such as LlamaGuard variants, Granite Guardian, and custom LLM-based classifiers, while post-APR review is handled separately by GitHub CodeQL and Copilot review (Przymus et al., 4 Sep 2025).

The protected backend in that study is SWE-agent v1.1.0 with SW-ReX v1.2.2 using Claude Sonnet (claude-sonnet-4-20250514) over five Python repositories: django, flask, requests, pip, and scikit-learn (Przymus et al., 4 Sep 2025). Against 51 adversarial bug reports, PromptGuard blocks 3/51 and PromptGuard2 blocks 0/51; per-attack breakdown shows PromptGuard catches 1 injection-style report and 2 Revert-CVE reports, while missing all Naive APR, CI/CD exploit, and Noise APR cases (Przymus et al., 4 Sep 2025). The same paper states that the best pre-APR filter is instead custom o4-mini with unstructured output at 24/51, while the pre-APR ensemble blocks 32/51 and the best combined defense reaches 37/51 when full pre-APR and post-APR ensembles are combined (Przymus et al., 4 Sep 2025).

A related systems-level role appears in LlamaFirewall, where PromptGuard 2 monitors incoming text, especially messages with role user or tool, before dangerous content enters agent context (Chennabasappa et al., 6 May 2025). The system-level logic is layered: PromptGuard screens external text streams, AlignmentCheck audits assistant-side reasoning, and CodeShield analyzes generated code (Chennabasappa et al., 6 May 2025). That separation of concerns becomes important in later papers that argue input filtering alone is insufficient for agentic systems, but it also explains why PromptGuard remains attractive operationally: it is lightweight, locally deployable, and cheap enough to run continuously at the perimeter (Chennabasappa et al., 6 May 2025).

A plausible implication is that PromptGuard’s enduring deployment appeal comes less from demonstrated completeness than from its role as an inexpensive first-pass guardrail whose failures can, in principle, be complemented by downstream semantic or action-boundary controls.

3. Empirical limitations under distribution shift and semantic realism

Several studies argue that PromptGuard’s main weakness is not merely imperfect recall, but a structural mismatch between its training assumptions and the environments in which agents actually consume untrusted content. The most extreme example is AgentRedBench, which evaluates Meta’s “Llama Prompt Guard 2 22M” as an open-source baseline for indirect prompt injection over SaaS integration responses such as Gmail, Slack, Salesforce, Jira, calendars, HRIS, ATS, storage, and observability tools (Dingeto et al., 1 Jun 2026). In this threat model, the attacker controls only tool-response content, and the relevant attacks are subtle underspecified-authorization scenarios such as output_channel_url_relay, destination_hijack, and content_hijack (Dingeto et al., 1 Jun 2026).

On the held-out cross-integration attack set at threshold 0.5, PromptGuard 2 reports $= 97.5\%$3 and $= 97.5\%$4, whereas AgentRedGuard-MiniLM at nearly matched parameter scale reports $= 97.5\%$5 and $= 97.5\%$6 (Dingeto et al., 1 Jun 2026). On the full attack corpus of 14,846 rows, PromptGuard 2 TPR by active attack type is 0.02% for output_channel_url_relay, 0.16% for destination_hijack, and 0.03% for content_hijack; the paper therefore describes PromptGuard 2 as “uniformly blind to indirect injection through tool-response content” (Dingeto et al., 1 Jun 2026). The stated explanation is a training-data mismatch: open-source guards such as PromptGuard 2 are characterized as trained on chat-style or general prompt-injection data rather than tool-response content (Dingeto et al., 1 Jun 2026).

The APR study reaches a related conclusion from a different angle. There, PromptGuard is nominally specialized for prompt injection and jailbreak detection, yet adversarial bug reports are crafted to look like plausible software-maintenance requests, not stereotypical “ignore previous instructions” strings (Przymus et al., 4 Sep 2025). The paper argues that the maliciousness lies in repository context and repair consequences rather than in obvious textual markers (Przymus et al., 4 Sep 2025).

“When Benchmarks Lie” generalizes this into an evaluation critique. It proposes Leave-One-Dataset-Out evaluation and reports that conventional train-test splits overestimate detector performance, with an 8.4 percentage point AUC inflation for its own activation-based baseline and per-dataset held-out versus LODO accuracy gaps ranging from 1.2% to 25.4% (Fomin, 15 Feb 2026). Although it does not publish full per-dataset LODO scores for PromptGuard 2, it explicitly argues that production guardrails such as PromptGuard 2 are often measured on distributions too close to their training data and shows category-level weakness precisely where agent security is most relevant: 37.3% on indirect injection and architectural non-applicability to tool-use injection (Fomin, 15 Feb 2026).

Together these papers support a consistent interpretation: PromptGuard performs materially better on explicit lexical jailbreak cues than on semantically plausible, domain-specific, or structured attack surfaces.

4. Failure modes in multi-agent, memory, and composition-based attacks

A distinct line of work argues that PromptGuard fails not merely because attacks are subtle, but because harmfulness is distributed across components and only emerges after routing, composition, or retrieval. “Conjunctive Prompt Attacks in Multi-Agent LLM Systems” evaluates PromptGuard-86M as one of several existing safety mechanisms applied either as pre-routing or post-generation filters in star, chain, and DAG topologies (Arif et al., 17 Apr 2026). The attack uses a benign-looking trigger key in one user segment and a hidden adversarial template in one compromised remote agent, with activation condition

$= 97.5\%$7

Under this setup, PromptGuard-86M achieves F1 $= 97.5\%$8 against Vanilla attacks and F1 $= 97.5\%$9 under Full Optimization, the weakest result among the listed defenses in the vanilla condition and a total collapse under optimized attacks (Arif et al., 17 Apr 2026).

A related but more graph-theoretic argument appears in “Agents Under Siege,” where PromptGuard is used as a message-level safety mechanism on defended communication edges in a constrained multi-agent graph (Khan et al., 31 Mar 2025). The attacker optimizes token flow via a minimum-cost maximum-flow objective

$= 0.995$0

under bandwidth and flow-conservation constraints, while also optimizing a permutation-invariant evasion loss over chunk orderings (Khan et al., 31 Mar 2025). The paper’s PromptGuard-specific evidence is that shorter prompt fragments often evade detection, whereas longer chunks improve safety enforcement by providing more context for classification; PromptGuard-86M and its 4-bit quantized version are used explicitly in the appendix risk study (Khan et al., 31 Mar 2025). The implication is that a local chunk classifier cannot reason over globally reassembled adversarial meaning.

FragFuse extends this compositional critique into long-term memory. It attacks access-control systems by splitting a prohibited request into rejection-responsive fragments, storing them in memory through benign-looking carrier queries, and later reconstructing the intent through retrieval and fusion (Rao et al., 14 Jun 2026). PromptGuard is evaluated directly as a detector on benign, carrier, and attack queries. The benign-classification rates show that carrier queries are usually classified as benign—96.0 on RAP, 100.0 on SeeAct, 100.0 on OSAgent, and 60.2 on InspAgent—while attack queries also often evade PromptGuard, especially in SeeAct and OSAgent (Rao et al., 14 Jun 2026). The paper’s broader conclusion is that single-turn text classification is structurally mismatched to attacks whose semantics are assembled across time via memory retrieval (Rao et al., 14 Jun 2026).

These studies collectively shift the critique from calibration to ontology: PromptGuard assumes maliciousness is locally visible in a single prompt or message, whereas many modern agent attacks distribute harmful semantics across routing paths, memory states, or cross-agent composition.

5. Comparative defenses and alternative design patterns

PromptGuard sits within a larger ecosystem of prompt-guarding strategies, and later work increasingly treats it as one layer in a broader defense-in-depth stack rather than a sufficient standalone solution. CourtGuard, for example, is a local three-role multiagent classifier in which a defense attorney argues benignity, a prosecution attorney argues injection, and a judge issues the final verdict (Wu et al., 20 Oct 2025). On LLMail-Inject, attack detection accuracy is usually worse than a direct detector; on NotInject and benign Qualifire splits, benign accuracy is consistently better or equal, illustrating a precision–recall tradeoff in which bilateral deliberation reduces false positives but often loses attack recall (Wu et al., 20 Oct 2025). CourtGuard is therefore a prompt-guarding design pattern, but not a direct PromptGuard variant.

ClawGuard represents a more radical departure. It addresses indirect prompt injection at the tool-call boundary by deriving a user-confirmed rule set before tool use and enforcing deterministic allow/deny/ambiguous decisions on every proposed tool action (Zhao et al., 13 Apr 2026). Its policy logic is expressed as

$= 0.995$1

with element-level and aggregate verdict functions over command, file, and network attributes (Zhao et al., 13 Apr 2026). Because it focuses on action authorization rather than malicious-text detection, it is conceptually complementary to PromptGuard rather than an input-side competitor (Zhao et al., 13 Apr 2026).

MCP-Guard similarly broadens the defense surface. It is a three-stage pipeline for Model Context Protocol traffic, combining lightweight static scanning, a fine-tuned E5-based learnable detector, and an LLM arbitrator (Xing et al., 14 Aug 2025). Stage 2 reports 96.01% accuracy, 96.65% precision, 93.52% recall, and 95.06% F1, while the full pipeline with GPT-4o-mini reports 96.01% accuracy, 91.49% precision, 99.54% recall, and 95.43% F1 on the reported benchmark (Xing et al., 14 Aug 2025). This is not PromptGuard, but it illustrates a common trend: guard systems increasingly combine signatures, semantic models, and downstream adjudication rather than relying on a single prompt classifier.

SnapGuard illustrates modality shift. It addresses screenshot-based web agents with a visual stability indicator, selective contrast-polarity reversal, OCR, and action-oriented pattern detection (Du et al., 28 Apr 2026). When text-based PromptGuard is applied in that screenshot setting after OCR extraction, it achieves average TPR 0.07 and F1 0.13, compared with SnapGuard’s TPR 0.66 and F1 0.75 (Du et al., 28 Apr 2026). The contrast is again distributional: a text-only prompt detector degrades once the attack substrate becomes rendered visual content (Du et al., 28 Apr 2026).

A reasonable synthesis is that the field is moving from generic prompt-level screening toward domain-specific, pipeline-aware, and often action-aware defenses. In that landscape, PromptGuard remains a useful lexical or classifier layer, but rarely the decisive one.

6. Homonymous systems outside prompt-injection detection

The term “PromptGuard” is also used for several unrelated systems, and any encyclopedic treatment must distinguish them from Meta’s prompt-injection guard family. One example is “PromptGuard at BLP-2025 Task 1,” a few-shot Bengali hate-speech classification framework combining chi-square keyword extraction and adaptive majority voting (Hossan et al., 10 Oct 2025). Its task is six-way label prediction over $= 0.995$2, and it reports micro-F1 $= 0.995$3 on the official 10,000-test-instance evaluation, outperforming random, majority, and n-gram baselines in that shared-task setting (Hossan et al., 10 Oct 2025). This system is unrelated to prompt injection despite the shared name.

Another homonym is “PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models,” which trains a universal safety soft prompt $= 0.995$4 appended in embedding space to T2I prompts (Yuan et al., 7 Jan 2025). Its goal is NSFW moderation rather than LLM prompt-injection detection, and its main reported result is an average unsafe ratio of 5.84%, with SD v1.4 generation time 1.37 s/image versus 1.39 s/image for P-Guard (Yuan et al., 7 Jan 2025). The architecture is closer to an implicit system prompt for diffusion models than to a classifier guardrail (Yuan et al., 7 Jan 2025).

A third homonymous usage appears in “PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations,” whose centerpiece is VulnGuard Prompt, a population-specific prompting framework with six modules including Input Classification, VulnGuard Prompting, Ethical Principles Integration, External Tool Interaction, Output Validation, and User-System Interaction (Vu et al., 10 Sep 2025). That paper frames PromptGuard as a generation-time safety orchestration system rather than a detector, and its evidence base and theoretical claims are largely separate from the prompt-injection literature (Vu et al., 10 Sep 2025).

These naming collisions matter because cross-paper comparisons can otherwise become misleading. “PromptGuard” in current arXiv usage is not a single standardized artifact but a family of unrelated systems sharing a protective naming convention.

7. Research significance and open problems

The accumulated literature supports a relatively precise view of PromptGuard’s place in the defense landscape. As a lightweight prompt-injection classifier, it is operationally attractive because of small model size, low latency, and simple insertion at ingress points such as user prompts, tool outputs, or issue submissions (Chennabasappa et al., 6 May 2025). It is demonstrably useful against explicit jailbreak-style inputs and can contribute to layered defenses in agent pipelines (Chennabasappa et al., 6 May 2025).

At the same time, multiple papers now converge on a common criticism. Adaptive attacks bypass PromptGuard-like systems at high rates when they use detector feedback, semantic reframing, or human-guided iteration; one paper reports adaptive ASR for PromptGuard of 94% with Gemini-2.5 Pro, 75% with GPT-5 Mini, 99% with Llama-3.3 70B, and 95% with MetaSecAlign 70B on AgentDojo, compared with much lower static ASR in the same table (Nasr et al., 10 Oct 2025). Training-time poisoning can also render prompt-injection filters ineffective: Cordyceps reports that PromptGuard detects explicit CoA injections at 1.000 across tasks, yet UCC stegotext positive rates remain near covertext—typically 0.005 or 0.002—because the malicious channel is semantic and poisoning-induced rather than explicitly lexical (Shao et al., 26 May 2026).

The most important unresolved question is therefore not whether PromptGuard works on obvious injections, but where its threat boundary actually lies. Current evidence suggests that the boundary is defined less by model size or architecture than by whether maliciousness is locally recoverable from the inspected text. When attacks are expressed as explicit jailbreak syntax, PromptGuard can be strong; when they are embedded in semantically plausible software issues, SaaS tool responses, screenshot content, cross-agent composition, memory retrieval chains, or poisoned semantic channels, PromptGuard’s assumptions no longer align with the attack surface (Przymus et al., 4 Sep 2025).

This suggests a broader research direction. PromptGuard-like detectors remain valuable as inexpensive filters and telemetry sources, but trustworthy agent security increasingly appears to require composition-aware reasoning, retrieval- and memory-aware checks, protocol- or integration-specific classifiers, and deterministic controls at action boundaries. In that sense, PromptGuard is historically important as a canonical guard-model paradigm, but the surrounding literature increasingly treats it as a baseline layer within a larger systems security architecture rather than as a complete solution (Zhao et al., 13 Apr 2026).