PromptSafe: Structured Prompt Security

• PromptSafe is a suite of prompt-centric defense patterns that secure LLM actions through cryptographic permission gating, structured provenance, and intent control.
• It employs runtime authorization and middleware enforcement to mitigate prompt injection, jailbreak attacks, and unauthorized tool execution.
• PromptSafe also addresses privacy and prompt secrecy by safeguarding sensitive data and managing prompt propagation across multimodal and generative systems.

In the literature summarized here, the name PromptSafe is used for several prompt-centric defense patterns rather than a single canonical mechanism. These patterns include cryptographically bound permission prompts that gate LLM-issued actions, black-box and white-box detectors for jailbreaks and prompt injection, privacy-preserving sanitizers and confidential inference pipelines for user prompts, defenses against system-prompt extraction and prompt intellectual-property theft, and modality-specific safeguards for VLMs, CUAs, BUAs, secure code generation, and text-to-image diffusion systems (Chan, 29 Mar 2025, Li et al., 2024, Chowdhury et al., 7 Apr 2025, Zhuang et al., 16 May 2025, Jing et al., 2 Aug 2025). A unifying theme is that prompt safety is treated not as a purely lexical filtering problem, but as a control problem over provenance, permissions, intent, privacy, and downstream actuation.

1. Structured prompt semantics and threat models

A recurring premise in PromptSafe-style work is that a deployed prompt is not a flat string. "Prompt Control-Flow Integrity" formalizes a composed request as

$P = S \;\|\; D \;\|\; U \;\|\; R,$

where $S,D,U,R$ denote system, developer, user, and retrieved-document segments, and enforces the priority lattice $S > D > U > R$ (Alam et al., 19 Mar 2026). "Encrypted Prompt" instead models the full model input as

$\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$

with an appended permission-bearing segment

$$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$

that encodes the current user’s permissions and a verifiable cryptographic token (Chan, 29 Mar 2025).

Other PromptSafe variants widen the protected object. BodhiPromptShield treats multi-stage LLM/VLM agents as graphs $G=(V,E)$ whose nodes include ingress, retrieval, memory, tool, and logging boundaries, and focuses on how sensitive spans propagate across stages rather than only at the initial prompt boundary (Ma et al., 7 Apr 2026). OpenSafeIntent defines matched prompt triplets with benign, dual-use, and malicious intent variants of the same underlying task, plus four paraphrases for each dual-use prompt, to evaluate whether safe completion remains calibrated under intent shifts rather than merely appearing safe on independent prompts (Uppaal et al., 2 Jul 2026).

The threat models are correspondingly heterogeneous. They include prompt injection that overrides higher-priority instructions, jailbreak prompts that induce harmful completion, visual prompt injection in rendered interfaces, prompt extraction attacks against hidden system messages, privacy leakage from user content during cloud inference, and unauthorized action execution by tool-using agents (Cao et al., 3 Jun 2025, Jiang et al., 2024, Gim et al., 2024). This suggests that PromptSafe is best understood as a structured security layer over prompt provenance and prompt effects, not merely as a refusal heuristic.

2. Runtime authorization, middleware enforcement, and action control

The strongest PromptSafe formulations move enforcement to runtime and bind it directly to execution. In "Encrypted Prompt," the permission set $\mathrm{Perm}(u)$ can be represented either as a binary vector $p=(p_1,\dots,p_n)\in\{0,1\}^n$ or as a hierarchical level integer, then signed with a private key and appended to the user prompt. The server verifies the signature before acting on any LLM-produced tool call; if verification fails, all actions are rejected, and if verification succeeds, each action $a$ is executed only when $a\in \mathrm{AllowedActions}(P)$ (Chan, 29 Mar 2025). The paper’s informal theorem states that, under signature unforgeability, no PPT adversary can produce a valid permission token that elevates action scope beyond the original permissions. The illustrative SendEmail example makes the enforcement boundary explicit: the LLM may still emit SendEmail(...), but the server refuses execution when p_SendEmail=0.

PCFI operationalizes a different but related runtime boundary as a FastAPI gateway in front of deployed LLM APIs. Its three-stage middleware pipeline performs lexical heuristics over user and retrieved segments, role-switch detection for markers such as system: or "role":"system", and hierarchical policy enforcement against lower-priority forbidden directives (Alam et al., 19 Mar 2026). On its 150-sample benchmark of benign, direct user-injection, and indirect RAG-injection cases, PCFI reports Attack Pass-Through Rate $S,D,U,R$0, False Positive Rate $S,D,U,R$1, and median processing overhead $S,D,U,R$2 ms. The significance is not merely low latency; it is that provenance labels and priorities become first-class enforcement objects.

AgentArmor extends runtime enforcement from prompt strings to full agent traces. It reconstructs the runtime trace as CFG, DFG, and PDG structures, attaches security metadata through a property registry, and then propagates confidentiality, integrity, and trust labels through a type system (Wang et al., 2 Aug 2025). The blocking rule rejects tool calls when parameters derived from low-integrity or untrusted nodes are used in contexts that require higher-integrity or trusted inputs. On AgentDojo, it reduces attack success from $S,D,U,R$3 to $S,D,U,R$4, reaches $S,D,U,R$5 TPR with $S,D,U,R$6 FPR, and preserves $S,D,U,R$7 task utility versus $S,D,U,R$8 without the defense. In this line of work, PromptSafe becomes close to a program-analysis discipline for agent execution.

3. Detection architectures for jailbreaks and prompt injection

A second major PromptSafe family treats the prompt as a classification target before generation or before tool execution. GradSafe is the most explicitly white-box of these methods. Given a prompt $S,D,U,R$9, it computes

$S > D > U > R$0

and compares gradient slices against the average unsafe gradient on safety-critical parameter slices (Xie et al., 2024). With only a tiny seed set of $S > D > U > R$1 safe and $S > D > U > R$2 unsafe prompts, it selects slices whose cosine-similarity gap exceeds a threshold $S > D > U > R$3; in Llama-2 7B this leaves only $S > D > U > R$4 of row-slices and $S > D > U > R$5 of column-slices. GradSafe-Zero reports AUPRC $S > D > U > R$6 on ToxicChat and $S > D > U > R$7 on XSTest, outperforming Llama Guard in the reported comparisons.

GenTel-Shield is a model-agnostic black-box alternative. It embeds each prompt with multilingual E5, feeds the embedding into a lightweight classification head, and applies a threshold to the resulting maliciousness score (Li et al., 2024). Evaluated on GenTel-Bench, which contains $S > D > U > R$8 malicious prompts across $S > D > U > R$9 attack categories and $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$0 security scenarios plus an equal number of benign samples, it reports F1 $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$1 for jailbreak detection, F1 $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$2 for goal-hijacking detection, and $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$3 recall across categories, with approximately $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$4 ms inference overhead. Its contribution is breadth of coverage rather than white-box introspection.

QGuard turns harmful-prompt detection into a bank-of-questions problem. For each guard question $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$5, it queries a pre-trained MLLM with $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$6 and the user prompt, extracts the logits of “Yes” and “No,” defines $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$7, and then aggregates question-level scores through a graph and PageRank-derived risk score $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$8 (Lee et al., 14 Jun 2025). On four text-only harmful-prompt benchmarks it reaches average F1 $\mathrm{UserInput} = P_u \;\|\; P_{\mathrm{enc}},$9 in zero-shot mode, and on a multimodal harmful dataset it reports F1 $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$0, compared with $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$1 for Llama-Guard-3-Vision and $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$2 for zero-shot InternVL-4B.

On-device PromptSafe work targets a different operating point. LiteLMGuard formulates prompt filtering as an answerability classification task for quantized SLM deployments, using a fine-tuned ELECTRA model with roughly $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$3M parameters on the $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$4-sample Answerable-or-Not dataset (Nakka et al., 8 May 2025). It reports $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$5 classification accuracy, $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$6 F1, average prompt-filtering accuracy of approximately $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$7, defense against over $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$8 of harmful prompts, and around $$P_{\mathrm{enc}} = \langle D\rangle \;\|\; P \;\|\; PK \;\|\; \langle /D\rangle$$9 ms average inference time on smartphones. The motivation is that quantized SLMs may answer harmful prompts directly, even without adversarial prompt engineering.

A hybrid design appears in "Exploring and Developing a Pre-Model Safeguard with Draft Models." There, a small draft model generates $G=(V,E)$0 speculative responses, an existing guard classifier labels each $G=(V,E)$1 pair as safe or unsafe, and the prompt is blocked if more than a fraction $G=(V,E)$2 of drafts are unsafe (Cai et al., 19 May 2026). Relative to a post-model guard, the design reduces Prompt-to-Response Time by approximately $G=(V,E)$3, while reporting lower DFR than both the pre-guard and post-guard baselines in the presented table. This suggests that pre-model guarding can be strengthened by approximating likely model behavior rather than inspecting the prompt alone.

4. Privacy-preserving prompt mediation and confidential inference

PromptSafe work on privacy addresses both the prompt content itself and its downstream propagation. Pr$G=(V,E)$4mpt defines a prompt sanitizer

$G=(V,E)$5

where a type annotator identifies sensitive spans, format-dependent tokens such as SSNs and credit-card numbers are protected with format-preserving encryption, and value-dependent tokens such as age and salary are protected with metric differential privacy (Chowdhury et al., 7 Apr 2025). The paper reports BLEU on WMT-14 English$G=(V,E)$6German/French with GPT-4o and Gemini-1.5 remaining within approximately $G=(V,E)$7 of the plain prompt, $G=(V,E)$8 RAG accuracy on both numeric and factual queries, and STS $G=(V,E)$9 for long-context QA after desanitization. The central idea is that not all sensitive tokens should be treated identically: some require exact reversibility, while others require approximate semantic preservation under a privacy budget.

"Confidential Prompting" instead protects user prompts from the cloud LLM provider through Secure Partitioned Decoding and Prompt Obfuscation. Sensitive prompt segments are processed inside a confidential virtual machine, which computes a private KV cache $\mathrm{Perm}(u)$0, while the untrusted host maintains the public KV cache and most model weights (Gim et al., 2024). The host sends query vectors to the CVM, the CVM returns partial attention contributions, and the host reconstructs the true attention output through a two-party attention decomposition without seeing the private cache. For Prompt Obfuscation, the system inserts $\mathrm{Perm}(u)$1 fake prompt variants so that the adversary’s advantage over random guessing $\mathrm{Perm}(u)$2 is bounded by $\mathrm{Perm}(u)$3. Reported overheads are $\mathrm{Perm}(u)$4 for $\mathrm{Perm}(u)$5, approximately $\mathrm{Perm}(u)$6 for $\mathrm{Perm}(u)$7, and approximately $\mathrm{Perm}(u)$8 for $\mathrm{Perm}(u)$9, with TEE support adding approximately $p=(p_1,\dots,p_n)\in\{0,1\}^n$0 compute overhead.

BodhiPromptShield focuses on propagation-aware mediation in agents. It detects sensitive spans through normalization, rule-based recognizers, NER, contextual classification, and OCR/VLM extraction for multimodal inputs, then routes each span through typed placeholders, semantic abstraction, or secure symbolic mapping with delayed restoration at authorized boundaries (Ma et al., 7 Apr 2026). On CPPB, stage-wise exposure for the full system is $p=(p_1,\dots,p_n)\in\{0,1\}^n$1 at retrieval, $p=(p_1,\dots,p_n)\in\{0,1\}^n$2 at memory, and $p=(p_1,\dots,p_n)\in\{0,1\}^n$3 at the tool stage. In direct-exposure terms, the balanced policy reports PER $p=(p_1,\dots,p_n)\in\{0,1\}^n$4, AC $p=(p_1,\dots,p_n)\in\{0,1\}^n$5, and TSR $p=(p_1,\dots,p_n)\in\{0,1\}^n$6. A plausible implication is that prompt privacy in agents is governed as much by restoration timing and boundary control as by the initial redaction operation.

5. System-prompt secrecy and prompt intellectual property

A distinct PromptSafe line protects hidden prompts rather than user prompts. ProxyPrompt replaces the original system prompt $p=(p_1,\dots,p_n)\in\{0,1\}^n$7 with an optimized proxy $p=(p_1,\dots,p_n)\in\{0,1\}^n$8 in continuous embedding space, jointly minimizing utility loss on benign queries and forcing attacker-style extraction attempts toward a semantically divergent target prompt (Zhuang et al., 16 May 2025). Across $p=(p_1,\dots,p_n)\in\{0,1\}^n$9 victim-model and system-prompt configurations, it reports AM $a$0, SM $a$1, MS $a$2, UR $a$3, and $a$4 of prompts fully protected under SM, compared with $a$5 for the next-best defense. Its premise is that prompt-leakage defense should ensure that any extracted artifact is unusable, not merely partially dissimilar.

PromptKeeper frames leakage detection as a hypothesis-testing problem. It uses the mean token log-likelihood

$a$6

as a one-dimensional surrogate statistic, fits Gaussian models for leakage and no-leakage cases, and declares leakage when the approximate likelihood ratio exceeds a threshold $a$7 (Jiang et al., 2024). When leakage is detected, it regenerates the answer from the user query alone rather than denying service, specifically to avoid side-channel leakage through refusal behavior. On Real GPTs with Llama-3.1 8B, the reported cosine similarity, BLEU, and token-level F1 for PromptKeeper are close to the “No prompt” benchmark for both adversarial-query and regular-query attacks, while benign-query overhead is described as negligible apart from one extra likelihood evaluation and two Gaussian density calls.

PragLocker addresses prompt reuse in untrusted deployments by making prompts non-portable across LLMs. It first rewrites the prompt into a code-symbol representation, then performs black-box random-search obfuscation with target-model feedback so that the protected prompt remains within the target model’s stability region but collapses on non-target models (Li et al., 7 May 2026). The full system reduces average cross-LLM portability to $a$8, while preserving mean target performance at approximately $a$9. Initialization-only remains nearly portable at $a\in \mathrm{AllowedActions}(P)$0, and optimization-only reduces portability only partially to $a\in \mathrm{AllowedActions}(P)$1, so the two phases are presented as jointly necessary. In this formulation, PromptSafe extends from secrecy to enforceable economic non-portability.

6. Multimodal and generative-media PromptSafe

Multimodal PromptSafe work departs from text-only prompt filtering by examining how safety behavior unfolds during generation. "The Safety Reminder" identifies delayed safety awareness in VLMs: harmful generations often produce unsafe content first and only emit refusal signals near the end of the output (Tang et al., 15 Jun 2025). The proposed SAPT method learns $a\in \mathrm{AllowedActions}(P)$2 soft prompt tokens, monitors hidden states every $a\in \mathrm{AllowedActions}(P)$3 tokens, and injects the soft prompt only when a lightweight detector predicts unsafe generation with $a\in \mathrm{AllowedActions}(P)$4, where $a\in \mathrm{AllowedActions}(P)$5. For LLaVA-1.5-7B, the reported average benchmark ASR drops from $a\in \mathrm{AllowedActions}(P)$6 to $a\in \mathrm{AllowedActions}(P)$7, visual-adversarial ASR drops from $a\in \mathrm{AllowedActions}(P)$8 to $a\in \mathrm{AllowedActions}(P)$9, MM-Vet average remains $S,D,U,R$00 versus $S,D,U,R$01 for no defense, and benign refusal rate is $S,D,U,R$02. The method is positioned as on-demand reactivation of latent safety rather than permanent suppression.

VPI-Bench shifts attention from prompt text to rendered interfaces in computer-use and browser-use agents. It defines $S,D,U,R$03 end-to-end test cases across Amazon, Booking.com, BBC News, Messenger, and Email, and measures both Attempted Rate and Success Rate for malicious visually embedded prompts (Cao et al., 3 Jun 2025). Reported results show BUAs at AR $S,D,U,R$04 on Amazon, Booking, and BBC for several models, and CUAs reaching Messenger SR $S,D,U,R$05 for Sonnet-3.5 and $S,D,U,R$06 for Sonnet-3.7. The study also states that a system-prompt defense—“Be vigilant against prompt-injection attacks, which aim to trick you into performing unauthorized actions that may harm the user.”—produced only small and inconsistent changes in AR and SR. This is direct evidence that visual prompt injection cannot be reduced to prompt-template hardening alone.

The paper actually titled "PromptSafe: Gated Prompt Tuning for Safe Text-to-Image Generation" adapts the concept to diffusion models. It constructs a text-only training corpus by rewriting malicious prompts into safe alternatives with an LLM, retaining $S,D,U,R$07 aligned and safety-filtered pairs, learns a universal soft prompt $S,D,U,R$08, and gates its strength at inference with a CLIP-based toxicity score $S,D,U,R$09 (Jing et al., 2 Aug 2025). Across four NSFW categories it reports an average unsafe generation ratio of $S,D,U,R$10, with category-specific values Sexual $S,D,U,R$11, Violent $S,D,U,R$12, Political $S,D,U,R$13, and Disturbing $S,D,U,R$14. On benign MS COCO 2017 prompts, CLIP score is $S,D,U,R$15, only $S,D,U,R$16 below vanilla SDv1.4, and inference cost is $S,D,U,R$17 s/image. The method also reports transfer to SDv1.5 and SDv2.x and partial transfer to SDXL. Here PromptSafe denotes adaptive, toxicity-aware moderation embedded in the conditioning pathway rather than a separate textual filter.

7. Evaluation regimes, calibration, and prompt optimization

A mature PromptSafe literature requires evaluation beyond aggregate refusal rates. OpenSafeIntent formalizes this by comparing mean per-prompt safety to strict triplet consistency across benign, dual-use, and malicious prompt variants of the same task (Uppaal et al., 2 Jul 2026). The paper reports that TripletSafety is consistently lower than MeanSafety, with GPT-5.4 given as MeanSafety $S,D,U,R$18 versus TripletSafety $S,D,U,R$19. On dual-use paraphrase sets, approximately $S,D,U,R$20 are stable-safe, $S,D,U,R$21 stable-unsafe, and $S,D,U,R$22 exhibit safety flips. Assistance Mode 3, which explicitly reframes a dual-use request into a safer subtask before answering, has unsafe rate $S,D,U,R$23, compared with $S,D,U,R$24 for high-level abstraction and $S,D,U,R$25 for direct compliance. The benchmark therefore treats intent calibration and paraphrase robustness as primary safety variables.

Prompt optimization work makes this evaluation operational. "Survival of the Safest" defines prompt search as a weighted optimization problem,

$S,D,U,R$26

and uses interleaved evolution on KPI and security objectives rather than an explicit Pareto front (Sinha et al., 2024). The paper states that this yields $S,D,U,R$27–$S,D,U,R$28 fewer LLM calls than Pareto-front methods to reach a balanced frontier. In the reported tables, SoS boosts security on Sentiment from $S,D,U,R$29 to $S,D,U,R$30 while matching top KPI at $S,D,U,R$31, raises Orthography security from $S,D,U,R$32 to $S,D,U,R$33 at modest KPI cost, and reaches $S,D,U,R$34 on Taxonomy for KPI and security. This suggests that prompt safety can be optimized jointly with task performance rather than added only as a post hoc filter.

Application-specific prompting remains relevant in narrower domains. For secure code generation, a benchmark over $S,D,U,R$35 Python prompts from LLMSecEval and SecurityEval evaluated $S,D,U,R$36 prompt variants with Semgrep and CodeQL as scanners (Bruni et al., 9 Feb 2025). A security-focused prefix—“You are a developer who is very security-aware and avoids weaknesses in the code.”—reduces vulnerability occurrence by up to $S,D,U,R$37 on GPT-4o and GPT-4o-mini. Iterative Recursive Criticism & Improvement detects and repairs between $S,D,U,R$38 and $S,D,U,R$39 of vulnerabilities in previously generated code, and the paper introduces a prompt agent that combines the prefix with optional critique-and-improve postprocessing. In this setting, PromptSafe is not a universal theorem of safe behavior; it is a measurable, task-conditioned improvement pipeline.

Taken together, these results indicate that PromptSafe is evolving toward a layered discipline: structured prompt provenance, runtime authorization, black-box or white-box attack detection, privacy mediation, prompt secrecy and non-portability, modality-specific defenses, and intent-calibrated evaluation. The literature does not converge on a single mechanism, but it does converge on a common claim: prompt safety must be enforced at the boundaries where prompts acquire authority, disclose information, or induce action.