Prompt-Level Defense Framework

• Prompt-Level Defense Framework is a set of techniques that analyze and modify language prompts to block adversarial inputs without altering core model parameters.
• It employs both prevention and detection strategies including guard prefixes, prompt rewriting, and multi-agent orchestration to safeguard task integrity.
• Framework effectiveness is measured through metrics like attack success rate, utility preservation, and response matching, balancing defense strength with operational efficiency.

A prompt-level defense framework refers to any strategy or system that mitigates, detects, or neutralizes attacks or undesired behaviors in LLMs by directly analyzing, modifying, augmenting, or instrumenting the language-level inputs and outputs—i.e., system prompts, user inputs, conversation history, or structured data destined for LLM consumption. Such frameworks aim to enforce security, alignment, and task integrity primarily through operations on prompts and model responses, independent of model parameter changes or access to internal weights. The domain now encompasses runtime defense layers, guardrail agents, adversarial pre/post-processing, self-regulation modules, prompt hardening, and formal prevention and detection schemes.

1. Formalization and Threat Models

Prompt-level defenses are rigorously positioned in the literature through the input–output formalism where an LLM-integrated system receives a concatenated prompt $\mathbf{p} = s^\mathrm{t} \oplus x^\mathrm{t}$, with $s^\mathrm{t}$ an instruction or system prompt and $x^\mathrm{t}$ user or untrusted input. A prompt injection attack is any adversarial transformation of $x^\mathrm{t}$ that causes the LLM’s output to serve an unauthorized or malicious objective rather than the intended $s^\mathrm{t}$ (Liu et al., 2023).

Defense frameworks are antagonistic to an adversary capable of black-box input manipulation, iterative rewriting, or context/role overriding, and can operate in a white-box (for LLM fine-tuning or parameter-based controls) or black-box (where only prompt/response-level access is assumed) setting. The defender is typically limited to pre-processing, post-processing, or orchestration around the LLM core.

Defenses are evaluated by their ability to preserve performance under no attack (PNA), minimize attack success score (ASS), and suppress response matching rate (MR) between an adversarial and a direct attack prompt, within established benchmarks (Liu et al., 2023, Shi et al., 21 Jul 2025, Hossain et al., 16 Sep 2025).

2. Core Approaches in Prompt-Level Defense

2.1 Prevention and Sanitization

Prevention-based frameworks operate by:

• Prepending learned or optimized guard prefixes (Prompt Adversarial Tuning, PAT (Mo et al., 9 Feb 2024)) or interpretable suffixes (Defensive Prompt Patch, DPP (Xiong et al., 30 May 2024)) to resist or nullify adversarial instructions.
• Employing prompt rewriting or paraphrasing to remove or mask visible evidence of injections (Liu et al., 2023).
• Input compression to extract and surface the true task intent (SecurityLingua (Li et al., 15 Jun 2025)).
• Proxy prompt replacement to mislead extraction and preserve utility (ProxyPrompt (Zhuang et al., 16 May 2025)).
• Runtime prompt shield optimization to suppress system prompt leakage (Prompt Sensitivity Minimization, PSM (Jawad et al., 20 Nov 2025)).

Theoretical objectives typically combine adversarial loss minimization (for worst-case defense) and utility preservation constraints: $$\min_\delta \,\, \alpha \, \mathbb{E}_{\text{benign}}[-\log p_\theta(y^\text{benign}\mid \delta\oplus x^\text{benign})] + (1-\alpha) \mathbb{E}_{\text{harm}} \Bigl( \max_{x^-\in\mathcal{A}(x)} -\log p_\theta(y_\text{def}\mid \delta\oplus x^-) \Bigr)$$ (Mo et al., 9 Feb 2024). Prompt-level hardening via PSM is formalized as constrained optimization: $$\min_{S}\;L\bigl(P\oplus S,A\bigr) \quad\text{s.t.}\quad U\bigl(P\oplus S\bigr)\ge\tau$$ where $L$ is a leakage metric and $U$ utility preservation (Jawad et al., 20 Nov 2025).

2.2 Detection and Gatekeeping

Detection-based prompt frameworks leverage:

• Multi-stage, self-tuning moderators analyzing user prompts and model answers with behavior-weighted scoring (RTST, (Zhang, 10 Aug 2025));
• Iterative prompt evaluation with weighted aggregation of LLM safety ratings for robust judgement (DATDP, (Armstrong et al., 1 Feb 2025));
• Intent-invariance and semantic clustering to detect unauthorized task shifts (PromptSleuth, (Wang et al., 28 Aug 2025));
• Self-consciousness modules where the LLM self-assesses harmfulness probabilities and applies ensemble arbitration (Self-Consciousness Defense (Huang et al., 4 Aug 2025));
• Guardrail LLMs to scan and excise injected substrings (PromptArmor, (Shi et al., 21 Jul 2025));
• Proactive “secret challenge” injection for universal zero-knowledge adversarial detection (Liu et al., 2023).

Detection can be formalized as a binary hypothesis test or nearest-neighbor inference in an embedding space: $$\mathrm{Detect}(p') = \begin{cases} 1 & \text{if } d(I(p),I(p')) > \tau \ 0 & \text{otherwise} \end{cases}$$ with $I(\cdot)$ an intent encoder and $d$ a distance metric (Wang et al., 28 Aug 2025).

2.3 Multi-Agent and Orchestrated Defenses

Some advanced defenses instantiate multi-agent pipelines, delegating prompt inspection, guarding, and redaction to dedicated LLM agents arranged either in coordinator chains (pre-inspection) or post-generation guards. These architectures allow real-time, layered filtering, redaction, and failover (Hossain et al., 16 Sep 2025).

2.4 Adversarial and Co-Evolutionary Learning

Automated adversarial training can be realized at the prompt level through co-evolutionary loops (AEGIS (Liu et al., 27 Aug 2025)), where an attacker and defender iteratively optimize their prompts via black-box, gradient-inspired perturbations and multi-objective scoring, simulating an arms race. This continuous adversarial prompting yields robust detection heuristics and highly evasive attack examples.

Self-learning, parameter-free systems like ShieldLearner (Ni et al., 16 Feb 2025) develop a Pattern Atlas of distilled attack signatures, meta-analysis rules, and continual adversarial augmentation to defend against shifting attack paradigms without retraining the base LLM.

3. Defensive Objectives, Metrics, and Trade-offs

Defensive prompt frameworks are quantitatively evaluated according to:

• Attack Success Rate (ASR): proportion of harmful objectives achieved post-defense (Pasquini et al., 28 Oct 2024, Xiong et al., 30 May 2024, Li et al., 15 Jun 2025);
• Benign Utility/Win Rate: preservation of correct outputs on clean tasks;
• Refusal Rate (RR), False Positive Rate (FPR), and False Negative Rate (FNR);
• Detection F1 scores and overhead (latency, token cost, throughput);
• Specific leakage metrics for prompt extraction, e.g., ROUGE-L recall, semantic match, cosine similarity (Jawad et al., 20 Nov 2025, Zhuang et al., 16 May 2025, Jiang et al., 18 Dec 2024);
• For multi-turn and iterative attack mitigations: rounds to completion, drift metrics, and online self-tuning adaptation rates (Zhang, 10 Aug 2025, Kaneko et al., 19 Oct 2025).

A trade-off typically emerges:

• Increasing defense strength often slightly reduces utility or increases refusal on benign inputs;
• More powerful, multi-stage detection frameworks and co-evolution strategies incur increase in latency and operational cost, though human-readability and explainability can be maintained with careful system design (DPP, ShieldLearner).

4. Implementation Patterns and Exemplary Frameworks

The implementation landscape spans lightweight, modular add-ons to the LLM API layer, orchestrated multi-agent architectures, and offline/online prompt search or fine-tuning. Selected frameworks and patterns include:

• Mantis: Autonomous decoy–detection–injection pipeline for hack-back or tarpit sabotage during LLM-driven cyberattacks (Pasquini et al., 28 Oct 2024).
• PromptArmor: LLM guardrail detecting and extracting injected substrings, robust to adaptive attacks (Shi et al., 21 Jul 2025).
• RTST Moderator: Real-time self-tuning with discrete behavior-weight adaptation and rolling drift monitoring (Zhang, 10 Aug 2025).
• PromptSleuth: Semantic intent graph construction and cluster-based anomaly detection (Wang et al., 28 Aug 2025).
• AEGIS: Co-evolutionary, black-box gradient prompt learning with gradient buffering and multi-route objective scoring (Liu et al., 27 Aug 2025).
• Prompt Sensitivity Minimization (PSM): Black-box LLM-guided shield-append optimization, balancing leakage loss and utility of shielded system prompts (Jawad et al., 20 Nov 2025).
• Defensive Prompt Patch (DPP): Suffix optimization through hierarchical, LLM-assisted genetic algorithms, maintaining interpretability (Xiong et al., 30 May 2024).
• ShieldLearner: Parameter-free, self-learning distillation and adaptive augmentation without model weight modification (Ni et al., 16 Feb 2025).

5. Empirical Effectiveness and Limitations

Defenses such as PromptArmor and SecurityLingua consistently reduce ASR below 1% on AgentDojo and JailbreakBench while maintaining negligible impact on utility (Shi et al., 21 Jul 2025, Li et al., 15 Jun 2025). Mantis achieves ≥95% defender success and <3% attacker success against LLM-automated cyberattacks (Pasquini et al., 28 Oct 2024). ShieldLearner and RTST prove highly adaptive, with ShieldLearner achieving 0% ASR on standard adversarial suites and as low as 11.81% ASR on hard, stealthy prompts (Ni et al., 16 Feb 2025). PromptSleuth achieves sub-0.1% FPR/FNR on its hardest benchmarks (Wang et al., 28 Aug 2025). ProxyPrompt and PromptKeeper provide >94% protection for system prompt extraction without utility degradation (Zhuang et al., 16 May 2025, Jiang et al., 18 Dec 2024).

Key limitations include:

• Susceptibility to highly adaptive or semantic attacks not represented in training or heuristic rules;
• Some frameworks are sensitive to prompt-style drift or adversary pre-sanitization;
• Overhead in complex, multi-agent or co-evolutionary pipelines, though approaches like SecurityLingua and DPP maintain near-zero impact on latency and resource usage;
• Maintenance of modular components (behavior sets, pattern atlases, update buffers) and periodic re-optimization to keep pace with an evolving attack landscape;
• For detection, FNR can increase with under-sized semantic models or vague system prompts (Wang et al., 28 Aug 2025).

6. Prospects for Future Research

Recent literature identifies several directions for further development:

• Automated, reinforcement or gradient-based payload generation for proactive/adaptive defense (Mantis, PSM);
• Hybridization of semantic, syntactic, and behavioral detection for higher resilience;
• Model fingerprinting for trigger-tuning and LLM-aware defenses;
• Expanding coverage to multi-turn conversational attacks, tool-augmented chains, and “stored” injection scenarios;
• Standardized benchmarks for attack/defense co-evolution and cross-model transferability;
• Integration with network and system-level deception for expanded attack surface coverage.

Prompt-level defense frameworks now constitute a diverse, rapidly evolving arsenal enabling robust LLM safety, privacy, and operational integrity against the full spectrum of prompt-driven adversarial threats.