Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

Abstract: Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents. These capabilities make prompt-injection and jailbreak attacks more consequential, especially as attackers adopt model-guided automation to scale probing, prompt refinement, and response evaluation. This work analyzes the resulting attack-defense setting through a probabilistic model of a target system, its defense mechanism, and the attacker's automated judge. Our analysis shows that conventional detect-and-block defenses can allow attacker success rate (ASR) to approach one as the query budget grows, since predictable refusals provide useful feedback to automated search. We then examine detect-and-misdirect, where detected malicious interactions receive controlled, non-operational responses designed to induce false-positive errors in the attacker's judge. This strategy reduces the positive predictive value of attacker-selected candidates and yields a bounded asymptotic ASR. We evaluate a proof-of-concept realization of this strategy through Contextual Misdirection via Progressive Engagement (CMPE), a lightweight conversational misdirection method designed to replace predictable refusal text with safe but strategically misleading responses in automated jailbreak settings. On jailbreak benchmarks, CMPE reduces estimated ASR upper bounds by up to two orders of magnitude and nearly eliminates verified attack success in end-to-end PAIR and GPTFuzz attack runs.

• The paper introduces detect-and-misdirect, a defense that replaces static blocking with engineered misdirection to limit attacker success rates.
• It develops a probabilistic model and the CMPE technique, demonstrating up to two orders of magnitude reduction in attack success compared to detect-and-block methods.
• Empirical simulations show that this approach reduces harmful outputs while imposing higher verification costs on attackers, highlighting key trade-offs in agentic LLM safety.

Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

Introduction and Problem Formulation

The paper systematically analyzes the defensive limitations and opportunities in securing agentic AI systems—particularly those with LLM components that interact autonomously with external tools and data—against model-guided, automated jailbreak and prompt-injection attacks. The prevalence of iterative, automated red-teaming has exposed the inadequacy of detect-and-block defenses, as adversaries increasingly leverage model-based search and evaluation to bypass safety mechanisms at scale. The central insight formalized is that deterministic defenses provide attack-side feedback that enables near-certain attacker success rates (ASR) as query budgets increase, thereby undermining the foundational assumptions of static blocking defenses.

Probabilistic Analysis of Attack-Defense Dynamics

A probabilistic model captures the interaction between defender and attacker, explicitly distinguishing the original response generation, defense mechanism, and attacker-side automated judge. The analysis proves that detect-and-block strategies have a fundamental incompleteness: for any nontrivial defense false-negative rate and attacker judge false-negative rate, the attacker's maximal ASR converges to 1 as the prompt search space is exhaustively explored (N attempts, $N \to \infty$). This result is robust against adaptive attacker strategies and holds under the mild non-degeneracy condition standard in random-search theory.

Detector and judge error rates, including the defender's type-I ($\alpha_D$) and type-II ($\beta_D$) error rates and the attacker's judge false-negative ($\beta_A$) and misdirection-induced false-positive ($\Upsilon_A$) rates, are explicitly modeled. The ASR is upper-bounded in terms of these quantities, attacker verification budget K, and search heuristics, with closed-form asymptotics for i.i.d. prompt generation revealing the core feedback-exploitation vulnerability.

Detect-and-Misdirect: Theoretical Basis

The paper introduces and formalizes a complementary defense mechanism: detect-and-misdirect. Unlike block-and-refuse, which delivers negative search signals, detect-and-misdirect delivers non-operational but contextually plausible replies upon detection of suspected malicious prompts. These engineered responses strategically induce misdirection-induced false positives (MI-FPs) in the attacker’s judge, thereby corrupting automated candidate selection and sharply reducing the positive predictive value (PPV) of discovered candidates.

Analytically, the detect-and-misdirect upper bounds on ASR no longer scale to 1 with increasing attacker queries but remain strictly bounded below unity for any nonzero MI-FP rate. This is a critical theoretical contribution—detect-and-misdirect introduces a qualitative shift in defense regimes for agentic LLM systems by leveraging errors in the adversarial red-teaming evaluation loop itself.

CMPE: Practical Instantiation and Empirical Results

The paper instantiates misdirection via Contextual Misdirection via Progressive Engagement (CMPE), a response generation method combining positive preamble, context expansion through token-reshuffling and lexical injection, and conversational follow-up to obfuscate operational intent from attacker-side LLM-based judges.

Comprehensive empirical simulation utilizes jailbreak benchmarks (AdvBench), a broad set of automated classifier (e.g., Llama-Guard-3-8B, GPTFuzz-RoBERTa) and rubric-based LLM judgers (PAIR, StrongREJECT), and a lightweight refusal-suppressed model (NeuralDaredevil-8B-abliterated) for realistic and adversarially rigorous threat modeling.

Key findings include:

• Under detect-and-block with N=100, verified ASR for most attacker-defender pairs is $\sim 1$ or order $0.5$–$1$; detect-and-misdirect (with CMPE) suppresses average ASR by up to two orders of magnitude across all pairs (e.g., from 0.999 to 0.000).
• End-to-end evaluations against the PAIR and GPTFuzz frameworks demonstrate that CMPE reduces validated attack success from 20%/10% to 2%/0%, with the majority of attacker-claimed successes terminating on MI-FPs rather than true harmful outputs.
• Average attacker iterations to termination are decreased by up to $6\times$ under CMPE, reflecting attack loop disruption, not increased attack efficiency.

Additionally, ablation experiments demonstrate that simple refusal, obliterated misdirection, or non-conversational variants do not achieve comparable effectiveness or safety.

Implications and Trade-offs

Detect-and-misdirect represents a shift from output-based safety to feedback degradation, which is critical as model-guided attacks become hyper-automated and reliant on LLM-based evaluation. It causes attackers to incur significantly higher verification costs, often requiring increased human oversight or judge ensembling to reduce $\Upsilon_A$, which in turn sacrifices sensitivity to true positives—a direct robustness–recall trade-off.

While analogies to classical cyber deception (e.g., honeypots) are outlined, the agentic LLM context is unique due to the necessity of semantic evaluation and the multi-modal nature of potential exploitation. The approach is of limited use against programmatically verifiable targets (e.g., classic software bugs), but is specifically suited to automated red-teaming settings in agentic systems.

Limitations and Open Directions

The analytical modeling treats attack cycles in isolation and adopts non-adaptive defender strategies. Multi-cycle attacker adaptation, cross-prompt feedback incorporation, and effects in complex, tool-using, and multi-agent agentic environments remain to be studied. Collateral effects on benign human users are not directly examined, and this strategy is best suited to agentic or high-confidence automated threat contexts. Integration into mainstream model-training or AI-firewall pipelines presents technical and policy challenges, notably in balancing misdirection against refusal and handling unintended hallucination risks.

Conclusion

This work provides a rigorous, theoretically grounded, and empirically validated argument for the adoption of detect-and-misdirect as a complement to block/reject in agentic LLM system defense, showing that structured misdirection can fundamentally alter defender–attacker asymptotics by exploiting the judge error surface of model-guided attacks. The CMPE instantiation provides a practical technique, but further work is warranted in full agentic environments and in the development of adaptive, context-sensitive response shaping for robust AI security architectures.

---

For full technical details and proofs, refer to "Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems" (2606.20470).