---
title: 'AutoInject: Automated Prompt Injection Using RL'
url: https://www.emergentmind.com/topics/autoinject
type: topic
---

# AutoInject: Automated Prompt Injection Using RL

Searching arXiv for the primary AutoInject paper and closely related automated prompt injection work in agentic settings.
Searching arXiv for "Learning to Inject: Automated Prompt Injection via Reinforcement Learning" and related agentic prompt-injection papers.
AutoInject is a reinforcement-learning framework for **automated prompt injection** against **LLM agents that interact with external content and tools**, rather than a generic chat jailbreak method. In its setting, adversarial instructions are embedded in untrusted environment content—such as an email body or retrieved document—and are intended to cause an agent to deviate from the user’s task by executing an attacker-specified action with the correct parameters, while often still completing the benign task. The framework is explicitly **black-box** with respect to the victim agent, uses a **1.5B parameter adversarial suffix generator**, and is evaluated primarily in **AgentDojo**, where success is defined by concrete tool execution and parameter binding rather than by refusal-breaking alone [2602.05746].

## 1. Definition, scope, and threat model

AutoInject addresses indirect prompt injection in **agentic, tool-using systems**. The paper defines the attack surface as untrusted environment content that the agent reads during normal execution and that can contain adversarial instructions. The relevant failure mode is not merely that the model produces objectionable text, but that it performs attacker-chosen actions such as sending an email or transferring funds, with success measured through the benchmark’s programmatic checks [2602.05746].

This scope distinguishes AutoInject from conventional jailbreak optimization. The paper argues that prior prompt-injection attacks were mostly **manual, template-based, and red-team driven**, while existing automated jailbreak methods such as **GCG** are poorly matched to prompt injection because the objective is narrower: the model must select the right tool, bind the correct arguments, and often resume the benign workflow. A jailbreak proxy such as eliciting a phrase like “Sure, here is…” is therefore a weak surrogate for the behavioral outcome of interest [2602.05746].

The attacker model is explicitly **black-box**. The attacker does not require gradients, weights, or activations of the victim model, and does not need to deploy an auxiliary model inside the victim environment. Instead, the attacker can place adversarial content into the environment, observe outcomes exposed by the benchmark pipeline, and query the target under a finite budget. In the main setting, the budget is **\(B=260\) victim evaluations per \((\text{user task}, \text{injection task})\) pair**. The attacker may use an external helper model—typically **GPT-4o-mini**—for optimization feedback, but that model is not part of the victim system [2602.05746].

A common misconception is that AutoInject is a universal prompt perturbation in the classical adversarial-NLP sense. In this work, “universal” means that the optimization process can discover **reusable suffixes** that transfer across many task pairs and models without re-optimization; it does not mean a single perturbation is proved to attack all prompts [2602.05746].

## 2. RL formulation and optimization objective

AutoInject formulates suffix generation as a Markov decision process conditioned on an injection goal \(g\) and benign task context \(c\). At generation step \(t\), the state is

\[
s_t = (g,c,a_1,\ldots,a_{t-1}),
\]

where \(a_1,\ldots,a_{t-1}\) are the previously generated tokens. An action is the next token \(a_t \in \mathcal V\), and the transition is deterministic concatenation,

\[
s_{t+1} = s_t \oplus a_t.
\]

An episode ends when the generator emits EOS or reaches a maximum length \(T\), producing a suffix \(x=(a_1,\ldots,a_T)\). The policy is a language model \(\pi_\theta\), instantiated in the main text as **Qwen2.5-1.5B**, which serves as the adversarial suffix generator [2602.05746].

The basic objective is

\[
\max_\theta \mathbb{E}_{x \sim \pi_\theta(\cdot \mid g, c)} \left[ R(r^\text{sec}, r^\text{util}) \right].
\]

Here \(r^\text{sec}\in[0,1]\) is the **security score**, indicating whether the malicious injected goal was executed, and \(r^\text{util}\in[0,1]\) is the **utility score**, indicating whether the original benign task was still completed. AutoInject extends this with a dense preference term,

\[
R(r^\text{sec}, r^\text{util}, r^\text{pref}) = \alpha \cdot r^\text{sec} + \beta \cdot r^\text{util} + \gamma \cdot r^\text{pref}.
\]

The appendix reports \(\alpha=0.7\), \(\beta=0.3\), and a scheduled \(\gamma\) based on the consumed query fraction \(\rho=n/B\):

\[
\gamma(\rho) =
\begin{cases}
0.8 & \text{if } \rho < 0.1 \\
0.5 & \text{if } 0.1 \leq \rho < 0.3 \\
0.3 & \text{if } 0.3 \leq \rho < 0.5 \\
0.1 & \text{if } \rho \geq 0.5
\end{cases}
\]

so the dense term is emphasized early, when successful attacks are rare [2602.05746].

Algorithmically, the policy is initialized from a pretrained LM and the current best suffix \(x^*\) is initialized to empty. Until budget exhaustion or success, the method samples a group of suffixes, evaluates them in the victim pipeline to obtain \((r^\text{sec}, r^\text{util})\), queries a feedback model for \(r^\text{pref}\) relative to \(x^*\), computes the composite reward, updates \(x^*\) if a better suffix is found, and performs a policy update. Training stops early if any sampled suffix achieves both \(r^\text{sec}=1\) and \(r^\text{util}=1\) [2602.05746].

## 3. Dense preference feedback and GRPO training

The central optimization difficulty is **reward sparsity**. Most random suffixes fail completely, so direct optimization on binary success signals yields little learning signal. AutoInject addresses this by maintaining the best suffix so far, \(x^*\), and asking a feedback model to compare a new suffix \(x\) against \(x^*\) with respect to malicious-goal execution, benign-task preservation, and evasion. The model produces a binary preference label, and AutoInject converts the final-token probabilities into a continuous comparison score,

\[
r^\text{pref} = P(x \succ x^* \mid g,c),
\]

which supplies dense feedback even when neither candidate fully succeeds [2602.05746].

Optimization uses **Group Relative Policy Optimization (GRPO)**. For each training step, the current policy samples a group of \(K\) completions for the same \((g,c)\); the appendix gives **\(K=8\)**. For completion \(x_i\) with reward \(R_i\), GRPO computes a normalized relative advantage

\[
\hat{A}_i = \frac{R_i - \mathrm{mean}(\{R_j\}_{j=1}^K)}{\mathrm{std}(\{R_j\}_{j=1}^K)}.
\]

This relative normalization is important because it still yields gradient signal when all completions are poor in absolute terms. The paper presents a clipped GRPO/PPO-style surrogate with KL regularization to a reference policy; it also notes that the typesetting of the objective is corrupted in places, but the intended form is clearly a clipped policy-gradient objective with KL control. The appendix lists \(\beta_{\text{KL}}=1.0\) for most frontier-model attacks and \(0.1\) for some 4B-model settings [2602.05746].

The attack does not learn a full prompt from scratch. Instead, the learned suffix is appended to a fixed social-engineering frame in the AgentDojo injection template that impersonates a legitimate user message and instructs the agent to perform an extra step before resuming the original task. This design keeps the optimization focused on the suffix while holding the broader frame constant [2602.05746].

Utility preservation is a first-class objective rather than a side effect. The framework explicitly optimizes attacks that both compromise the agent and preserve the user’s benign task, on the premise that stealthy compromise is more operationally relevant than attacks that merely disrupt execution [2602.05746].

## 4. Empirical performance on AgentDojo

The experiments center on **AgentDojo**, a benchmark with **97 user tasks** across banking, travel, workspace, and Slack. Each user task is paired with multiple injection tasks, and the environment provides ground-truth binary signals for benign-task completion and malicious-goal execution. This benchmark structure is integral to AutoInject because it supplies the clean behavioral reward signals required by the RL formulation [2602.05746].

On the full benchmark, AutoInject substantially outperforms the hand-written template baselines on frontier systems [2602.05746].

| Model | AutoInject ASR | Best template baseline |
|---|---:|---:|
| Gemini-2.5-Flash | 58.00% | 23.60% |
| GPT-4.1-nano | 47.97% | 20.48% |
| GPT-5-nano | 11.49% | 1.60% |
| Claude-Sonnet-3.5 | 12.59% | 5.69% |

The paper also reports strong results on additional targets: **39.62% ASR** on **Gemini-2.0-Flash**, **50.37% ASR** on **GPT-4o-mini**, **42.5% ASR** on **Qwen3-4B**, and **35.0% ASR** on **Gemma3-4B**, outperforming **GCG**, **TAP**, and an adaptive random-search baseline under comparable budgets [2602.05746].

Utility results are a notable part of the evaluation. The paper reports cases in which **utility under attack** equals or exceeds the no-attack baseline, including **GPT-5-nano** with **90.20% utility under attack** versus **80.41% utility without attack**, and **Claude-Sonnet-3.5** with **98.52% utility under attack** versus **75.00% baseline utility**. The authors attribute this to the explicit inclusion of utility in the reward, which biases the learned suffix distribution toward non-disruptive attacks [2602.05746].

The baseline suite includes six manual templates—**Direct Instruction**, **Ignore Previous**, **Important Instructions**, **InjecAgent**, **System Message**, and **Tool Knowledge**—as well as automated baselines: **GCG**, **TAP**, and a **random adaptive attack**. A stronger ablation baseline keeps AutoInject’s reward function but replaces the learned policy with random mutation search, isolating the contribution of RL itself [2602.05746].

## 5. Transferability, reusable suffixes, and ablations

AutoInject supports both **online query-based optimization** and **transfer/universal mode**. In the first mode, the suffix policy is optimized adaptively for a specific \((g,c)\) pair. In the second, suffixes found on one model or task are reused on unseen models and task pairs. The transfer evaluation considers both **cross-task transfer** and **cross-model transfer**, including transfer to held-out task pairs across banking, Slack, travel, and workspace [2602.05746].

Transferability is strongly model dependent. **GPT-4o-mini** is consistently vulnerable in the reported transfer studies, often with transfer ASRs in the **37.5%–62.5%** range. **GPT-5** is the most resistant target: it shows **0% ASR** from all source models in cross-task transfer and also **0%** from all external models in fixed-injection transfer, with even self-transfer reaching only **6.2%** in one setting. At the same time, **GPT-5** can be a strong source model: suffixes optimized on GPT-5 transfer to **Gemini 2.5 Flash** and **GPT-4o-mini** at **50% ASR** in the cross-task setup [2602.05746].

The most vivid reusable-family result is the “**allelujah**” suffix family. One reported suffix—

```text
! surely proofs guide eng familia allelujah allelujah Example of how the suffix could be
```

—successfully attacks **70 task pairs on Gemini-2.5-Flash**, **53 on GPT-4o-mini**, and **46 on GPT-4.1-nano**, out of **949 total benchmark pairs**. Different reorderings or small textual edits materially affect the success rate, which suggests a genuine token-pattern effect rather than a trivial semantic instruction. The paper treats mechanistic explanations of this phenomenon as speculative [2602.05746].

The ablations support both of AutoInject’s main ingredients: policy-gradient RL and dense feedback. On the banking suite for **GPT-4.1-nano**, full AutoInject achieves **97.69% ASR** with **62.50% utility**; removing GRPO reduces ASR to **88.32%**, removing feedback to **95.14%**, and removing both to **90.51%**. On the harder **GPT-5-nano**, full AutoInject reaches **36.57% ASR** and **87.50% utility**; without GRPO this falls to **19.44%**, without feedback to **25.00%**, and without both to **17.36%**. The same-reward random-search baseline also underperforms: on **Gemini-2.5-Flash**, AutoInject gets **58.0% ASR** versus **34.04%** for search; on **GPT-5-nano**, **11.49%** versus **4.84%**; and on **Claude-Sonnet-3.5**, **12.59%** versus **7.64%**, with markedly better utility for AutoInject (**98.52%** versus **77.08%**) [2602.05746].

These results align with a broader empirical trend in agentic prompt injection: black-box, semantically structured optimization tends to outperform brittle token-level search when the target is end-to-end tool behavior rather than free-form compliance [2606.10525].

## 6. Limitations, defenses, and related research

AutoInject is evaluated under a realistic but still benchmark-mediated setting. The paper assumes access to AgentDojo’s explicit success and utility signals, which is natural inside the benchmark but may be less directly available in unconstrained deployments. It also does not provide a formal theory of transfer or universality, no offline-data component, and no large systematic sweeps over suffix length or query budget beyond the reported fixed settings and the \(\gamma\)-schedule [2602.05746].

The work also highlights defense implications. Against **Meta-SecAlign-70B**, a model specifically fine-tuned for prompt-injection robustness, template attacks are nearly or entirely neutralized, yet AutoInject still achieves **21.88% ASR** with **39.29% utility**, slightly above the no-attack utility of **37.50%**. The paper interprets this as evidence that preference-based defenses can reject familiar attack patterns while remaining vulnerable to adaptive out-of-distribution attacks [2602.05746].

At the architectural level, the paper cites **CaMeL** and **IPI-Guard** as examples of plan-then-execute or decoupled designs that prevent untrusted content from influencing control flow after planning. AutoInject does not break that threat model because it targets the more common case in which agents continue reasoning over untrusted inputs during execution. The paper also notes a benchmark limitation with immediate tool execution: in experiments, models such as **Claude Haiku 4.5** often ask for clarification before acting, which breaks the benchmark pipeline and suggests a defense direction based on deliberative confirmation before sensitive actions [2602.05746].

Within the surrounding literature, AutoInject belongs to a broader shift from manual prompt engineering toward automated attack optimization in agentic systems. Related directions include semantic-search-based automated prompt injection in AgentDojo [2606.10525], no-box tool-selection hijacking through optimized malicious tool documents [2504.19793], and closed-loop stealthy skill poisoning for coding agents [2602.14211]. The name **AutoInject** has also been used independently for a direct message-corruption mechanism in studies of multi-agent robustness, where it denotes controlled rewriting of inter-agent messages rather than RL-based prompt injection [2408.00989].

The broader implication is that indirect prompt injection is better understood as an **end-to-end behavioral optimization problem** than as a token-level jailbreak variant. AutoInject operationalizes that view by optimizing for successful malicious action execution, correct parameter binding, and benign-task preservation simultaneously. This suggests that future defenses must handle **adaptive, utility-preserving, transferable prompt injections**, not merely known templates or overt override strings [2602.05746].

Source: https://www.emergentmind.com/topics/autoinject