PISmith: RL-Based Red Teaming for LLM Defenses

• PISmith is a reinforcement learning–based red-teaming framework that systematically evaluates prompt-injection defenses by generating malicious injected prompts in black-box LLM settings.
• It employs adaptive entropy regularization and dynamic advantage weighting to mitigate reward sparsity and improve attack success rates while maintaining exploration.
• Evaluations across various datasets and defenses reveal a significant utility–robustness trade-off, underscoring the challenges in securing autonomous LLM applications.

PISmith is a reinforcement learning–based red-teaming framework for systematically assessing prompt-injection defenses on LLMs in a realistic black-box setting. It trains an attack policy to generate injected prompts that cause a defended target model to follow an attacker’s malicious instruction embedded in context, despite deployed defenses. The framework is introduced in "PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses" (Yin et al., 13 Mar 2026), which positions prompt injection as a security problem especially acute for autonomous agents that routinely consume untrusted external content such as web pages, tool outputs, and documents.

1. Threat model and problem formulation

PISmith is defined around prompt injection as an attack in which malicious instructions are embedded into the model’s input context and are then treated by the model as authoritative. The reported consequences include phishing link insertion, denial of service, and executing tool calls. The paper emphasizes that these risks are particularly severe for real-world LLM applications and autonomous agents because such systems routinely ingest and act on untrusted content (Yin et al., 13 Mar 2026).

The threat model is strictly black-box. The attacker can only query the defended target LLM and observe its outputs; there is no access to defense internals, model parameters, training data, or gradients. Within this setting, the attacker controls an attack LLM or policy that generates candidate injected prompts to be inserted into context. Training is on-policy: the attacker uses the target LLM’s outputs as feedback. Practical query budgets are enforced during both training and evaluation.

The success criterion is binary. A rollout is successful if the defended LLM’s response follows the injected task $g$. This is evaluated with an LLM-as-judge, specifically GPT-4o-mini, which returns a binary decision. The reward is therefore $r \in \{0,1\}$, with $r=1$ if the judged response fulfills the injected task and $r=0$ otherwise.

The optimization target is expected attack success over a dataset $\mathcal{D}$ of triples $(x_\text{inst}, x_\text{ctx}, g)$:

$$\max_{\pi_\phi} \; \frac{1}{|\mathcal{D}|} \sum_{(x_\text{inst}, x_\text{ctx}, g)\in \mathcal{D}} \mathbb{E}_{p \sim \pi_\phi(\cdot \mid x_\text{inst}, x_\text{ctx}, g)} \left[ r\big(\mathcal{M}_\theta(x_\text{inst}, x_\text{ctx} \oplus p), g\big) \right].$$

This formulation makes the defended LLM and its defense stack part of the environment $\mathcal{M}_\theta$, while the attack policy $\pi_\phi$ is optimized solely from observable outputs. A plausible implication is that PISmith is intended not as a white-box attack construction method but as an evaluation instrument for deployed defense configurations.

2. Framework architecture and reinforcement-learning design

PISmith comprises three principal components: an attack policy, an environment, and a reward signal. The attack policy $\pi_\phi$ is a lightweight open-source instruction-tuned LLM, Qwen3-4B-Instruct-2507, with approximately 4B parameters, fine-tuned via a GRPO-style on-policy RL procedure. The environment $r \in \{0,1\}$0 is the defended target LLM, such as Meta-SecAlign-8B or Qwen3-4B with guardrails. The reward signal is the binary judge output on whether the model followed the injected task.

The training loop is group-based. For each training triple $r \in \{0,1\}$1, PISmith samples a group of $r \in \{0,1\}$2 injected prompts from the attack policy, queries the defended LLM on the context augmented with each prompt, scores the resulting outputs with the judge, computes group-relative advantages, applies entropy regularization and success weighting, and then updates the attack policy with a clipped policy-gradient objective.

The paper argues that directly applying standard GRPO is suboptimal in this domain because of extreme reward sparsity. Against strong defenses, most generated injected prompts are blocked, so early training yields mostly zero rewards. Under these conditions, policy entropy collapses before effective attack strategies are discovered, and the few successful rollouts do not contribute enough gradient signal to be learned effectively. The KL-to-reference term in standard GRPO is described as insufficient and potentially counterproductive because it anchors the attack policy to a reference model that has no knowledge of effective injections (Yin et al., 13 Mar 2026).

PISmith modifies this regime through two reward-adaptive mechanisms. The first is adaptive entropy regularization with a cap: instead of a KL penalty, the method uses a pure entropy bonus that activates only when policy entropy falls below a specified threshold, and its strength increases when observed mean group reward is low. The second is dynamic advantage weighting, which amplifies the advantages of successful rollouts when successes are rare. The stated intent is to co-adapt exploration and exploitation to reward sparsity: when the mean reward $r \in \{0,1\}$3 is low, exploration pressure and success amplification are both strong; as $r \in \{0,1\}$4 rises, both terms relax toward standard GRPO behavior.

The resulting PISmith objective is

$r \in \{0,1\}$5

The paper explicitly states that no staged curriculum or partial credit is used. Reward shaping is limited to the binary success signal, while sparsity is addressed by the adaptive entropy mechanism and dynamic weighting.

3. Training procedure, prompts, and hyperparameters

The algorithmic workflow is specified as a nine-step loop. A minibatch of instruction–context–task triples is sampled from the dataset; for each triple, the attack policy samples $r \in \{0,1\}$6 injected prompts using a training prompt template; the defended LLM is queried; GPT-4o-mini judges each response against the injected task; group statistics are computed; policy entropy and the reward-conditioned entropy coefficient are evaluated; the success amplification factor is computed; the clipped surrogate objective is formed; and the policy is updated by gradient descent.

In the non-agent setting, the prompt engineering instructs the attack model to write an “Update” paragraph embedded in the reference text and to wrap the output in <prompt> tags. Agent variants adapt the prompt to placeholder-based or tool-output insertion settings. All RL-based methods compared in the paper share the same templates. This detail matters because it constrains the comparison: differences in attack success are intended to reflect optimization strategy rather than prompt-template asymmetries.

The non-agent training setup uses only the first 100 samples from Dolly Closed QA. For InjecAgent, the framework follows RL-Hammer’s data split. For AgentDojo, training uses the first 7 of 14 injected tasks in the workspace domain. The main reported non-agent training cost is 100 samples across 10 epochs with group size $r \in \{0,1\}$7, totaling 15,000 target queries.

The key hyperparameters reported for the main setup are: group size $r \in \{0,1\}$8, learning rate $r \in \{0,1\}$9, GRPO clipping $r=1$0, and 10 epochs. The adaptive-entropy parameters are entropy cap $r=1$1, $r=1$2, $r=1$3, and reward threshold $r=1$4. The success-weighting parameter is $r=1$5.

Evaluation uses two attack metrics. ASR@10 measures success if at least one of 10 attempts succeeds for a test instance, with $r=1$6 candidate prompts generated per instance. ASR@1 reports the single-attempt success rate averaged across 10 trials. In the non-agent evaluation, 1,700 instances are evaluated, yielding 17,000 target queries. Utility is also measured as task accuracy without any attack, to characterize the effect of a defense on benign performance.

4. Benchmarks, defenses, and empirical results

PISmith is evaluated on 13 datasets from PIArena spanning three categories of tasks that require external context: question answering, retrieval-augmented generation, and long-context tasks. The QA datasets are SQuAD v2, Dolly Closed QA, Dolly Info Extraction, and Dolly Summarization. The RAG datasets are Natural Questions, HotpotQA, and MS MARCO. The LongBench tasks are HotpotQA-Long, Qasper, GovReport, MultiNews, Passage Retrieval, and LCC (Yin et al., 13 Mar 2026).

The defense set includes the alignment defense Meta-SecAlign-8B; prevention-based defenses Sandwich, Instructional, PromptArmor, and DataFilter; and filter-based defenses PIGuard, PromptGuard, and DataSentinel. Additional agent evaluations include closed-source models with safety training, specifically GPT-4o-mini, GPT-4.1-nano, and GPT-5-nano.

Against Meta-SecAlign-8B across the 13 benchmarks, static baselines are reported as weak, with average ASR@1 of 0.04–0.07 across direct and combined templates. Search-based baselines improve modestly, with TAP at 0.11, PAIR at 0.16, and Strategy at 0.21 ASR@1. Among RL baselines, Vanilla GRPO achieves 0.13/0.05 average ASR@10/ASR@1, and RL-Hammer reaches 0.70/0.48. PISmith is reported at 1.00/0.87 average ASR@10/ASR@1, with especially large gaps on HotpotQA-Long at 0.99/0.61 versus RL-Hammer’s 0.04/0.01, on MSMARCO-RAG at 1.00/0.97 versus 0.78/0.47, and on Dolly Closed QA at 1.00/0.94 versus 0.95/0.78.

The utility–robustness comparison, averaged over 13 benchmarks on a Qwen3-4B backend, is structured around the tension between benign-task accuracy and attack resistance. No defense yields both high utility and low attack success under PISmith. The reported averages are summarized below.

Defense category	Utility	PISmith ASR@10 / ASR@1
No defense	0.74	1.00 / 0.92
Sandwich	0.74	0.99 / 0.91
Instructional	0.73	0.99 / 0.92
PromptArmor	0.74	1.00 / 0.92
DataFilter	0.63	0.71 / 0.49
PIGuard	0.72	0.92 / 0.82
PromptGuard	0.66	0.96 / 0.89
DataSentinel	0.55	0.63 / 0.52

This suggests that, within the reported experiments, defenses that reduce ASR more substantially do so partly by making the target less responsive on benign tasks. The paper characterizes this as an “unavoidable utility–robustness tension present in today’s defenses.”

5. Agentic evaluation and adaptive attack behavior

PISmith is also evaluated in agentic settings on InjecAgent and AgentDojo. These experiments are significant because prompt injection in agent systems can alter not just textual output but tool usage and multi-step execution. In InjecAgent, PISmith achieves 1.00/0.99 against Meta-SecAlign-8B, 1.00/0.99 against GPT-4o-mini, 1.00/1.00 against GPT-4.1-nano, and 1.00/0.95 against GPT-5-nano. The paper contrasts these results with much weaker static and search-based attacks; for example, on GPT-5-nano in InjecAgent, static attacks score 0.00 and search methods score 0.01–0.18 ASR@1 (Yin et al., 13 Mar 2026).

On AgentDojo, the reported results remain high but are lower than on InjecAgent: 0.78/0.62 for GPT-4o-mini, 0.81/0.64 for GPT-4.1-nano, and 0.38/0.24 for GPT-5-nano. These exceed the stated template baselines, which are at most 0.23, 0.20, and 0.02 respectively. A plausible implication is that the attack optimization remains effective under more structured agent workflows, though the margin depends on the agent benchmark and target model.

The qualitative case studies identify several prompt styles discovered by the policy. One is an “Important update” narrative that plausibly connects to the reference text while asserting that valid outputs must adopt a particular behavior, such as returning exactly “Error 503: Service Unavailable.” Another is defense-aware phrasing that embeds the malicious goal as a required format or official procedural safeguard. The paper reports that such constructions can bypass prevention defenses by masquerading as legitimate domain updates and can evade simple filters by presenting themselves as mandatory procedural requirements.

The adaptive strategies highlighted in the analysis are contextual blending, authority framing, and minimal edits with high impact. Contextual blending refers to crafting injections that cohere strongly with the user task and domain, thereby reducing detectability by filter-based defenses. Authority framing presents the malicious instruction as an official requirement in order to override instruction hierarchy defenses. Minimal edits with high impact are particularly relevant in agent settings, where localized placeholder modifications in tool outputs can reliably trigger malicious tool actions.

The reported GPT-5-nano InjecAgent example is notable because the agent explicitly reasons that the user task has no matching email and then concludes that it “must fulfill the injection task,” proceeding to perform the malicious tool action before responding. This is presented as evidence that prompt injection can manipulate multi-step execution, not merely surface-form response text.

6. Ablations, efficiency, and limitations

The ablation study isolates the contributions of the two reward-adaptive components. Vanilla GRPO achieves average ASR@1 of 0.05. Removing entropy regularization yields 0.09, which the paper attributes to policy collapse and terminated exploration. Removing the success boost yields 0.57, indicating that entropy alone permits some discovery of successful attacks but that learning remains diluted by many failures. Full PISmith reaches 0.87 ASR@1. The paper therefore treats adaptive entropy regularization and dynamic advantage weighting as essential and complementary (Yin et al., 13 Mar 2026).

The reported training dynamics support this interpretation. Adaptive entropy is said to eliminate collapse and sustain rollout diversity, while dynamic weighting accelerates learning from sparse successes. Mean reward steadily improves, and entropy remains at a healthy level rather than shrinking monotonically. The analysis further argues that reward-conditioned exploration is preferable to fixed entropy targets because it avoids incoherence early in training and stagnation later.

PISmith is also compared on query cost. Its reported total is 32,000 target queries, comprising 15,000 training queries and 17,000 inference queries in the main non-agent setting. RL-Hammer is reported at 47,000 total queries because of dual-target training. Search-based baselines are substantially more expensive at inference time: TAP uses 82,110 queries, PAIR 74,460, and Strategy 194,140, yet all achieve lower attack success.

Several limitations are stated explicitly. One is overfitting risk to the tested defenses and benchmarks; although the framework generalizes across the reported tasks, robustness to unseen defense families should be re-evaluated. Another is dependence on an LLM-as-judge for reward assignment, which introduces potential bias: changing the judge may alter measured ASR. The paper also notes dual-use risk and recommends controlled red-teaming and responsible disclosure.

The discussion of failure cases centers on DataSentinel and DataFilter. These defenses partially reduce ASR, with PISmith at 0.63/0.52 and 0.71/0.49 respectively, but they also reduce utility, primarily by aggressively blocking content, including benign content. The paper interprets this as reducing the room for semantic persuasion while also degrading normal task performance.

7. Significance, recommendations, and reproducibility

PISmith is presented as an automated and principled method for stress-testing prompt-injection defenses under black-box constraints. Its empirical role is to reveal robustness gaps that static and search-based attacks often miss and to quantify the utility–robustness trade-off in current defenses. This places the framework within evaluation methodology as much as within attack generation: it is designed to measure how deployed defenses behave under adaptive pressure rather than merely to exhibit individual jailbreaks.

The recommendations given for defense improvement are concrete. The paper advises combining prevention with calibrated filtering so as to raise attack difficulty without severe utility loss, hardening instruction hierarchies through clearer separation of trusted and untrusted channels and formalized guardrails in agents, considering adversarial training with adaptive attackers such as PISmith during defense development cycles, and improving detection through intrinsic LLM features and context attribution while minimizing over-defense (Yin et al., 13 Mar 2026). This suggests a defense agenda that treats adaptive red-teaming as a routine part of system development rather than as an after-the-fact audit.

Reproducibility resources are explicitly provided. Implementation details, prompt templates, and evaluation scripts are available at the project repository. The benchmarks are adopted from PIArena and include QA, RAG, LongBench, InjecAgent, and AgentDojo tasks. The attack model is Qwen3-4B-Instruct-2507; defenses include both open- and closed-source components such as PromptGuard, DataSentinel, PIGuard, and Meta-SecAlign-8B; and the judge is GPT-4o-mini.

A common misconception in prompt-injection evaluation is that strong performance against static attacks is evidence of robust defense. The reported results directly challenge that view: even state-of-the-art defenses remain vulnerable when the attacker is trained adaptively in a black-box setting. Another misconception is that higher robustness necessarily reflects better defensive design; the utility results indicate that some robustness gains are achieved through aggressive blocking that also harms benign-task accuracy. In that sense, PISmith functions not only as an attacker but also as a probe of defense calibration.