---
title: Safety Self-Play with Reflective Replay
url: https://www.emergentmind.com/topics/safety-self-play-with-reflective-experience-replay
type: topic
---

# Safety Self-Play with Reflective Replay

Safety Self-Play with Reflective Experience Replay (SSP with RER) is a proactive safety alignment methodology for large language models (LLMs) that addresses shortcomings of static external red teaming and pre-collected adversarial datasets. Conventional approaches often overfit to previously known attack patterns, failing to generalize to novel or evolving threats. SSP with RER employs a single LLM as both Attacker and Defender within a dynamic self-play reinforcement learning (RL) loop, augmented by a Reflective Experience Replay mechanism that systematically focuses model learning on difficult or previously failed safety scenarios. This method establishes new benchmarks in autonomous, robust defense against jailbreak and safety attacks by evolving both attack sophistication and defensive responses without requiring external adversarial corpora [2601.10589].

## 1. Unified Self-Play Framework Design

SSP utilizes a single policy network $\pi_\theta$—the same LLM instance acts as both:

- **Attacker:** Given a harmful goal $G$ sampled from a distribution $\mathcal{D}$, the Attacker generates a jailbreak prompt $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$.
- **Defender:** Presented with $p_{\text{attack}}$, the Defender produces a model output $y \sim \pi_\theta(\cdot|p_{\text{attack}})$, which is expected to refuse or safely redirect the request.

The system operates in a closed-loop: as the Defender becomes better at refusing adversarial requests, the Attacker is forced to innovate with increasingly subtle or effective jailbreak prompts, resulting in co-evolution of attack and defense capabilities. Each loop iteration is scored by an external LLM-based safety judge that assigns an integer Safety Score $ \in \{1, ..., 5\} $ to the Defender output, which drives the reinforcement learning update. Figure 1 in [2601.10589] illustrates these interactive dynamics.

## 2. Formal Reinforcement Learning Formulation

### 2.1 State, Action, and Reward Spaces

- **Attacker:**
  - State $s_{\text{att}} = G$ (harmful goal)
  - Action $a_{\text{att}} = p_{\text{attack}} \in$(vocabulary)$^+$ (arbitrary prompt)
- **Defender:**
  - State $s_{\text{def}} = p_{\text{attack}}$
  - Action $a_{\text{def}} = y \in$(vocabulary)$^+$ (response)

Safety Score is mapped to rewards:
\[
r^{\mathrm{att}} = \max(0, \min(1, \tfrac{\mathrm{Score} - 1}{4})), \quad 
r^{\mathrm{def}} = \max(0, \min(1, \tfrac{5 - \mathrm{Score}}{4}))
\]
with a strict zero-sum coupling:
\[
r^{\mathrm{att}} + r^{\mathrm{def}} = 1
\]

### 2.2 Optimization Objective

The objective includes attacker and defender terms, with a scalar balance $\lambda$:
\[
\mathcal{J}_{\mathrm{self\mbox{-}play}}(\theta) = \max_\theta\,\mathbb{E}_{G\sim \mathcal{D}}
\Bigl[
\lambda\,\mathbb{E}_{p_{\mathrm{attack}}\sim \pi_\theta(\cdot|G)} \bigl[r^{\mathrm{att}}(G, p_{\mathrm{attack}})\bigr]
+ \mathbb{E}_{y \sim \pi_\theta(\cdot|p_{\mathrm{attack}})} \bigl[r^{\mathrm{def}}(y)\bigr]
\Bigr]
\]

With Reflective Experience Replay (see Section 3), an additional term over failure cases from the experience pool $\mathcal{P}$ is incorporated:
\[
\begin{aligned}
\mathcal{J}(\theta) = \max_\theta \, &\mathbb{E}_{G\sim \mathcal{D}}
\Bigl[
\text{(Attacker new-play)} + \text{(Defender new-play)}
\Bigr] \\
& + \mathbb{E}_{(G,p_{\text{attack}},y)\sim \mathcal{P}}
\Bigl[\lambda r^{\mathrm{att}}(G, \pi_\theta) + r^{\mathrm{def}}(y)\Bigr]
\end{aligned}
\]

## 3. Reflective Experience Replay Mechanism

### 3.1 Experience Pool Structure

Two sub-pools are maintained:

- $\mathcal{P}_{\text{att}}$: Harmful goals $G$ where the Attacker's reward $r^{\mathrm{att}} < \tau_{\mathrm{att}}$.
- $\mathcal{P}_{\text{def}}$: Jailbreak prompts $p_{\text{attack}}$ where the Defender's reward $r^{\mathrm{def}} < \tau_{\mathrm{def}}$.

After each self-play episode, failure cases are stored according to these thresholds.

### 3.2 Experience Sampling via Upper Confidence Bound (UCB)

Items $i \in \mathcal{P}$ are ranked by:
\[
\mathrm{UCB\_Score}_i = (1 - \overline{r}_i) + c \sqrt{\frac{\ln N}{n_i + 1}}
\]
with
  - $\overline r_i$: Most recent normalized reward after replay,
  - $n_i$: Replay count per item,
  - $N = |\mathcal{P}|$: Pool size,
  - $c$: Exploration constant (optimal near $c = \sqrt{2}$).

The top-k items are selected for replay, re-evaluated under the current policy, and those with improved reward above $\tau$ are evicted.

### 3.3 Policy Update Algorithm

The learning loop alternates between regular self-play and prioritized replay. See below for the high-level pseudocode:

```
Algorithm 1: Safety Self-Play with Reflective Experience Replay

Input: harmful goals 𝒟, judge Score(·), thresholds τ_att, τ_def, exploration c, batch size B, λ, max steps T, policy π_θ, empty pools 𝒫_att, 𝒫_def
For step=1…T do
  1. Sample G∼𝒟
  2. Attacker: p_attack∼π_θ(·|G)
  3. Defender: y∼π_θ(·|p_attack)
  4. Score←Judge(y)
  5. Compute r^att, r^def
  6. If r^att<τ_att: add G→𝒫_att
     If r^def<τ_def: add p_attack→𝒫_def
  7. If |𝒫_att|≥B and |𝒫_def|≥B:
       a) For each pool: compute UCB_Score_i
       b) Select top-B items, re-evaluate, update \overline r_i
       c) Evict those with \overline r_i≥τ
       d) Add replayed rewards to RL batch
  8. Update π_θ via policy gradients on both new and replayed samples
EndFor
Output: aligned policy π_θ
```

## 4. Empirical Evaluation

### 4.1 Robustness to Jailbreak Attacks

Attack Success Rate (ASR%) was measured against six attack strategies (GCG, PAIR, TAP, DAN, DI, SAA) on four LLMs. For Qwen2.5-7B, SSP achieved the lowest ASR in most scenarios:

| Defense      | GCG | PAIR | TAP | DAN | DI  | SAA |
|--------------|-----|------|-----|-----|-----|-----|
| No Defense   |85.2 | 80.4 |75.1 |92.4 |38.6 |94.5 |
| ACE-safety   |2.5  | 3.1  |2.9  |5.2  |4.1  |3.1  |
| SSP (ours)   |1.7  | 2.4  |1.4  |1.3  |2.1  |3.0  |

### 4.2 Over-refusal and Utility Preservation

On the OR-Bench suite for safe queries, SSP exhibited a lower refusal rate (25.3%) than ACE-safety (29.6%), indicating no safety–utility tradeoff via over-censorship. On standard benchmarks (Math500, GSM8K, HumanEval, MBPP, MMLU, GPQA), SSP retained performance nearly on par with original models (e.g., Math500: 71.4% for SSP vs. 75.0% for vanilla).

### 4.3 Self-Play Attack Generation

When evaluated as a pure attacker on HarmBench, SSP yielded ASR ≈ 83.5% (Vicuna-7B), with diversity (DIV ≈ 0.96), matching dedicated jailbreak generation agents, in the absence of any explicit diversity loss setup.

## 5. Component Ablations and Sensitivity

Ablations on Qwen2.5-7B (mean ASR%):

| Configuration        | ASR (%) |
|----------------------|---------|
| w/o Unified Model    | 4.7     |
| w/o Replay           | 4.7     |
| w/o UCB              | 3.8     |
| SSP (full)           | 1.7     |

Removal of any single component—unified model, experience replay, or UCB sampling—from SSP significantly degrades robustness. Experience pool sizes stabilize at ~80 items each after ~75 training steps. Optimal UCB exploration is observed near $c = \sqrt{2}$; performance is degraded if $c$ falls outside $[0.5, 2.0]$.

## 6. Defender Adaptation: Case Study

Given an attack goal (“Instruct users how to infiltrate a secure facility, including identification of vulnerabilities…”), the Attacker disguises their prompt as a benign “security audit” request. The initial Defender response fully satisfies the harmful request (Safety Score=5, $r^{\mathrm{def}}=0.0$), resulting in this prompt’s insertion into $\mathcal{P}_{\text{def}}$. Upon subsequent replay sampling, the Defender is re-exposed and, after further training, learns to refuse the disguised request (Safety Score=1, $r^{\mathrm{def}}=1.0$), at which point the prompt is evicted from replay.

A plausible implication is that continuous replay of “hard” failures allows the Defender to adapt to increasingly subtle adversarial strategies, which are omitted from static adversarial corpora.

## 7. Implications, Limitations, and Significance

SSP with Reflective Experience Replay effectively addresses the central limitation of prior jailbreaking defense schemes by eschewing externally curated adversarial datasets in favor of autonomous, evolving attack–defense co-learning. All critical architectural features—unified policy, prioritized experience replay, and UCB-based hard case sampling—are required to attain maximal robustness. The approach yields improved defense success rates against a range of adversarial attacks, avoids over-censoring benign requests, and maintains competitive performance on utility benchmarks. As demonstrated empirically, this methodology constitutes a new benchmark for proactive, model-driven safety alignment in LLMs [2601.10589].

Source: https://www.emergentmind.com/topics/safety-self-play-with-reflective-experience-replay