Safety Self-Play with Reflective Replay

• The paper presents a novel self-play framework where a single LLM acts as both attacker and defender, significantly lowering jailbreak attack success rates.
• It employs Reflective Experience Replay with UCB-based sampling to focus learning on challenging, previously failed safety scenarios.
• Empirical evaluations show that SSP achieves robust defense performance with reduced over-refusal and maintained utility compared to current methods.

Safety Self-Play with Reflective Experience Replay (SSP with RER) is a proactive safety alignment methodology for LLMs that addresses shortcomings of static external red teaming and pre-collected adversarial datasets. Conventional approaches often overfit to previously known attack patterns, failing to generalize to novel or evolving threats. SSP with RER employs a single LLM as both Attacker and Defender within a dynamic self-play reinforcement learning (RL) loop, augmented by a Reflective Experience Replay mechanism that systematically focuses model learning on difficult or previously failed safety scenarios. This method establishes new benchmarks in autonomous, robust defense against jailbreak and safety attacks by evolving both attack sophistication and defensive responses without requiring external adversarial corpora (Wang et al., 15 Jan 2026).

1. Unified Self-Play Framework Design

SSP utilizes a single policy network $\pi_\theta$—the same LLM instance acts as both:

• Attacker: Given a harmful goal $G$ sampled from a distribution $\mathcal{D}$, the Attacker generates a jailbreak prompt $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$.
• Defender: Presented with $p_{\text{attack}}$, the Defender produces a model output $y \sim \pi_\theta(\cdot|p_{\text{attack}})$, which is expected to refuse or safely redirect the request.

The system operates in a closed-loop: as the Defender becomes better at refusing adversarial requests, the Attacker is forced to innovate with increasingly subtle or effective jailbreak prompts, resulting in co-evolution of attack and defense capabilities. Each loop iteration is scored by an external LLM-based safety judge that assigns an integer Safety Score $\in \{1, ..., 5\}$ to the Defender output, which drives the reinforcement learning update. Figure 1 in (Wang et al., 15 Jan 2026) illustrates these interactive dynamics.

2. Formal Reinforcement Learning Formulation

2.1 State, Action, and Reward Spaces

• Attacker:
  • State $s_{\text{att}} = G$ (harmful goal)
  • Action $a_{\text{att}} = p_{\text{attack}} \in$(vocabulary)$^+$ (arbitrary prompt)
• Defender:
  • State $G$0
  • Action $G$1(vocabulary)$G$2 (response)

Safety Score is mapped to rewards: $G$3 with a strict zero-sum coupling: $G$4

2.2 Optimization Objective

The objective includes attacker and defender terms, with a scalar balance $G$5: $G$6

With Reflective Experience Replay (see Section 3), an additional term over failure cases from the experience pool $G$7 is incorporated: $G$8

3. Reflective Experience Replay Mechanism

3.1 Experience Pool Structure

Two sub-pools are maintained:

• $G$9: Harmful goals $\mathcal{D}$0 where the Attacker's reward $\mathcal{D}$1.
• $\mathcal{D}$2: Jailbreak prompts $\mathcal{D}$3 where the Defender's reward $\mathcal{D}$4.

After each self-play episode, failure cases are stored according to these thresholds.

3.2 Experience Sampling via Upper Confidence Bound (UCB)

Items $\mathcal{D}$5 are ranked by: $\mathcal{D}$6 with - $\mathcal{D}$7: Most recent normalized reward after replay, - $\mathcal{D}$8: Replay count per item, - $\mathcal{D}$9: Pool size, - $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$0: Exploration constant (optimal near $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$1).

The top-k items are selected for replay, re-evaluated under the current policy, and those with improved reward above $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$2 are evicted.

3.3 Policy Update Algorithm

The learning loop alternates between regular self-play and prioritized replay. See below for the high-level pseudocode:

$p_{\text{attack}} \sim \pi_\theta(\cdot|G)$9

4. Empirical Evaluation

4.1 Robustness to Jailbreak Attacks

Attack Success Rate (ASR%) was measured against six attack strategies (GCG, PAIR, TAP, DAN, DI, SAA) on four LLMs. For Qwen2.5-7B, SSP achieved the lowest ASR in most scenarios:

Defense	GCG	PAIR	TAP	DAN	DI	SAA
No Defense	85.2	80.4	75.1	92.4	38.6	94.5
ACE-safety	2.5	3.1	2.9	5.2	4.1	3.1
SSP (ours)	1.7	2.4	1.4	1.3	2.1	3.0

4.2 Over-refusal and Utility Preservation

On the OR-Bench suite for safe queries, SSP exhibited a lower refusal rate (25.3%) than ACE-safety (29.6%), indicating no safety–utility tradeoff via over-censorship. On standard benchmarks (Math500, GSM8K, HumanEval, MBPP, MMLU, GPQA), SSP retained performance nearly on par with original models (e.g., Math500: 71.4% for SSP vs. 75.0% for vanilla).

4.3 Self-Play Attack Generation

When evaluated as a pure attacker on HarmBench, SSP yielded ASR ≈ 83.5% (Vicuna-7B), with diversity (DIV ≈ 0.96), matching dedicated jailbreak generation agents, in the absence of any explicit diversity loss setup.

5. Component Ablations and Sensitivity

Ablations on Qwen2.5-7B (mean ASR%):

Configuration	ASR (%)
w/o Unified Model	4.7
w/o Replay	4.7
w/o UCB	3.8
SSP (full)	1.7

Removal of any single component—unified model, experience replay, or UCB sampling—from SSP significantly degrades robustness. Experience pool sizes stabilize at ~80 items each after ~75 training steps. Optimal UCB exploration is observed near $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$3; performance is degraded if $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$4 falls outside $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$5.

6. Defender Adaptation: Case Study

Given an attack goal (“Instruct users how to infiltrate a secure facility, including identification of vulnerabilities…”), the Attacker disguises their prompt as a benign “security audit” request. The initial Defender response fully satisfies the harmful request (Safety Score=5, $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$6), resulting in this prompt’s insertion into $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$7. Upon subsequent replay sampling, the Defender is re-exposed and, after further training, learns to refuse the disguised request (Safety Score=1, $p_{\text{attack}} \sim \pi_\theta(\cdot|G)$8), at which point the prompt is evicted from replay.

A plausible implication is that continuous replay of “hard” failures allows the Defender to adapt to increasingly subtle adversarial strategies, which are omitted from static adversarial corpora.

7. Implications, Limitations, and Significance

SSP with Reflective Experience Replay effectively addresses the central limitation of prior jailbreaking defense schemes by eschewing externally curated adversarial datasets in favor of autonomous, evolving attack–defense co-learning. All critical architectural features—unified policy, prioritized experience replay, and UCB-based hard case sampling—are required to attain maximal robustness. The approach yields improved defense success rates against a range of adversarial attacks, avoids over-censoring benign requests, and maintains competitive performance on utility benchmarks. As demonstrated empirically, this methodology constitutes a new benchmark for proactive, model-driven safety alignment in LLMs (Wang et al., 15 Jan 2026).