Safety Alignment of LMs via Non-cooperative Games (2512.20806v1)

Abstract: Ensuring the safety of LMs while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models.

• The paper introduces AdvGame, a non-zero-sum game-theoretic framework that aligns LM safety via concurrent online RL and pairwise preference assessments.
• It demonstrates improved robustness to adaptive jailbreaks while preserving task utility across benchmarks like MMLU, TruthfulQA, and others.
• The approach produces a robust Defender and an effective Attacker for red-teaming, advancing practical safety evaluations in language models.

Safety Alignment of LLMs via Non-cooperative Games: AdvGame

Introduction and Problem Formulation

Ensuring the safety of LMs, particularly their resilience to adversarial jailbreaking and prompt injection attacks while maintaining high task utility, remains one of the most consequential challenges in contemporary AI alignment. Traditional sequential adversarial finetuning, entailing alternation between Attacker prompt generation and Defender retraining, is empirically limited by instability, inefficiency, and potential overfitting to the current attack distribution. The authors of "Safety Alignment of LMs via Non-cooperative Games" (2512.20806) introduce AdvGame—a non-zero-sum, non-cooperative game-theoretic paradigm—jointly optimizing Attacker and Defender LMs via preference-based online RL, eschewing parameter-sharing and alternating updates.

The core innovation is the explicit formulation of safety alignment as an adversarial game, wherein the Defender maximizes compliance and safety under ever more adaptive distributional shifts induced by a concurrently optimized Attacker, with supervision delivered through pairwise preference judgements rather than scalar rewards (which are known to be brittle and susceptible to reward hacking).

AdvGame Framework

The AdvGame system comprises two LMs: an Attacker $\rho$ and a Defender $\pi$. For a given seed $s$, the Attacker rephrases $s$ (faithfully preserving semantic intent) to $x$, aiming to elicit policy violations (on harmful seeds) or trigger over-refusals/non-compliance (on benign seeds) from the Defender. The Defender, in response to $x$, generates output $y$ aiming for safe deflection (for harmful prompts) or compliance (for benign prompts). Both LMs are trained with preference-based objectives: the Attacker is rewarded when $y$ increases the chance of a Defender failure (as judged pairwise on compliance, deflection, and faithfulness), and the Defender is rewarded when $y$ aligns with target safety and utility.

The optimization of both LMs is performed via concurrent online RL, employing DPO- and IPO-MD-style objectives adapted to the game-theoretic context. Crucially, no parameter sharing occurs, in contrast to self-play regimes (e.g., Self-RedTeam), mitigating entanglement between attack and defense capacities. KL regularization ensures outputs remain human-readable, focusing the adversarial pressure on distributional robustness to naturalistic prompts, rather than synthetic sequences.

Figure 1: The AdvGame framework: the Attacker LM generates prompt modifications from seeds, the Defender LM responds, and both are optimized via preference judges.

Preference-based Reward and Judge Design

A primary methodological contribution pertains to reward supervision. The authors eschew scalar reward models (as used in GRPO and related approaches), instead leveraging pairwise preference judges instantiated as LMs. Judges are assigned to each axis relevant to safety: compliance, deflection, and faithfulness. On benign queries, the judge prefers more compliant, helpful responses; on harmful queries, those with higher-quality deflection take precedence. Faithfulness judges penalize attack prompts that deviate from the seed's intent, mitigating reward-hacking pathways like topic drift.

This preference-based reward formalism renders optimization more robust, avoiding the collapse modes typical in pointwise scalar scoring schemes. The authors provide an explicit derivation connecting their loss to the DPO objective under the Bradley-Terry model assumption, and further extend to IPO-MD-style games, supporting Nash equilibrium computation via geometric mixtures or parameter EMA.

Empirical Results

Utility–Safety Pareto Analysis

Quantitative analysis demonstrates that AdvGame shifts the Pareto frontier—improving safety (lower attack success rate, ASR) while preserving core utility (task performance on MMLU, TruthfulQA, ARC, BBH, etc.), outperforming baseline models and alternative adversarial alignment approaches (including Self-RedTeam and uncensored variants).

Figure 2: Utility and safety of different safety post-training methods for Qwen2.5-7B-Instruct; AdvGame approaches enhance safety without significant loss in accuracy.

Across Qwen2.5-7B and Llama3.1-8B, AdvGame’s DPO-MD and IPO-MD variants consistently maintain or improve performance on utility benchmarks while reducing ASR across HarmBench, DAN, WildJailBreak, WildGuardTest, and XSTest. Notably, Self-RedTeam and similar self-play frameworks incur higher utility degradation and exhibit less robust safety gains.

Robustness to Adaptive Attacks

The authors evaluate robustness against strong adaptive jailbreakers (PAIR, TAP, GCG) under white-box assumptions. AdvGame-defended models exhibit markedly lower ASR in both black-box (human-like, distributional) and white-box (gradient-based) regimes relative to baselines, evidencing non-trivial gains in adversarial robustness.

Attacker as a Red-Teaming Tool

An ancillary result is the emergence of a highly effective Attacker LM, trained adversarially as part of AdvGame. This model can be deployed as a general-purpose red-team probe for arbitrary target LMs, rivaling hand-engineered methods (PAIR, TAP) in eliciting safety failures, making it an immediately valuable artifact for practitioners in safety evaluation pipelines.

Training Stability and Ablations

Extensive ablation studies dissect the impact of judge design (pairwise vs. pointwise), Attacker adaptation (static vs. learned), generator policy (on-policy vs. off-policy/EMA), and preference optimization axis (DPO, IPO-MD, GRPO). Results demonstrate:

• Pairwise judges and off-policy (EMA) sampling together confer stability and prevent reward hacking;
• Active Attacker training (rather than freezing) significantly enhances safety for the Defender;
• Preference-based objectives (AdvGame-DPO/IPO-MD) reliably outperform scalar-reward RL (GRPO) on both convergence stability and safety metrics.

Figure 3: Training dynamics: DPO-MD and IPO-MD exhibit stable, low-variance learning, while GRPO is unstable with reduced Defender reward.

Theoretical and Practical Implications

The formalization of safety alignment as a non-zero-sum, non-cooperative game fundamentally generalizes previous adversarial training paradigms. By eschewing the zero-sum, self-play/symmetric assumptions, AdvGame aligns the optimization landscape with the actual objectives in LM safety: attackers and defenders have misaligned but not strictly inverse goals. This is able to drive long-run co-evolution and more realistic empirical equilibria.

The demonstrated efficacy of pairwise preference-based optimization suggests that alignment and safety-critical reinforcement learning for LMs will increasingly utilize self-supervised feedback approaches, with game-theoretically grounded mechanisms for reward definition and model iteration. The fact that AdvGame produces not only a robust Defender but also a high-quality Attacker is highly advantageous for continuous safety auditing and will likely catalyze orthogonal advances in red-teaming methodologies.

Moreover, the modular approach to judge design explicitly isolates axes of failure (deflection/compliance/faithfulness), potentially supporting fine-grained monitoring and targeted interventions in deployment pipelines.

Limitations and Future Directions

The authors acknowledge that AdvGame, as instantiated, is tailored to adversarial attacks expressible in human-like, semantically faithful language, and does not address non-naturalistic attack modalities (e.g., synthetic or opaque adversarial sequences). Additionally, while GRPO is found less robust in this context, the potential for improvement with better reward models or alternative optimization is noted but not claimed. Further work is needed to integrate other modalities of attack and to refine reward/feedback mechanisms for improved generalization and stability.

Conclusion

"Safety Alignment of LMs via Non-cooperative Games" establishes a new operational regime for safety alignment: concurrent, non-zero-sum, preference-based RL between independently optimized Attacker and Defender LMs. This approach yields simultaneous improvements in safety and utility, significant robustness to adaptive attacks, and delivers a byproduct strong Attacker for ongoing safety testing. The framework and empirical demonstration delineate novel directions for future research in game-theoretic model alignment and robust, scalable LM deployment.