Self-Play Security Testing

• Self-play security testing is an automated regime where attacker and defender strategies are refined through repeated adversarial interactions with simulated or surrogate environments.
• It spans applications from web penetration testing and network security games to LLM red teaming and code vulnerability analysis, offering diverse use cases.
• Research leverages reinforcement learning, game-theoretic models, and modular architectures to iteratively improve exploit detection and threat mitigation.

Searching arXiv for papers on self-play security testing, automated pentesting, and LLM red teaming. Self-play security testing denotes a family of automated security evaluation regimes in which attacker-like and defender-like strategies are refined through repeated adversarial interaction with a target system, a simulator, or a learned surrogate. In the current literature, the term is used both for explicit two-player zero-sum games—such as jailbreak generation versus refusal in LLMs—and for adjacent regimes in which a single autonomous attacker repeatedly probes procedurally generated vulnerable environments in a manner that follows the self-play philosophy (Wang et al., 15 Jan 2026, Malfa et al., 8 May 2026, López-Montero et al., 30 Jun 2025). Across web penetration testing, network security games, code security, and LLM safety alignment, the unifying idea is iterative adversarial improvement under structured state, action, and reward or utility abstractions, combined with evaluation procedures that distinguish superficial findings from validated exploits.

1. Historical emergence and conceptual scope

An early explicit formulation appeared in intrusion prevention, where attacker and defender were modeled as a Markov game and their strategies were allowed to evolve through reinforcement learning and self-play without human intervention; the resulting policies were reported to reflect common-sense knowledge and to resemble human strategies (Hammar et al., 2020). Subsequent work on large-scale network security games extended this direction with Neural Fictitious Self-Play, reformulating best-response and average-policy learning so that large extensive-form attacker–defender games on graphs could be handled with state–action value functions, metric-based classifiers over legal actions, high-level actions, and graph node embeddings (Xue et al., 2021). A complementary line focused on incomplete-information dynamic games, where Temporal-Induced Self-Play used belief-space representation, backward induction, policy learning, and non-parametric approximation to obtain approximate Perfect Bayesian Equilibrium in zero-sum one-sided stochastic Bayesian games with finite horizon (Chen et al., 2021).

More recent work broadened the scope from abstract security games to operational security testing. MEGA-PT introduced a meta-game penetration testing framework with Micro Tactic Games at each node and a Macro Strategic Process over the network, yielding a hierarchical attacker–defender formulation that is directly interpretable as a substrate for self-play security testing (Ge et al., 2024). Reinforcement-learning-based automated web penetration testing then adopted a single autonomous attacker trained on procedurally generated vulnerable web environments, explicitly noting that the paper did not use the term “self-play” but did instantiate its core pattern: an autonomous attacker-like agent repeatedly interacting with a simulated vulnerable system and improving via trial and error (López-Montero et al., 30 Jun 2025). In parallel, self-play became explicit in LLM safety, where Safety Self-Play cast a single LLM as both attacker and defender in a unified RL loop (Wang et al., 15 Jan 2026), and later work analyzed the theoretical and architectural consequences of parameter sharing in such systems (Malfa et al., 8 May 2026).

Regime	Representative papers	Core mechanism
Markov-game security self-play	(Hammar et al., 2020, Xue et al., 2021, Chen et al., 2021)	Attacker–defender learning in zero-sum or Bayesian games
Hierarchical pentesting meta-games	(Ge et al., 2024)	Micro tactic games coupled to a macro strategic process
Simulation-trained autonomous attacker	(López-Montero et al., 30 Jun 2025)	RL over procedurally generated vulnerable web environments
LLM red-team self-play	(Wang et al., 15 Jan 2026, Malfa et al., 8 May 2026)	Jailbreak generator versus safety-aligned responder
Fine-grained code self-play	(Chen et al., 2 Jun 2026)	Branching secure and vulnerable continuations at risk nodes

A persistent distinction in this literature is between explicit multi-agent self-play and what may be called self-play-style automated testing. The former optimizes attacker and defender policies in the same game. The latter trains an attacker against a procedurally generated or otherwise adversarial environment whose variation substitutes for a co-learning defender. This distinction is central to both methodology and evaluation (López-Montero et al., 30 Jun 2025, Malfa et al., 8 May 2026).

2. Formal models and solution concepts

The web-application penetration-testing formulation in “Reinforcement Learning for Automated Cybersecurity Penetration Testing” models the problem as an MDP

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R} \rangle$$

while explicitly observing that real cybersecurity environments are often partially observable and conceptually closer to a POMDP (López-Montero et al., 30 Jun 2025). Its action space is defined at the level of tool configurations rather than packets: for a discovered site with $n$ URLs,

$$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$

so the global action space has size $134n$ and grows dynamically during an episode as the crawler finds new URLs (López-Montero et al., 30 Jun 2025). The reward function is explicitly bi-objective,

$$\mathcal{R}(s_t, a_t, s_{t+1}) = \mu \,\mathcal{V}(s_t, s_{t+1}) + (1-\mu)\,\mathcal{C}(a_t),$$

balancing newly discovered vulnerabilities or useful information against action cost (López-Montero et al., 30 Jun 2025).

MEGA-PT formalizes penetration testing as a coupled meta-security game. Each node $v$ carries a Micro Tactic Game $\Gamma^v$, an extensive-form attacker–defender game whose outcomes are neighboring nodes; these are coupled to a Macro Strategic Process

$$\Lambda^g = \langle \mathcal{S}, \mathcal{A}^g, T, R, \gamma \rangle,$$

with $\mathcal{S}=\mathcal{V}$ and $\mathcal{A}^g=\mathcal{E}$ on the network graph $n$0 (Ge et al., 2024). The meta-game

$n$1

is solved by iterating local equilibrium or best-response computations and macro-level value evaluation until a meta penetration playbook

$n$2

is obtained (Ge et al., 2024). This formulation supports red teaming, Stackelberg-style purple teaming, and Nash-equilibrium-based risk assessment within the same hierarchy.

For LLM safety, Safety Self-Play defines a two-step episodic game with a single policy network $n$3: in the attacker role it samples a jailbreak prompt $n$4 from a harmful goal $n$5, and in the defender role it produces a response $n$6 to that prompt (Wang et al., 15 Jan 2026). An external judge assigns a Safety Score in $n$7, mapped to complementary rewards

$n$8

with $n$9, making the game strictly zero-sum (Wang et al., 15 Jan 2026).

“The Attacker in the Mirror” sharpens the formal discussion by writing self-play red teaming as

$$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$0

where $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$1 and safe refusal is represented by a response $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$2 such that $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$3 for all attacker prompts (Malfa et al., 8 May 2026). That analysis shows that the Nash equilibria reachable in the standard game include both a degenerate always-refuse equilibrium and an oracle-like defender equilibrium, and argues that when attacker and defender share and update the same base model, optimization dynamics collapse toward self-consistency rather than sustained adversarial pressure (Malfa et al., 8 May 2026).

Fine-grained secure-code self-play departs from whole-trajectory minimax training and instead localizes adversarial branching to annotated risk points. Tree-like Self-Play models code generation as conditional language modeling over a generation tree and defines CWE Risk Nodes where a single alternative token choice may reintroduce a vulnerability. At each such node, the previous model iteration samples a divergent continuation $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$4, and the current model is trained to prefer the secure golden path $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$5 through a preference-style loss over the pair $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$6 (Chen et al., 2 Jun 2026). This reframes self-play as localized adversarial discrimination at causally decisive decision points.

3. Penetration-testing systems, environments, and architectural substrates

Simulation is central to penetration-testing self-play because real targets are slow, hard to parallelize, and limited in diversity. The RL web-pentesting system of (López-Montero et al., 30 Jun 2025) therefore trains almost entirely on synthetic websites whose topology is generated as a tree by a modified Barabási–Albert preferential attachment procedure with $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$7. The authors report a roughly $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$8 speedup and full parallelization relative to training directly on real vulnerable web applications, while preserving identical state and action spaces between simulation and deployment to enable sim-to-real transfer (López-Montero et al., 30 Jun 2025). Its policy operates over tool families such as crawling, parameter detection, SQL injection, brute force, and XSS; PPO with a two-layer network $$\mathcal{A} := [n] \times \mathcal{A}_{url}, \qquad |\mathcal{A}_{url}| = 134,$$9, learning rate $134n$0, and batch size 256 emerged as the best configuration in a Bayesian hyperparameter search, and the trained PPO agent successfully extracted all vulnerabilities that were in its action capability on DVWA and DockerLabs (López-Montero et al., 30 Jun 2025).

A notable architectural contribution of the same system is its use of geometric deep learning priors over sets of URLs. The critic is permutation invariant,

$134n$1

while the actor is permutation equivariant by concatenating per-URL logits. This reduces parameter count from a naïve estimate of more than 36 million parameters with one hidden layer for $134n$2 to 69,304 parameters in the symmetric design, while also reducing effective search space by respecting URL permutation symmetries (López-Montero et al., 30 Jun 2025).

MEGA-PT attacks the same scalability problem from a different direction. Instead of a monolithic RL state space, it decomposes the problem into node-local Micro Tactic Games and a graph-level Macro Strategic Process, which the authors describe as distributed, adaptive, collaborative, and fast (Ge et al., 2024). This modularity allows independent MTGs to be solved in parallel and reused across identical node types, and it permits local changes—such as changing the app-server MTG or adding many more user devices—to be incorporated by recomputing only the affected local games plus the macro policy. In the reported experiments, MEGA-PT’s runtime increased very slowly with user count, while an RL baseline diverged under the same scaling experiment (Ge et al., 2024).

Red-MIRROR shifts from simulator-centric RL to a multi-agent LLM architecture specialized for long-horizon web exploitation. Its Planner, Collector, Exploiter, and Summarizer agents are coordinated by a tightly coupled memory-reflection backbone built from Retrieval-Augmented Generation, a Shared Recurrent Memory Mechanism, and a Dual-Phase Reflection mechanism (Khang et al., 28 Mar 2026). SRMM provides immutable write-once shared text memory indexed by agent and time step, while local intra-reflection iteratively updates exploit payloads from responses and global inter-reflection revises the task graph via self-consistency checks and summarized shared memory. On the XBOW benchmark, this system attained an overall success rate of 86.0 percent and a 93.99 percent subtask completion rate, outperforming PentestAgent, AutoPT, and the VulnBot baseline; on Vulhub scenarios it achieved performance comparable to state-of-the-art agents (Khang et al., 28 Mar 2026).

4. LLM and code-model self-play red teaming

Safety Self-Play is the clearest instance of explicit self-play security testing for LLMs. A single LLM acts as both jailbreak generator and safety-aligned responder, with an external LLM judge turning each trajectory into attacker and defender rewards, and a Reflective Experience Replay pool storing hard goals or prompts whose reward remains low (Wang et al., 15 Jan 2026). Replay sampling uses a UCB-style priority

$134n$3

with $134n$4 in experiments, so that difficult and under-replayed failure cases are emphasized (Wang et al., 15 Jan 2026). On Qwen2.5-7B, reported Attack Success Rates for SSP were 1.7% on GCG, 2.4% on PAIR, 1.4% on TAP, 1.3% on DAN, 2.1% on DeepInception, and 3.0% on SAA, while OR-Bench refusal rate dropped to 25.3 and major capability benchmarks remained close to vanilla models (Wang et al., 15 Jan 2026). Ablations showed that removing the unified model, replay, or UCB increased ASR, which the authors interpret as evidence that single-policy role sharing plus reflective replay materially improves robustness (Wang et al., 15 Jan 2026).

Anchored Bipolicy Self-Play challenges precisely that unified-model design. It argues that shared-parameter self-play produces gradient collision and self-consistency bias, then replaces shared fine-tuning with a frozen backbone $134n$5 and separate role-specific LoRA adapters $134n$6 and $134n$7 for attacker and defender (Malfa et al., 8 May 2026). This preserves a true bipolicy game while remaining parameter efficient: for Qwen2.5-7B-IT, rank-16 adapters train about 80M parameters and rank-32 adapters about 160M parameters, compared with full-model updates in the billions; the paper reports up to $134n$8 greater parameter efficiency than fine-tuning and cross-play results in which ABSP attackers are stronger and ABSP defenders more robust than standard self-play models (Malfa et al., 8 May 2026).

Tree-like Self-Play adapts self-play to secure code generation by treating vulnerabilities as localized branch decisions rather than whole-sequence failures. Starting from patched “golden” code and annotated CWE Risk Nodes, it samples vulnerable continuations from the previous model and trains the current model to prefer the secure branch (Chen et al., 2 Jun 2026). In Python security evaluation, TSP raised CodeLlama-7B’s SPR@1 to 75.8%, compared with 57.0% for SFT and 69.6% for an unstructured self-play baseline; in C/C++ and cross-language experiments it reduced total detected vulnerabilities and generalized to unseen CWE categories, which the paper characterizes as evidence that the model internalizes abstract, language-agnostic security logic rather than merely memorizing patches (Chen et al., 2 Jun 2026).

Taken together, these LLM and code-model systems show two major interpretations of self-play security testing. One uses explicit zero-sum attacker–defender games with shared or separated roles. The other uses a model’s own high-probability but unsafe continuations as adversarial opponents at finer granularity. Both rely on repeated adversarial feedback, but they differ sharply in where the game is instantiated: between roles, between model copies, or between branches in a generation tree (Wang et al., 15 Jan 2026, Malfa et al., 8 May 2026, Chen et al., 2 Jun 2026).

5. Evaluation substrates, oracles, and methodological constraints

A recurring obstacle in self-play security testing is the distinction between apparent findings and validated exploits. The Go-Explore adaptation for AI red team testing makes this explicit by separating broad “findings” from “verified attacks,” defined causally by an untrusted source containing injection phrases, a dangerous follow-up operation, and successful execution of that operation (Bhatt et al., 31 Dec 2025). Across 28 runs, the study found that random-seed variance dominated algorithmic parameters, producing an $134n$9 spread in outcomes; simple state signatures outperformed complex ones; reward shaping consistently harmed performance, causing exploration collapse in 94% of runs or yielding 18 false positives with zero verified attacks; and ensembles traded off depth within one attack class for diversity across attack types (Bhatt et al., 31 Dec 2025). For self-play security testing methodology, this is a direct warning that seed control, multi-seed averaging, and exploit verification can matter more than nominal algorithmic sophistication.

Execution environments and test oracles are equally decisive. SandboxEval supplies a concrete substrate for self-play against code-executing infrastructures: it is a suite of 51 hand-written Python programs covering sensitive information exposure, filesystem manipulation, external communication, and dangerous operations, with outcomes classified as Accessed, Denied, or Unknown (Rabin et al., 27 Mar 2025). Because LLM-generated code can realize these attacks, the suite functions as a capability taxonomy and reward surface for any self-play agent that tries to compromise its own evaluation sandbox (Rabin et al., 27 Mar 2025). CodeHacker contributes a different lesson: before attacking the target program, it adversarially calibrates its own Validator and Checker using self-generated adversarial probes, then uses multi-strategy attack generation to raise Hack Success Rate to 51.40% and dataset True Negative Rate to 96.05% (Shi et al., 23 Feb 2026). This “first harden the judge, then attack the target” pattern recurs across robust self-play pipelines.

Two complementary strands address the oracle problem directly. MST-wi introduces Metamorphic Security Testing for Web-interactions, a metamorphic testing approach with a catalog of 76 system-agnostic metamorphic relations that automates 39% of the OWASP security testing activities not automated by state-of-the-art techniques and can discover 102 vulnerability types (Chaleshtari et al., 2022). Automatic web security unit testing for XSS extracts encoder-specific unit tests and executes 497 generated attack strings per sink; in one evaluation, this found 0-day XSS vulnerabilities missed by Fortify SCA and covered 83.5% of 400 randomly selected attacks from xssed.com by exact match, with the rest mapped to generated attacks (Mohammadi et al., 2018). A more formal closed-loop variant appears in “Learn, Check, Test,” where active black-box learning infers Mealy machines, Context-based Proposition Maps annotate them with propositions, the result is translated to Rebeca for LTL model checking, and counterexample traces are replayed against the real SUL; if the trace does not reproduce, it becomes a counterexample for a new learning iteration (Marksteiner et al., 26 Sep 2025). These systems are not all self-play in the strict game-theoretic sense, but they provide the oracles, abstractions, and validation loops on which self-play security testing depends.

6. Misconceptions, limitations, and future directions

A common misconception is that self-play security testing always means two simultaneously learning agents in a strict minimax game. The literature is more heterogeneous. Some systems are explicit attacker–defender games with zero-sum rewards and equilibrium language; others are single-agent attacker training in simulators that merely “match the self-play philosophy”; still others implement closed-loop adversarial testing through reflective verification, automata learning, or metamorphic relations rather than direct policy optimization (Ge et al., 2024, López-Montero et al., 30 Jun 2025, Marksteiner et al., 26 Sep 2025).

Several limitations recur across domains. In automated web pentesting, the action space remains narrow—mainly crawling, SQLi, XSS, brute force, and parameter discovery—and the authors explicitly state that the RL pentester is not yet human-level, that simulated tool behavior is simplified, and that no defender or explicit adversarial training is present (López-Montero et al., 30 Jun 2025). In LLM safety self-play, the current scope is text-only, the process is more compute-intensive than standard SFT or RLHF, long-term robustness against future attack families remains open, and the same machinery that strengthens defenses also produces powerful jailbreak prompts whose access must be constrained (Wang et al., 15 Jan 2026). ABSP reports LoRA-instability failure modes in some settings, including collapse to nonsense outputs, while TSP performs worse on several long-range memory and data-flow vulnerabilities such as CWE-690, CWE-125, CWE-416, and CWE-457, indicating that localized node supervision can be insufficient for deeper semantic flaws (Malfa et al., 8 May 2026, Chen et al., 2 Jun 2026). Red-MIRROR, despite strong long-horizon performance, remains weak on blind low-signal vulnerabilities, framework-specific exploit semantics, and cryptographic tasks, and its success depends heavily on a frontier-scale reasoning model (Khang et al., 28 Mar 2026).

Future directions in the literature are correspondingly convergent. One direction is richer multi-agent co-evolution: introducing explicit defenders into currently attacker-only simulators, extending purple-teaming formulations to macro-level defenses, and using self-play to generate progressively harder environments or curricula (Ge et al., 2024, López-Montero et al., 30 Jun 2025). A second is architectural: stronger graph-based reasoning, model-based planning, LoRA- or adapter-based role separation, and memory-reflection backbones that preserve long-horizon state without inducing self-consistency collapse (López-Montero et al., 30 Jun 2025, Malfa et al., 8 May 2026, Khang et al., 28 Mar 2026). A third is methodological: better exploit verification, multi-seed evaluation, more realistic benchmarks with WAFs and adaptive defenses, and tighter integration of security-specific oracles, validators, and model checkers into adversarial training loops (Bhatt et al., 31 Dec 2025, Rabin et al., 27 Mar 2025, Marksteiner et al., 26 Sep 2025). The field’s trajectory suggests that self-play security testing is becoming less a single algorithm and more a design pattern: repeated adversarial interaction, explicit memory of past failures, formal or operational validation of exploit success, and continual refinement of both offensive and defensive strategies.