Flip-PSRO: Robust Cyber Defense Training

• Flip-PSRO is a population-based multi-agent RL method that trains robust cyber defenders by leveraging PPO and tailored meta-game utilities.
• It adapts training objectives with ownership-based win rates and normalized performance gaps to address stealth and costly intervention in cyber defense.
• Empirical evaluations demonstrate that Flip-PSRO improves defender performance against unseen attacks nearly twofold compared to standard PSRO methods.

Searching arXiv for Flip-PSRO and closely related PSRO variants to ground the article in current papers. Flip-PSRO is a population-based multi-agent reinforcement learning method introduced for training robust defenders in a FlipIt-style cyber defense game. It is defined within the PoolFlip framework, where a defender and an attacker compete for control of one or more resources under partial observability, action costs, and stale information. The method adapts the policy-space response oracles paradigm to this domain by training a PPO defender against a meta-strategy over a pool of attacker policies, with meta-game utilities based not only on environment reward but also on ownership-oriented criteria such as win rate by ownership threshold and normalized performance gap (Cadet et al., 27 Aug 2025).

1. Origins and problem setting

Flip-PSRO was proposed together with PoolFlip as a response to a specific cyber-defense difficulty: a defender that is trained only against one fixed attacker can become brittle when deployed against unknown, potentially adaptive opponents. The underlying game model comes from FlipIt, in which attacker and defender compete to control a shared resource by taking a Flip action and paying a cost. In this formulation, stealth is central: when compromise occurs, the defender does not automatically observe the change in ownership, and knowledge about the resource state becomes stale unless refreshed by explicit action. The paper frames the goal as training a generalist defender that maintains control under partial observability, costly intervention, deceptive timing, and evolving attacker behavior (Cadet et al., 27 Aug 2025).

PoolFlip extends prior FlipIt formulations in several directions that matter directly for Flip-PSRO. It is a gym-style multi-agent reinforcement learning environment; both attacker and defender can be trainable; heuristic populations are parameterized; a Check action is added alongside Flip and Sleep; multiple resources are supported in principle; and observations are preprocessed into vector inputs suitable for deep RL. Within this setup, Flip-PSRO is the training method rather than the environment itself. The environment is PoolFlip; Flip-PSRO is the PSRO-inspired population-training procedure run inside it (Cadet et al., 27 Aug 2025).

A central conceptual distinction from much of the classical PSRO literature is that Flip-PSRO is not evaluated through exploitability or regret in the standard two-player zero-sum sense. Instead, it uses empirical utilities tailored to cyber takeover games, especially average reward, ownership-based win rates, and normalized performance gaps. This places it in the PSRO family while also making it domain-specific in objective design (Cadet et al., 27 Aug 2025).

2. PoolFlip game model

The PoolFlip game has two players, Blue Agent as defender and Red Agent as attacker, competing over one or more resources such as “a password, software, server, network, etc.” Ownership of a resource is the key latent state variable. In the reported experiments, the defender starts with the resource, reflecting the intended cyber-defense interpretation (Cadet et al., 27 Aug 2025).

The action set for each player consists of Sleep, Check(resource), and Flip(resource). With $R$ resources, the action space has size

$1 + 2 \times R,$

namely

$$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$

The semantics differ by side. Defender Flip corresponds to remediation, reset, patch, or reclaim; attacker Flip corresponds to exploit, compromise, escalation, or persistence; defender Check corresponds to investigation or scanning; attacker Check corresponds to reconnaissance or persistence validation (Cadet et al., 27 Aug 2025).

The environment evolves in discrete time, with both players able to act at each step. Ownership transitions are deterministic except when simultaneous contested flips occur without the prior owner also flipping. For resource $i$ at time $t$,

$$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$

where $C$ is the set of contestants who claimed ownership of resource $i$ at that step. This rule preserves ownership if nobody flips or if the current owner also flips, and otherwise randomizes ownership among simultaneous non-owner claimants (Cadet et al., 27 Aug 2025).

Reward couples control and action efficiency: $$\text{Reward}(A,t)=\sum_{i \in S}\text{Gain}(A,i)-\text{Cost}(A,a_t),$$ where $S$ is the set of resources owned by agent $1 + 2 \times R,$0 at time $1 + 2 \times R,$1, and $1 + 2 \times R,$2 is the action taken. This construction makes constant flipping suboptimal when flip cost is high relative to per-step gain. The paper explicitly notes that if

$1 + 2 \times R,$3

and $1 + 2 \times R,$4, then the agent must wait at least $1 + 2 \times R,$5 time steps before flipping again to achieve positive reward (Cadet et al., 27 Aug 2025).

Observability is partial and stale. Players do not automatically observe the opponent’s last move; they must Flip or Check to refresh knowledge about a resource. The learned observation for each resource encodes ownership indicator, time since the agent captured the resource, and time since the opponent captured it. With memory limit $1 + 2 \times R,$6, the elapsed time is clipped by $1 + 2 \times R,$7, then one-hot encoded with an extra unknown-status cell. The observation size per resource is

$1 + 2 \times R,$8

so for $1 + 2 \times R,$9 resources the total observation vector size is

$$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$0

Episodes in the reported experiments use 100 time steps, 1 resource, and 2 players (Cadet et al., 27 Aug 2025).

3. Algorithmic structure

Flip-PSRO instantiates PSRO in PoolFlip by combining a restricted opponent pool, a meta-strategy solver, empirical utility updates, and PPO-based approximate response oracles. The initial pool $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$1 is a set of heuristic policies. The response objective $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$2 is chosen from $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$3. An expected utility matrix $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$4 is maintained, and the first-iteration meta-strategy is uniform over the pool (Cadet et al., 27 Aug 2025).

At each training iteration, the current defender policy is optimized against opponents sampled from the current meta-strategy. The loop described in the paper is:

1. For each episode $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$5, sample opponent policy $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$6.
2. Train PPO policy $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$7 against $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$8.
3. For each $$[\text{Sleep}, \text{Flip Resource 1}, \text{Check Resource 1}, \ldots, \text{Flip Resource } R, \text{Check Resource } R].$$9, update $i$0 with expected utilities for games $i$1.
4. If $i$2, set

$i$3

Otherwise use a uniform distribution over $i$4.

1. In self-play variants, extend the pool by adding the current trained policy:

$i$5

2. Output the current policy $i$6 (Cadet et al., 27 Aug 2025).

In this formulation, PPO is the response oracle. Best responses are therefore only approximate, and meta-game utilities are estimated by simulation rather than analytically. The paper is explicit that the pool initially consists of heuristic attackers rather than a complete strategy space, and that softmax over utility entries is used as a practical meta-strategy solver (Cadet et al., 27 Aug 2025).

The reported PPO configuration is: Adam optimizer, learning rate $i$7, discount factor $i$8, clipping parameter $i$9, 4 epochs per update, update every 10 episodes, mean squared error loss, and entropy regularization coefficient $t$0. Flip-PSRO training is run for 200 epochs or iterations, with 100 episodes per epoch, 10 episodes stored in memory between updates, and evaluation averaged over 100 episodes (Cadet et al., 27 Aug 2025).

The paper also states that the framework is symmetric in the sense that attacker and defender use the same reward formulation and can execute an identical strategy. However, the main empirical use of Flip-PSRO is asymmetric in focus: it trains a PPO defender against a pool of attacker policies and studies defender robustness (Cadet et al., 27 Aug 2025).

4. Meta-game utilities and response objectives

A distinctive feature of Flip-PSRO is that the restricted-game utility is not fixed to raw environment reward. The paper introduces ownership-based and performance-gap-based objectives for computing the meta-strategy, with the goal of aligning training more closely with cyber-defense desiderata such as sustained control (Cadet et al., 27 Aug 2025).

The first objective is win rate by ownership. For a single resource $t$1 and agent $t$2, the ownership score is

$t$3

that is, the fraction of the horizon during which the resource is controlled by agent $t$4. A defender is considered the winner if this ownership exceeds a chosen threshold $t$5, producing meta-strategy variants such as MSS-O50% and MSS-O70% (Cadet et al., 27 Aug 2025).

The second objective is normalized performance gap. Let $t$6 be a heuristic attacker, $t$7 a specialist trained against $t$8, and $t$9 the specialist’s average reward against $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$0 after convergence. Then for any agent $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$1,

$$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$2

Across heuristics $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$3, the normalized gap is

$$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$4

This objective emphasizes where the current defender is weakest relative to specialist performance, and the corresponding meta-strategy solver is denoted MSS-Gap (Cadet et al., 27 Aug 2025).

The paper evaluates three principal meta-strategy solvers: MSS-Unif, which samples uniformly from the pool; MSS-Gap, which uses normalized performance gap; and MSS-O$$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$5, which uses an ownership threshold such as $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$6 or $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$7. The first iteration is uniform because no trained defender checkpoints yet exist, while later iterations use softmax-transformed utility scores for ownership and gap objectives (Cadet et al., 27 Aug 2025).

This objective design is the main sense in which Flip-PSRO departs from canonical PSRO formulations. In standard zero-sum PSRO variants, the meta-solver is usually discussed in terms of restricted-game equilibrium or exploitability. Here, the meta-game is explicitly reoriented toward control and transfer performance. A plausible implication is that Flip-PSRO should be understood less as an equilibrium-quality algorithm in the classical sense and more as a robust-response training procedure whose restricted-game semantics are specialized to stealthy cyber defense (Cadet et al., 27 Aug 2025).

5. Empirical evaluation

The experiments use a single-resource training environment with 100 timesteps per episode, 2 players, and action costs Sleep $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$8, Check $$O_i(t) = \begin{cases} O_i(t-1) & \text{if } C = \emptyset \text{ or } O_i(t-1) \in C \ X \sim Uniform(C) & \text{otherwise,} \end{cases}$$9, Flip $C$0, with ownership gain per timestep $C$1. The primary training attacker pool contains Periodic, Burst, Awakening, Periodic Check, and Periodic Aggressive Check heuristics, with default parameters phase 4, random delay, burst length 3, Burst using phase 8, and awakening rate $C$2. Baselines are specialists trained against single heuristics, Iterated Best Response (IBR), and the best heuristic defender, Awakening(0.05) (Cadet et al., 27 Aug 2025).

The main in-distribution and transfer results are summarized below.

Method	Avg reward over training pool	Avg reward on unseen attack variants
IBR	23.2	14.2
MSS-Unif	25.8	27.2
MSS-O50%	31.1	32.3
MSS-O70%	27.4	20.1

These transfer numbers are the basis for the paper’s claim that Flip-PSRO defenders are approximately $C$3 more effective than baselines on unseen heuristic attacks: the best reported variant, MSS-O50%, achieves 32.3 average reward on unseen attack variants, compared with 14.2 for IBR (Cadet et al., 27 Aug 2025).

On the training pool, specialists perform best on their own target opponents, with 40.0 average reward, but the paper treats them as upper-bound-style baselines rather than robust generalists. The best heuristic baseline, Awakening(0.05), reaches 13.8 average reward, and IBR improves this to 23.2. Flip-PSRO variants outperform IBR on the same pool, with MSS-O50% reaching 31.1 average reward. For MSS-O50%, the per-opponent reported rewards are 36.8 for Periodic(4), 27.1 for Burst(8,3), 27.6 for Awake(0.05), 47.4 for PC(4), and 16.4 for PAC(4) (Cadet et al., 27 Aug 2025).

Ownership statistics exhibit the effect of objective design especially clearly. Average ownership is 60.5 for MSS-Unif, 72.28 for MSS-Gap, 76.3 for MSS-O50%, and 81.12 for MSS-O70%. Increasing the ownership target from $C$4 to $C$5 raises actual average ownership from about 76% to 81%, although this higher control level comes with weaker transfer reward than O50% (Cadet et al., 27 Aug 2025).

The paper also studies out-of-distribution generalization using unseen attacker variants $C$6, $C$7, $C$8, $C$9, $i$0, $i$1, and $i$2. The best transfer result is again MSS-O50%. The authors note that transfer is better for unseen opponents lying “within the convex hull” of the training strategies; for example, phase multiples of 4 transfer more effectively than off-grid variants such as phase 6. This constrains the meaning of generalization in the reported setting (Cadet et al., 27 Aug 2025).

A self-play extension, in which earlier versions of the learned policy are added to the pool, produces a different outcome. Reported performance against heuristics decreases, and the learned strategy shifts toward frequent checking and reactive flipping, producing stalemate-like behavior in which both versions inspect often and flip upon losing control. The paper concludes that self-play may be more beneficial in richer environments with larger action spaces (Cadet et al., 27 Aug 2025).

6. Position within the PSRO literature and known limitations

Within the PSRO family, Flip-PSRO occupies a notably different position from methods that are organized around exploitability minimization in two-player zero-sum games. "Anytime PSRO for Two-Player Zero-Sum Games" (McAleer et al., 2022) redefines the restricted-game objective so that exploitability is monotonically non-increasing across iterations, while "Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Games" (McAleer et al., 2022) adds a stochastic policy each iteration to reduce exploitability faster in games whose equilibria are fundamentally mixed. Flip-PSRO, by contrast, does not report exploitability or regret in the standard zero-sum PSRO sense; its empirical criteria are average reward, ownership fraction, ownership-threshold win rate, and normalized performance gap (Cadet et al., 27 Aug 2025).

It also differs from PSRO variants that modify best-response construction itself. "Conflux-PSRO: Effectively Leveraging Collective Advantages in Policy Space Response Oracles" (Huang et al., 2024) builds a stronger oracle by routing among selected historical policies at state level, and "Policy Space Diversity for Non-Transitive Games" (Yao et al., 2023) regularizes best-response training toward policy-hull enlargement so as to reduce population exploitability. Flip-PSRO does neither: it uses PPO as a conventional approximate oracle, while the distinctive intervention is the choice of opponent pool and the utility used to compute the sampling distribution over that pool (Huang et al., 2024, Yao et al., 2023).

The paper’s limitations are explicit. All reported Flip-PSRO experiments use a single-resource environment even though PoolFlip supports multiple resources. The main robustness claims are against heuristic attacker populations and parameter variants rather than fully co-trained adaptive attacker populations. PPO provides only approximate best responses. The paper invokes approximate Nash ideas from PSRO, but does not prove convergence or equilibrium quality in this domain. Transfer depends on similarity between unseen attackers and the training pool. MSS-Gap can hurt transfer by overfocusing on training-pool weaknesses. Self-play reduces heuristic performance in the reported setup. These constraints make the method best understood as a domain-specific, empirically validated robust-defense training procedure rather than a theoretically characterized equilibrium solver (Cadet et al., 27 Aug 2025).

A common misconception is to identify Flip-PSRO with PoolFlip itself. The distinction is strict: PoolFlip is the multi-agent gym environment with stealthy ownership dynamics, while Flip-PSRO is the population-based PPO-plus-meta-strategy training method run inside that environment (Cadet et al., 27 Aug 2025). Another misconception is to read Flip-PSRO as directly interchangeable with canonical PSRO baselines from zero-sum imperfect-information games. The surrounding literature suggests otherwise: the most relevant comparison axis is not exploitability monotonicity or restricted-game Nash computation, but the design of response objectives for robust control under partial observability and costly intervention (McAleer et al., 2022, McAleer et al., 2022).