Zero-Sum Markov Security Game

• Zero-Sum Markov Security Game is a stochastic formulation that models multi-stage adversarial interactions with strictly opposed attacker and defender utilities.
• It employs attack graphs, CVSS-based parameters, and dynamic programming to capture state transitions and optimize strategic resource allocation.
• Practical applications include intrusion detection, moving target defense, and strategic IDS placement, with equilibrium computed via Shapley-type methods.

Searching arXiv for relevant papers on zero-sum Markov security games and related stochastic game formulations. A zero-sum Markov security game is a stochastic game formulation of adversarial security dynamics in which an attacker and a defender repeatedly choose actions in a Markovian environment, the system state evolves according to a transition kernel, and one side’s gain is the other side’s loss. In the security literature, such games are used to model multi-stage intrusion, intrusion detection, moving target defense, patrolling, and resource allocation over attack graphs or network states, while preserving the sequential dependence of compromise, detection, and recovery. In the most basic finite-state form, the model is specified by a state space, attacker and defender action sets, transition probabilities, a stage payoff, and either a discount factor or probabilistic termination; equilibrium analysis is then expressed through Shapley-type dynamic programming equations, with stationary mixed strategies under standard discounted assumptions (Renault, 2019). In cyber-security applications, the formalism has been instantiated with attack-graph states, CVSS-derived transition and reward parameters, and defender actions such as IDS placement or monitor deployment (Alavizadeh et al., 2021, Chowdhary et al., 2018, Chowdhary et al., 2018).

1. Conceptual and mathematical definition

A zero-sum Markov security game is a two-player stochastic game in which the defender and attacker jointly control state transitions while having strictly opposed utilities. In the generic finite-state model, the state space is a finite set $S$, the defender chooses actions from a finite set $A$, the attacker chooses actions from a finite set $B$, the transition kernel is $P(s' \mid s,a,b)$, and the stage payoff to the defender is $r(s,a,b)$, with the attacker receiving $-r(s,a,b)$ (Renault, 2019). In security language, the state can encode alarm status, privilege level, compromised hosts, attacker location, or belief over hidden attacker states, while the players’ actions correspond to patrol moves, exploit choices, monitoring allocations, or mitigation operations (Zhao et al., 2021, Horák et al., 2020).

The defining zero-sum condition is that for every state-action profile, the two instantaneous payoffs sum to zero. This induces the standard equilibrium objective

$$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$

with $\pi$ denoting a defender policy and $\mu$ an attacker policy (Zhao et al., 2021). In the discounted setting, the relevant performance criterion is the infinite-horizon discounted return, and the equilibrium is characterized by a fixed point of a Shapley operator; in finite-horizon or asymptotic-average settings, the corresponding $n$-stage and $A$0-discounted values are linked through classical zero-sum stochastic game theory (Renault, 2019).

Security-specific formulations often refine the generic model in one of two directions. One direction uses explicit attack graphs and privilege states, so that each state denotes a phase of a multi-stage intrusion and actions represent exploit attempts or monitoring choices (Chowdhary et al., 2018, Chowdhary et al., 2018). The other direction uses network-security states with correlated node conditions, where compromise of one node changes the effective assets and vulnerabilities of the remainder of the network (Nguyen et al., 2010). This suggests that the term “Markov” in this context is not merely a temporal abstraction but a structural assumption about how security posture evolves under adversarial interaction.

2. State spaces, action spaces, and transition structure in security models

The attack-graph formulation is a central specialization. In one model, the cloud is represented as an attack graph $A$1 of 10 VMs on 5 physical hosts, and one Markov state is defined per attack step along the shortest attack path to the database (Alavizadeh et al., 2021). The concrete state set is

$A$2

with $A$3 meaning that no VM has yet been exploited, $A$4 that $A$5 has been exploited, $A$6 that $A$7 has been exploited, $A$8 that $A$9 has been exploited, and $B$0 that the database has been exploited, which is terminal (Alavizadeh et al., 2021). In a closely related cloud MTD formulation, the starting object is also an attack graph, but its nodes are typed as fact nodes, exploit nodes, privilege nodes, and a root or goal node, and the non-terminal nodes are partitioned into subgraph states $B$1 (Chowdhary et al., 2018).

The action sets are correspondingly security-specific. In the shortest-attack-path model, at each non-terminal state $B$2 the attacker may choose either $B$3 or an exploit $B$4 against a neighboring VM on the path, while the defender may choose either $B$5 or $B$6, meaning placement of an IDS on one host $B$7 (Alavizadeh et al., 2021). In the cloud MTD model, the adversary chooses among known exploits whose preconditions are satisfied in the current state, plus a no-op action, and the defender chooses among monitoring or IDS deployments, plus a no-monitor action; the defender can also be budget-constrained, for example to at most one port monitor per state (Chowdhary et al., 2018). In the interdependent-node model, the attacker may attack any uncompromised node or do nothing, while the defender may defend any node or do nothing (Nguyen et al., 2010).

Transition structure is where the Markov abstraction becomes operational. In the shortest-path cloud model, the attacker chooses exploits in proportion to their CVSS exploitability scores and succeeds with that same probability, while the defender is assumed to randomize uniformly in the baseline examples (Alavizadeh et al., 2021). Specifically, if $B$8 is the CVSS exploitability of $B$9, then

$P(s' \mid s,a,b)$0

and the combined attack-action probability of moving on-path is

$P(s' \mid s,a,b)$1

Because attacker and defender act independently, the paper writes

$P(s' \mid s,a,b)$2

for the example construction (Alavizadeh et al., 2021). In the MTD attack-graph formulation, the probability of a successful exploit is proportional to the CVSS Exploitability Score and is set to zero if the defender’s monitor matches the exploit’s corresponding IDS, so that detection prevents forward movement and leaves the game in the same state (Chowdhary et al., 2018).

A distinct formulation appears in the interdependent-node network model. There the state is a bit vector $P(s' \mid s,a,b)$3, with each component indicating whether a node is safe or compromised (Nguyen et al., 2010). The key complication is that both “effective security assets” and vulnerability supports change with the compromise pattern. The success probability of an attack on node $P(s' \mid s,a,b)$4 is an affine function of a support quantity $P(s' \mid s,a,b)$5, with different boundary probabilities depending on whether the defender chooses to defend node $P(s' \mid s,a,b)$6 or not (Nguyen et al., 2010). Failed attacks may restore the system to the all-zero state, terminate the game, or leave the state unchanged. This model emphasizes correlated network structure rather than a single explicit attack path.

3. Payoff construction and CVSS-based parameterization

In security applications, immediate payoffs are usually not arbitrary numerical utilities but are built from vulnerability scores or network asset weights. In the shortest-path cloud model, the reward function uses CVSS impact $P(s' \mid s,a,b)$7 and a fixed defense cost $P(s' \mid s,a,b)$8, with the game specified as zero-sum through $P(s' \mid s,a,b)$9 (Alavizadeh et al., 2021). The attacker’s payoff takes different forms depending on whether the attacker exploits and whether the defender places an IDS on the corresponding host, on an off-target host, or not at all. An exploit that succeeds without matching defense yields $r(s,a,b)$0, while detection on the correct host yields $r(s,a,b)$1 (Alavizadeh et al., 2021). This directly links stage utility to the security significance of the compromised VM.

In the cloud MTD model, the adversary gains the CVSS Impact Score $r(s,a,b)$2 when an exploit succeeds undetected, and gains $r(s,a,b)$3 if the exploit is detected; the defender receives the negative of that amount and additionally incurs a fixed performance penalty $r(s,a,b)$4 whenever a monitor is deployed (Chowdhary et al., 2018). The defender reward is written as

$r(s,a,b)$5

This construction makes the security-performance tradeoff explicit: monitoring is beneficial for high-impact exploits but never free (Chowdhary et al., 2018).

A related mapping appears in the adaptive MTD framework, which derives both rewards and transition probabilities from CVSS metrics (Chowdhary et al., 2018). There, Access Complexity determines success and detection probabilities through a simple discrete mapping: AC = EASY gives $r(s,a,b)$6 and $r(s,a,b)$7, AC = MEDIUM gives $r(s,a,b)$8 and $r(s,a,b)$9, and AC = HIGH gives $-r(s,a,b)$0 and $-r(s,a,b)$1 (Chowdhary et al., 2018). CIA score values in $-r(s,a,b)$2 determine immediate payoff magnitudes, including the positive payoff for undetected compromise, the negative payoff for detected exploitation, and a $-r(s,a,b)$3 false-positive style cost when the defender monitors and the attacker does nothing (Chowdhary et al., 2018).

The interdependent-node model uses a different payoff semantics. There, the attacker’s immediate payoff from attacking node $-r(s,a,b)$4 in state $-r(s,a,b)$5 under defender action $-r(s,a,b)$6 is

$-r(s,a,b)$7

where $-r(s,a,b)$8 is the effective asset value after applying the influence-reweighted asset model (Nguyen et al., 2010). The defender’s reward is the negative of this amount. This design internalizes network externalities into the instantaneous payoff rather than solely into state transitions (Nguyen et al., 2010).

A plausible implication is that “zero-sum” in security modeling often coexists with heterogeneous operational semantics: some models interpret payoff as attacker gain, some as defender loss, and some incorporate explicit monitoring cost or performance penalty. What remains invariant is the use of a single scalar stage utility with opposite signs for the two players.

4. Dynamic programming, equilibrium, and value equations

The canonical solution concept is the discounted min-max equilibrium characterized by Bellman or Shapley equations. In the shortest-path AI-based attack mitigation model, if $-r(s,a,b)$9 denotes the value from the attacker’s viewpoint and $$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$0, the global objective is

$$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$1

where $$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$2 is the attacker’s state-conditional mixed strategy (Alavizadeh et al., 2021). With

$$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$3

the recursive form becomes

$$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$4

If the attacker is restricted to pure actions, the equation reduces to $$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$5 (Alavizadeh et al., 2021).

In the cloud MTD monitoring formulation, the defender is the maximizing player. The paper writes

$$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$6

and then

$$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$7

By Shapley’s theorem, the associated Bellman operator is a $$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$8-contraction in the sup norm and admits a unique fixed point $$V^*(s)=\max_{\pi}\min_{\mu}V^{\pi,\mu}(s)=\min_{\mu}\max_{\pi}V^{\pi,\mu}(s),$$9, together with stationary mixed defender policies $\pi$0 that attain the discounted min-max equilibrium (Chowdhary et al., 2018). The contraction argument is given explicitly through an operator $\pi$1 defined by the stage min-max expression (Chowdhary et al., 2018).

The interdependent-node stochastic security game uses the Shapley equation from the defender’s perspective: $\pi$2 Because every pure-action pair has a positive probability of termination, the total expected payoff is well defined and finite even in an undiscounted interpretation, though a discount factor can also be introduced (Nguyen et al., 2010). The paper invokes Shapley’s theorem and related stochastic-game theory to state the existence and uniqueness of the bounded value vector and the existence of equilibrium stationary mixed strategies (Nguyen et al., 2010).

The broader theory clarifies the status of these security-specific equations. For finite state and action spaces, both $\pi$3-stage values $\pi$4 and $\pi$5-discounted values $\pi$6 satisfy Shapley recursions and exist by standard minimax results; in the discounted case, stationary optimal strategies suffice (Renault, 2019). The uniform value also exists in any finite zero-sum stochastic game and coincides with the asymptotic limits of the finite-horizon and vanishing-discount values (Renault, 2019). This places security models built on discounted Markov games within a well-established equilibrium framework rather than a merely heuristic optimization setting.

5. Solution methods: value iteration, linear programming, and learning

The most direct computational method is value iteration. For the shortest-path attack mitigation model, standard zero-sum value iteration is described as initializing $\pi$7 and repeatedly computing

$\pi$8

followed by

$\pi$9

until convergence (Alavizadeh et al., 2021). In the interdependent-node model, the recursion is expressed at each state as a zero-sum matrix game $\mu$0 with entries

$\mu$1

and the update is $\mu$2 (Nguyen et al., 2010). Policy iteration and linear programming are also mentioned as applicable alternatives (Nguyen et al., 2010).

The MTD monitoring model makes the linear programming step explicit. At each state, given current $\mu$3, a payoff matrix $\mu$4 is formed and the zero-sum matrix game

$\mu$5

is solved via linear programming by maximizing a scalar $\mu$6 subject to linear lower-bound constraints over attacker actions and simplex constraints over $\mu$7 (Chowdhary et al., 2018). The paper uses asynchronous value iteration over the state space (Chowdhary et al., 2018).

When model parameters are not known a priori, Q-learning variants become relevant. A standard two-player zero-sum Q-learning extension is given in the shortest-path model summary: $\mu$8 The paper notes that no such learning was actually carried out there, but identifies it as the standard extension to two-player zero-sum settings (Alavizadeh et al., 2021).

More recent reinforcement-learning work studies decentralized and function-approximate learning in zero-sum Markov games. A radically uncoupled, two-timescale Q-learning dynamic updates local Q-functions and value estimates using only each agent’s own rewards and states, without observing the opponent’s action or reward (Sayin et al., 2021). Under stochastic-approximation conditions, the dynamics converges to a Nash equilibrium in self-play and to a best response against an asymptotically stationary opponent (Sayin et al., 2021). For larger two-team zero-sum Markov games, FM3Q factorizes a joint minimax $\mu$9-function into individual ones under the individual-global-minimax principle, and uses a monotonic mixing network with fitted-Q style minimax targets (Hu et al., 2024). This is not a cyber-security paper in the narrow sense, but it is directly relevant where defender or attacker resources are distributed across multiple coordinated agents.

Policy-based methods have also acquired formal guarantees. A two-phase policy-optimization procedure with function approximation alternates a state-wise greedy step solving zero-sum matrix games and an iteration step approximating an attacker best response via Natural Policy Gradient, with polynomial sample and iteration complexity under standard assumptions (Zhao et al., 2021). This suggests that equilibrium computation in large zero-sum security games need not be limited to tabular dynamic programming.

6. Security applications and empirical illustrations

Several empirical security studies instantiate the formalism on cloud or network examples. In the AI-based cyber attack mitigation paper, the “attacker” is assumed AI-aided and solves a shortest-path approximation via a DNN embedding approach, while the defender re-deploys a single IDS per step under a baseline uniform-random strategy (Alavizadeh et al., 2021). The paper computes transition probabilities and stage payoffs manually for a 10-VM toy cloud and draws the corresponding Markov chain, but does not include a large-scale simulation (Alavizadeh et al., 2021). The model is therefore primarily a formal demonstration of how CVSS exploitability and impact can be embedded in a zero-sum Markov game under an AI-enabled attacker model.

The cloud MTD study provides both a toy example and a small real-world case. The toy network has three hosts—LDAP, Web-Server, and FTP—with vulnerabilities CVE-2016-5195, CVE-2017-5095, and CVE-2015-3306, respectively; the resulting attack graph yields $n$0 states $n$1, where $n$2 is terminal, and the defender can monitor at most one port per state (Chowdhary et al., 2018). The paper compares the optimal mixed-strategy equilibrium against two baselines: Min-Max Pure Strategy and Uniform-Random Strategy. Across all $n$3, the optimal mixed-strategy defender value strictly exceeds both baselines (Chowdhary et al., 2018). A key reported qualitative finding is that when a state is close to the goal, the equilibrium assigns zero probability to the no-monitor action, whereas when the state is far from the goal, the defender may randomize with positive probability on no-monitor in order to trade off performance (Chowdhary et al., 2018).

The same paper also includes a real-world APT case study in a flat DMZ with an IPFire gateway and VMs running Windows 2012 R2, Windows 7, Debian 6 with vsftpd, and CentOS 6 (Chowdhary et al., 2018). The attack stages include a low-and-slow SSH exploit on the gateway, exploitation of MS17-010 from Windows 2012 to Windows 7, exploitation of vsftpd on Debian for data exfiltration, and post-exploit persistence or jumping (Chowdhary et al., 2018). In a sub-experiment on state $n$4, there are 5 exploitable CVEs and 5 possible monitors, with defender budget $n$5, and the optimal mixed strategy yields higher expected utility than uniform randomization (Chowdhary et al., 2018).

The adaptive MTD paper provides one of the clearest quantitative comparisons. Its OpenStack testbed contains three VMs and 100 real CVEs with CVSS 2.0 CIA and AC values obtained via cve-search (Chowdhary et al., 2018). The attacker’s goal is to obtain root on all VMs via monotonic multi-stage exploitation paths, while the defender allocates limited monitoring budget. Two heuristics are compared: a naive strategy that patches or monitors the top $n$6 of vulnerabilities by CIA alone, and a strategic one using the Markov-game equilibrium to choose which $n$7 of states to monitor (Chowdhary et al., 2018). The reported attacker equilibrium rewards are as follows:

Coverage %	Naive Reward	Strategic Reward
10	163.40	132.82
20	120.23	111.93
30	96.19	43.45
40	75.09	34.86
50	67.34	30.79

The paper states that at 50% coverage the strategic approach reduces the attacker’s value from approximately 67 to approximately 31, which it describes as a nearly $n$8 improvement (Chowdhary et al., 2018). This is one of the stronger application-level arguments for Markov-game-based defense over purely static, score-sorted heuristics.

The interdependent-node paper illustrates a different empirical scale. For $n$9, with independent assets $A$00, a chosen support matrix, specific boundary probabilities, and restoration and termination probabilities, value iteration converges in about 56 steps at tolerance $A$01 (Nguyen et al., 2010). At the all-safe state $A$02, the attacker’s optimal mix is $A$03 on node 1 and $A$04 on node 3, while the defender’s optimal mix is $A$05 on node 1 and $A$06 on node 3; the corresponding equilibrium value is $A$07, interpreted as defender loss (Nguyen et al., 2010). This example shows how interdependence can shift equilibrium effort away from apparently symmetric nodes.

7. Extensions, limitations, and related models

A recurrent limitation of attack-graph Markov security games is state-space simplification. The shortest-path model explicitly handles only one “shortest path” at a time and one defender token, whereas real cloud environments may contain multiple concurrent attack paths and simultaneous IDS placements (Alavizadeh et al., 2021). The same summary notes assumptions of full observability and rationality, even though real attackers and defenders often face partial and asymmetric information (Alavizadeh et al., 2021). Transition probabilities are also derived from CVSS alone, while exploit success in practice depends on additional factors such as network configuration, code hardening, and zero-days (Alavizadeh et al., 2021). These are not incidental caveats; they delimit the scope of interpretability of equilibrium outputs.

Moving beyond full observability leads to one-sided partially observable stochastic games. In such models, the defender maintains a belief $A$08, the attacker may observe the true state, and the value function is defined on belief space (Horák et al., 2020). The associated Bellman operator is a $A$09-contraction on bounded convex continuous functions, exact value iteration can be performed using piecewise linear and convex representations, and heuristic-search value iteration has been shown to solve non-trivial patrolling, pursuit-evasion, and intrusion games (Horák et al., 2020). Related work on one-sided zero-sum partially observable stochastic games with public actions studies Stackelberg equilibrium rather than simultaneous-move minimax equilibrium and proves the existence of an $A$10-Stackelberg equilibrium for finite-horizon settings via belief-space partitioning (Zheng et al., 2021). This indicates that the classical fully observed zero-sum Markov security game is only one point in a wider design space of adversarial security models.

Another axis of extension concerns coordination and decentralization. Decentralized Q-learning in zero-sum Markov games removes the requirement that agents observe the opponent’s actions or even be aware of the opponent’s existence, a setting described as radically uncoupled (Sayin et al., 2021). FM3Q generalizes to two-team zero-sum games and seeks deterministic decentralized minimax policies via factorization of a joint minimax $A$11-function (Hu et al., 2024). These results are not themselves cyber-defense formulations, but they are methodologically relevant for security systems involving multiple defenders or coordinated multi-agent attacks.

A common misconception is that a zero-sum assumption automatically means a security model is unrealistic. The literature instead treats zero-sum as a deliberately conservative worst-case abstraction: it is appropriate when the defender optimizes against an attacker whose gains are identified with defender loss, and it yields robust min-max allocations of scarce monitoring or patrol resources (Chowdhary et al., 2018, Alavizadeh et al., 2021). Another misconception is that attack-graph models are inherently static. The stochastic-game formulations discussed here are explicitly dynamic: they represent multi-stage attacks, evolving privilege states, probabilistic detection, restoration, and sometimes partial observability (Chowdhary et al., 2018, Horák et al., 2020, Nguyen et al., 2010).

The practical takeaway stated in the shortest-path cloud model is that by quantifying both the attacker’s likelihood of success through exploitability and the value of each step through impact, a zero-sum Markov game provides a principled way to allocate scarce detection resources in a cloud environment, even against an intelligent, AI-enabled attacker (Alavizadeh et al., 2021). A plausible broader implication is that the enduring value of the zero-sum Markov security game lies not in any single instantiation, but in its role as a unifying interface between attack-graph semantics, vulnerability scoring, equilibrium computation, and increasingly general learning-based solution methods.