Strategy Improvement Algorithm

• Strategy Improvement Algorithm is an iterative method that optimizes graph-based games by sequentially updating strategies using locally profitable switches.
• It employs positional determinacy and best-response computations to progressively enhance strategy valuations until reaching optimality.
• Variants like symmetric and non-oblivious improvements refine performance and reveal deep connections with combinatorial optimization and simplex pivoting.

A strategy improvement algorithm, also called strategy iteration or policy iteration, is an iterative local-search method for solving graph-based infinite-duration games and related stochastic games. In its standard form, one fixes a positional strategy for one player, computes an optimal counterstrategy or best response for the opponent, evaluates the induced game, switches profitable local choices, and repeats until no profitable switch remains. This scheme is central in parity games, mean-payoff games, discounted-payoff games, simple stochastic games, and concurrent reachability or safety games, where positional determinacy or memoryless optimality makes local improvement meaningful (Schewe et al., 2015, Auger et al., 2021, Chatterjee et al., 2012).

1. Formal setting and basic mechanism

A general turn-based formulation uses a graph game $\mathcal G=(\mathcal A,\eta,\prec)$ with arena $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$, where the arena induces infinite plays, $\eta$ evaluates the resulting color sequence, and $\prec$ is the preference order. A strategy for Player Max is a function $\sigma:V^*V_{\max}\to V$, and similarly $\tau:V^*V_{\min}\to V$ for Player Min. For every strategy pair $(\sigma,\tau)$, the induced play from $v$ is unique, and the corresponding value is

$$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$

The value of a fixed strategy against all opponents is then

$$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$

For positionally determined games, one assumes optimal positional strategies and optimal positional counterstrategies $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$0 and $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$1 (Schewe et al., 2015).

The classical strategy-improvement loop is asymmetric. For a current Max strategy $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$2, one determines Min’s globally optimal counterstrategy $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$3, evaluates the game under $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$4, identifies profitable local switches, and applies some subset of them. A switch $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$5 is profitable if changing only $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$6 to $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$7 yields a strictly better valuation,

$\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$8

This produces a monotone search over the finite space of positional strategies (Schewe et al., 2015).

A common abstraction is that a game class is good for strategy improvement if it is positionally determined, has combinable profitable updates, and is maximum identifying: $\mathcal A=(V_{\max}\cup V_{\min},E,C,\phi)$9 Under these conditions, strict valuation improvement and finiteness of positional strategies imply termination at an optimal strategy (Schewe et al., 2015).

In simple stochastic games, the same logic appears in value-vector form. For a MAX strategy $\eta$0, the switch set is

$\eta$1

and a strategy is optimal iff its switch set is empty. Any $\eta$2-switch strictly increases the value vector, which is the basic monotonicity principle behind recursive and generic strategy-improvement algorithms for SSGs (Montjoye, 2021, Auger et al., 2021).

2. Valuations, best responses, and switching rules

The effectiveness of strategy improvement depends on a valuation scheme that refines the raw win/lose objective. In the discrete parity-game algorithm of Vöge and Jurdziński, a node valuation is a triple

$\eta$3

where $\eta$4 is the dominating cycle node, $\eta$5 is the set of more relevant nodes on the path, and $\eta$6 is the path length. The induced total ordering on valuations allows the algorithm to compare local successors and define an improvement arena

$\eta$7

where only edges that are not worse than the current choice are kept (0901.2731).

Different switching rules determine how profitable edges are combined. The locally optimizing or switch-all rule performs the best local switch at every improvable vertex. The globally optimizing or switch-best rule takes cross-effects of switches into account. Other rules studied in the literature include random-edge, switch-half, random-facet, least-recently-considered, least-entered, and Zadeh’s pivoting rule (0901.2731, &&&10&&&). In all of these cases, the per-iteration work is polynomial-time computable; the central issue is therefore the number of iterations rather than the local step itself (Canavoi et al., 2012).

A distinct valuation framework appears in the non-deterministic strategy-iteration approach for parity games. There the value space is

$\eta$8

finite profiles count occurrences of each color along a play, and the total order $\eta$9 is chosen so that player $\prec$0 prefers even colors to occur more often and odd colors less often. For a reasonable strategy $\prec$1, the valuation $\prec$2 is the least fixed point of a Bellman–Ford-style operator $\prec$3, and improving edges are

$\prec$4

This representation makes “all profitable switches” literally mean $\prec$5 (0806.2923).

Best-response computation is often the practical bottleneck. In the parity-game implementation studied in parallel strategy improvement, fixing Even’s strategy reduces Odd’s task to a shortest-path-like computation with possibly negative contributions from odd priorities. The standard baseline is Bellman–Ford, but one-player strategy improvement can be used instead: an Odd edge $\prec$6 is switchable if it improves the current valuation, and once no Odd-switchable edges remain, the current strategy satisfies the Bellman optimality equations and is a best response. Experimentally, Bellman-Ford was reported to be 8.43× slower than one-player strategy improvement and to use 5.30× more iterations on average (Fearnley, 2017).

3. Correctness, convergence, and generic frameworks

The standard correctness argument has two parts. First, every accepted switch must improve the relevant valuation monotonically. Second, when no profitable switch exists, the current strategy must already be optimal. For turn-based graph games that are good for strategy improvement, this suffices for global correctness, since the strategy space is finite and valuations strictly improve until a fixed point is reached (Schewe et al., 2015).

A more general meta-theoretic formulation is given by the Generic Strategy Improvement Algorithm (GSIA) for simple stochastic games. Given an SSG $\prec$7, a subset of arcs $\prec$8, and a strategy $\prec$9, the transformed game $\sigma:V^*V_{\max}\to V$0 replaces each arc $\sigma:V^*V_{\max}\to V$1 by a new sink $\sigma:V^*V_{\max}\to V$2 with value $\sigma:V^*V_{\max}\to V$3. GSIA chooses $\sigma:V^*V_{\max}\to V$4 such that

$\sigma:V^*V_{\max}\to V$5

and then proves that this implies genuine improvement in the original game. The central transfer facts are that $\sigma:V^*V_{\max}\to V$6 is optimal in $\sigma:V^*V_{\max}\to V$7 iff it is optimal in $\sigma:V^*V_{\max}\to V$8, and that improvement in the transformed game implies improvement in the original one. This yields termination and optimality without requiring the game to be stopping (Auger et al., 2021).

For $\sigma:V^*V_{\max}\to V$9-SSGs with $\tau:V^*V_{\min}\to V$0 random vertices, GSIA also gives a parameterized complexity statement. The paper proves a tight denominator bound: for any pair of strategies $\tau:V^*V_{\min}\to V$1, there exists $\tau:V^*V_{\min}\to V$2 such that each value is of the form $\tau:V^*V_{\min}\to V$3. From this, together with switch monotonicity, it derives that GSIA makes at most

$\tau:V^*V_{\min}\to V$4

iterations. A plausible implication is that strategy iteration for SSGs is fixed-parameter tractable in the number of random vertices, at least at the level captured by this generic framework (Auger et al., 2021).

Recursive SSG algorithms sharpen the convergence analysis further. For degree-$\tau:V^*V_{\min}\to V$5 SSGs, RecursivePair obtains an iteration bound

$\tau:V^*V_{\min}\to V$6

while for binary SSGs, DecreasingFixedSet achieves

$\tau:V^*V_{\min}\to V$7

The latter is stated as the first deterministic strategy improvement algorithm in this setting that visits $\tau:V^*V_{\min}\to V$8 strategies with $\tau:V^*V_{\min}\to V$9 (Montjoye, 2021).

4. Lower bounds and complexity barriers

The modern theory of strategy improvement is shaped by strong lower bounds. For parity games, Friedmann constructed a quadratic-size family $(\sigma,\tau)$0 with deceleration lanes and stubborn cycles and proved that the discrete strategy-improvement algorithm with the locally optimizing policy requires

$(\sigma,\tau)$1

iterations. Since $(\sigma,\tau)$2, this gives a super-polynomial worst-case lower bound. The construction works by simulating an $(\sigma,\tau)$3-bit binary counter: a closed stubborn cycle represents a bit set to $(\sigma,\tau)$4, an open one represents $(\sigma,\tau)$5, and the deceleration lane stretches each increment over many iterations (0901.2731).

These lower bounds are structurally stronger than a purely worst-case statement on arbitrary graphs. Friedmann’s counterexamples remain hard even when analyzed by graph-complexity measures for which parity games are otherwise polynomial-time solvable. For the switch-all counterexamples $(\sigma,\tau)$6, the paper proves

$(\sigma,\tau)$7

This means that super-polynomial behavior persists on graph classes of very small directed pathwidth, DAG-width, Kelly-width, entanglement, and cliquewidth (Canavoi et al., 2012).

The computational complexity of the trajectory itself can also be hard. For greedy all-switches strategy improvement, the EdgeSwitch problem—whether a designated edge is ever switched—and the OptimalStrategy problem—whether a designated edge appears in the final strategy—are both $(\sigma,\tau)$8-complete for parity games, mean-payoff games, discounted-payoff games, and simple stochastic games. The reductions simulate iterated Boolean circuit evaluation by a carefully synchronized family of parity-game gadgets (Fearnley et al., 2015).

Symmetric strategy improvement does not escape worst-case barriers either. Although the 2015 symmetric algorithm solves Friedmann’s known worst-case examples quickly, a later lower-bound construction shows that symmetric strategy improvement takes exponentially many steps in the worst case for parity, mean-payoff, and discounted-payoff games, independently of the improvement rule. On the basic hard family, the recurrence

$(\sigma,\tau)$9

gives

$v$0

A generalized symmetric variant, which relaxes the dependence on the opponent’s exact counterstrategy, still requires

$v$1

iterations on a modified family (Dijk et al., 2023).

5. Major variants and refinements

A central refinement is symmetric strategy improvement. Rather than improving one player against a fixed best response, it computes optimal counterstrategies for both sides and updates them simultaneously, but only along moves compatible with the opponent’s current optimal counterstrategy: $v$2 The sequence of Max strategies has nondecreasing value,

$v$3

and similarly Min improves in the opposite direction. At termination,

$v$4

so both strategies are globally optimal. The method was introduced precisely because the naïve simultaneous-update idea can cycle, whereas the intersection with the opponent’s counterstrategy restores monotone improvement (Schewe et al., 2015).

Another influential refinement is non-oblivious strategy improvement. Here the algorithm records structures called snares, where a pair $v$5 consists of a vertex set $v$6 and a partial Max strategy $v$7 that is winning for Max on the subgame $v$8. The key structural fact is that any winning Min strategy on a set containing a Max snare must use an escape edge from that snare. Profitable back edges in a strategy tree expose snares, and if the current strategy is inconsistent with a recorded snare then some profitable edge agrees with the snare strategy. The procedure FixSnare repeatedly switches such edges until Min’s best response uses an escape from the snare; it reaches consistency in at most $v$9 steps and yields polynomial-time behavior on Friedmann’s hard examples (Fearnley, 2010).

The non-deterministic variant extends memoryless strategies from single chosen successors to non-empty subsets of outgoing edges. In this setting, direct improvements satisfy $$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$0, the update $$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$1 is locally optimal among direct improvements, and for out-degree $$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$2 the number of iterations under all-profitable-switches is bounded by

$$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$3

The paper presents this as a direct parity-game generalization of earlier deterministic schemes and as an alternative route to bounds previously known only through randomization (0806.2923).

Several refinements address locality or implementation cost rather than asymptotic worst case. Local strategy improvement for parity games maintains two partial local views $$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$4 and $$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$5, expands them on demand, and alternates the two players so that only the subgraph needed to decide a queried initial node is explored. Empirically, it can outperform global strategy improvement and the previously known local model-checking approach by orders of magnitude when the witness strategy occupies only a small fraction of the graph (Friedmann et al., 2010). Parallel strategy improvement, by contrast, targets step complexity: valuations are reduced to list ranking through an Euler-tour construction, best responses are computed by one-player strategy improvement rather than Bellman-Ford, and a GPU implementation yields an average speedup of 10.37 over the best CPU implementation, rising to 12.17 when instances below 1 million edges are excluded (Fearnley, 2017).

Strategy-improvement ideas have also been adapted to quantitative objectives beyond parity winning conditions. KASI (“Keep Alive Strategy Improvement”) solves the lower-weak-upper-bound problem in mean-payoff games, computes minimum sufficient initial energies, alternates exact evaluation of the current Min strategy with local strategy improvement on Min’s edges, and terminates with

$$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$6

Its basic pseudopolynomial bound is

$$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$7

with a refined bound

$$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$8

and the experiments reported it as the fastest tested algorithm for the lower-bound problem (Brim et al., 2010).

6. Extensions, related algorithms, and structural interpretations

In concurrent games, strategy improvement is no longer a purely turn-based best-response procedure, but the same monotone template survives through predecessor operators. For a valuation $$val_{\mathcal G}(v,\sigma,\tau)=\eta\big(\phi(\pi(v,\sigma,\tau))\big).$$9,

$$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$0

and one defines

$$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$1

For concurrent reachability games, the algorithm yields a monotone sequence of lower bounds converging to the value. For concurrent safety games, local $$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$2-improvement alone is not sufficient, so the algorithm adds a turn-based reduction $$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$3; this is described as the first strategy improvement algorithm for concurrent safety games and the first practical method for converging lower bounds in that setting (0804.4530, Chatterjee et al., 2012).

For stochastic parity objectives, a notable hybridization appears in 2.5-player parity games. There, profitable switches are defined by strict increases in the value

$$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$4

but when no profitable switch exists the algorithm examines neutral edges

$$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$5

forms the neutral subgame, and uses a recursive qualitative parity solver to enlarge the almost-sure winning region if possible. This synthesizes strategy improvement for the quantitative part with recursive parity solving for the qualitative part, and was reported to handle games with several million states (Hahn et al., 2016).

The relation between strategy improvement and other parity-game algorithms is not merely competitive; it is also comparative. A modification of Small Progress Measures derives winning strategies for both players in one pass, but the paper explicitly states that this is not a new general-purpose strategy-improvement framework. Rather, it is an SPM-based strategy derivation method whose operational interpretation of the least progress measure makes it resemble strategy-improvement valuations (Gazda et al., 2015).

A more recent structural development establishes a direct formal connection with the simplex algorithm. For nondegenerate longest-shortest-path problems, admissible Player $$val_{\mathcal G}(v,\sigma)=\inf_{\tau\in T^\infty}val_{\mathcal G}(v,\sigma,\tau),\qquad val_{\mathcal G}(v,\tau)=\sup_{\sigma\in \Sigma^\infty}val_{\mathcal G}(v,\sigma,\tau).$$6 strategies correspond exactly to feasible bases of a linear program, basis inverses are given by walk polynomials, and a nonbasic edge has negative reduced cost iff it is an improving move. Under this translation, one-switch-at-a-time strategy improvement for longest shortest path problems, mean-payoff games, and parity games is literally an instance of simplex pivoting. The same framework implies that, when each Maximizer node has at most two outgoing edges, the set of admissible strategies is lopsided (Maat, 19 Sep 2025).

Taken together, these results show that “strategy improvement algorithm” denotes not a single procedure but a technically rich family of monotone local-search methods. Its core invariant is always the same—improving a current strategy by reference to a valuation or best response—but its behavior depends sharply on the valuation domain, the switching rule, the structural information retained across iterations, and the game model under study. The resulting theory spans elegant correctness proofs, strong negative complexity results, substantial practical accelerations, and deep links to linear programming and combinatorial geometry.