Game Generalized Policy Improvement (GGPI)

• GGPI is a transfer method for zero-sum, complete-information Markov games that leverages successor features and a minimax rule for robust policy improvement in new tasks.
• It stores policies and successor features from base tasks and reevaluates them under new reward vectors, enabling one-shot transfer without further optimization.
• The method offers theoretical guarantees with performance bounds that scale with task similarity and approximation error, ensuring effective transfer when tasks share dynamics.

Game Generalized Policy Improvement (GGPI) is a transfer method for zero-sum, complete-information, turn-based Markov games that combines successor features with a robust max–min decision rule. In the formulation introduced in "Successor Features for Transfer in Alternating Markov Games" (Amatya et al., 29 Jul 2025), GGPI addresses alternating Markov games with linear rewards, stores policies and successor features learned on base tasks, and performs one-shot transfer to a new task by re-evaluating stored successor features under a new reward vector and selecting actions through a pessimistic minimax aggregation. The method is designed for settings in which tasks share dynamics and feature structure but differ in reward weights, and it is accompanied by an error bound expressed in terms of task similarity and approximation error (Amatya et al., 29 Jul 2025).

1. Problem setting and conceptual motivation

GGPI is defined for a two-player, turn-based (alternating) Markov game with complete information. The state space is $S$, and at each state $s$ exactly one player has the move. The available actions are written $A_1(s)$ for player 1, the “agent,” and $A_2(s)$ for player 2, the “opponent.” The transition kernel is $P(s' \mid s,a,b)$. In the alternating setting, the next state encodes whose turn it is, so only the player whose turn it is chooses a nontrivial action, while the other player’s action is a dummy or no-op; nevertheless, the joint-action notation $(a,b)$ is retained because the standard zero-sum formulation with $Q(s,a,b)$ and max–min over $(a,b)$ remains convenient (Amatya et al., 29 Jul 2025).

The reward is zero-sum: player 1 receives $r_1(s,a,b)$ and player 2 receives $r_2(s,a,b)=-r_1(s,a,b)$. Across tasks, the reward is assumed linear in a fixed feature map $s$0. Each task is specified by a weight vector $s$1, with

$s$2

The motivation for GGPI is specific to transfer in games. Prior work in single-agent systems had shown that successor features can provide a “jump start” when agents face new tasks with different reward structures, whereas transfer in games had often relied on value transfer or equilibrium transfer. In the setting studied here, that reliance can fail when tasks differ significantly. GGPI addresses this by transferring both learning values and policies through successor features and a game-specific policy-improvement rule (Amatya et al., 29 Jul 2025).

2. Value functions, minimax structure, and successor features

For stationary policies $s$3 of the agent and opponent, the value and action-value functions for player 1 are

$s$4

and

$s$5

The minimax value is

$s$6

and the corresponding optimal $s$7-function satisfies the Shapley/Littman Bellman equations for zero-sum Markov games (Amatya et al., 29 Jul 2025).

Successor features decouple dynamics from rewards by accumulating discounted feature vectors along trajectories. In the alternating game setting, they are defined as

$s$8

Under linear rewards,

$s$9

and

$A_1(s)$0

The successor-feature recursion retains the same form as in the single-agent case, but the transition kernel and action sampling reflect turn-taking:

$A_1(s)$1

The opponent’s policy $A_1(s)$2 affects $A_1(s)$3 through the future action choices that determine feature visitation. This is the central structural reason successor features can be used for transfer in games rather than only in single-agent MDPs (Amatya et al., 29 Jul 2025).

3. The GGPI decision rule

The immediate precursor to GGPI is single-agent generalized policy improvement, which evaluates each stored successor-feature model on a new task and acts greedily with respect to the maximum over the corresponding action values. GGPI extends that idea to zero-sum games by making the transfer rule robust both to the opponent’s action and to the choice of policy in the stored library (Amatya et al., 29 Jul 2025).

Let $A_1(s)$4 be an approximation to the action-value function associated with stored policy $A_1(s)$5 under the current task or environment. GGPI defines the agent’s transfer policy by

$A_1(s)$6

The three nested operators have distinct roles:

• $A_1(s)$7: adversarial opponent action.
• $A_1(s)$8: pessimistic aggregation across the library to guard against approximation error or task mismatch.
• $A_1(s)$9: the agent chooses the action that maximizes this worst-case value.

When successor features are stored, transfer to a new task $A_2(s)$0 is performed by re-evaluating them as

$A_2(s)$1

and then applying the same GGPI rule. The ideal minimax value for a new task is

$A_2(s)$2

while GGPI approximates it by restricting the maximization to the stored library and using the pessimistic aggregation

$A_2(s)$3

A common misconception is to read GGPI as simply “max over policies in a game.” In the formulation analyzed here, the distinctive feature is the additional $A_2(s)$4 over the policy library. The paper explicitly uses this pessimistic construction in its theorem, and it is this construction that underpins the stated robustness and transfer bound (Amatya et al., 29 Jul 2025).

4. Learning procedure and one-shot transfer

The algorithmic pipeline has three stages: learning base tasks, storing policies and successor features, and one-shot transfer to a new task. Base tasks $A_2(s)$5 are selected to span the space of interest. For each task, the agent and opponent are trained in the alternating zero-sum game until convergence, for example by tabular $A_2(s)$6/successor-feature learning. In the reported implementation, both agents update temporal-difference errors asynchronously with a one-step lookahead into the other’s action (Amatya et al., 29 Jul 2025).

For each stored task, the algorithm keeps the agent’s policy $A_2(s)$7, optionally the opponent’s policy $A_2(s)$8, and the successor features $A_2(s)$9. The Bellman recursion is estimated in practice with temporal-difference updates:

$P(s' \mid s,a,b)$0

where $P(s' \mid s,a,b)$1 are sampled from $P(s' \mid s,a,b)$2 and $P(s' \mid s,a,b)$3 is sampled from $P(s' \mid s,a,b)$4. The paper also stores $P(s' \mid s,a,b)$5, and in the pursuer–evader experiments it uses successor-feature tables per task for both pursuer and evader (Amatya et al., 29 Jul 2025).

Transfer to a new task $P(s' \mid s,a,b)$6 is explicitly one-shot. At state $P(s' \mid s,a,b)$7, the acting player computes $P(s' \mid s,a,b)$8 for every stored policy and joint action, applies

$P(s' \mid s,a,b)$9

and executes the selected action. No additional optimization or fine-tuning is required: transfer consists of evaluating the stored successor features on $(a,b)$0 and applying the robust action selector.

The implementation details reported for the experimental setting are tabular. The method uses tabular successor-feature tables (“sf-table”) per task and per agent, which permits exact multiplication by $(a,b)$1. For larger problems, the paper notes that function approximation, including neural networks, can be used, but the reported experiments employ tables and asynchronous $(a,b)$2-updates with one-step opponent lookahead (Amatya et al., 29 Jul 2025).

5. Theoretical guarantees and transfer bound

The paper proves a variant of generalized policy improvement for games. The starting assumption is a uniform approximation bound: for each stored policy $(a,b)$3,

$(a,b)$4

for all states, actions, opponent actions, and library indices. Defining

$(a,b)$5

the GGPI policy is

$(a,b)$6

Two operator properties are used in the proof: the Bellman operator for zero-sum games is monotone, and it is a $(a,b)$7-contraction under suitable norms. Using these properties, the paper shows that repeated application of the GGPI operator yields an error bounded by

$(a,b)$8

in the limit, starting from the pointwise approximation error on $(a,b)$9 (Amatya et al., 29 Jul 2025).

The task-similarity result is stated as Proposition 1. Let $Q(s,a,b)$0 and $Q(s,a,b)$1 be two zero-sum Markov games in the class $Q(s,a,b)$2, meaning they share features and dynamics but have different weights $Q(s,a,b)$3. Define the maximum instantaneous reward difference

$Q(s,a,b)$4

If $Q(s,a,b)$5 is the optimal $Q(s,a,b)$6-function in task $Q(s,a,b)$7, $Q(s,a,b)$8 is the value of executing an optimal policy from task $Q(s,a,b)$9 in task $(a,b)$0, and the approximations satisfy the same uniform $(a,b)$1 bound, then the transfer performance obeys

$(a,b)$2

The interpretation given in the paper is direct: transfer error scales with task dissimilarity $(a,b)$3 and approximation error $(a,b)$4, both amplified by $(a,b)$5. When tasks are similar and the $(a,b)$6/successor-feature approximations are accurate, GGPI’s one-shot transfer is provably close to optimal; improvement over any single stored policy is guaranteed by the GGPI construction (Amatya et al., 29 Jul 2025).

6. Pursuer–evader evaluation

The empirical study is conducted in a turn-based gridworld pursuer–evader game. At each time step, one agent moves and the other agent’s action is effectively a no-op, but the joint-action form $(a,b)$7 is preserved for minimax evaluation. The feature vector $(a,b)$8 concatenates Manhattan distances between the evader and pursuer, Manhattan distances between the evader and each candidate goal, and terminal reward indicators for reaching each goal. Tasks differ by weights assigned to these distance and terminal-reward components; one example task vector reported in the paper is

$(a,b)$9

The reported discount factor is $r_1(s,a,b)$0, described as best performing in the experiments (Amatya et al., 29 Jul 2025).

The preliminary training result is structurally important. Base Task 1 was trained for 20,000 iterations, and without terminal rewards, transfer to Task 2 yielded no improvement in cumulative return. After adding terminal reward features and tuning weights, successful policy and reward transfer was obtained. The paper therefore identifies feature engineering—specifically the inclusion of terminal reward features—as critical for effective transfer (Amatya et al., 29 Jul 2025).

At transfer time, the successor-feature table from Task 1 is copied, re-evaluated with the new task weights, and GGPI initializes the policy. The action selector forms the $r_1(s,a,b)$1 tensor of $r_1(s,a,b)$2-values by multiplying stored successor-feature tables by $r_1(s,a,b)$3, then applies the max–min rule. Both players use a one-step lookahead to anticipate each other’s action (Amatya et al., 29 Jul 2025).

The reported results are qualitative. Introducing the terminal reward feature and appropriate weights leads to improved one-shot transfer, better cumulative returns, and effective goal-directed behavior in Task 2 across 9 different initial configurations. In a wider set of initial conditions, GGPI achieves higher success rates with improved path efficiency compared to the baseline algorithms. The paper also reports that transfer degrades as task dissimilarity increases, consistent with the $r_1(s,a,b)$4-dependent bound (Amatya et al., 29 Jul 2025).

7. Scope, limitations, and related uses of “game GPI”

GGPI, in the strict sense introduced for alternating Markov games, has a narrow scope. The assumptions are zero-sum, alternating, complete-information games; stationary policies; linear rewards of the form $r_1(s,a,b)$5; and uniform approximation error bounds over stored $r_1(s,a,b)$6-functions. The paper further notes that the $r_1(s,a,b)$7 aggregation is conservative, and that in practice one might prefer $r_1(s,a,b)$8 combined with $r_1(s,a,b)$9; however, the analyzed variant uses $r_2(s,a,b)=-r_1(s,a,b)$0 for robustness to approximation error and task shifts. Extensions to general-sum or simultaneous-move stochastic games would require adapting both the minimax structure and the successor-feature analysis (Amatya et al., 29 Jul 2025).

The term “GGPI” is also used more loosely in nearby literature, and distinguishing these usages avoids a common terminological confusion. In "Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards" (Nigam et al., 17 Oct 2025), the authors do not use the term GGPI, but they do implement GPI in a game or multi-agent context through

$r_2(s,a,b)=-r_1(s,a,b)$1

where transfer occurs across teammate-induced dynamics and no formal guarantee is claimed in the ad hoc setting. In "CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration" (Xiao et al., 2021), “game GPI” refers instead to generalized policy iteration in Atari, emphasizing gradient alignment and entropy-regularized improvement rather than successor-feature transfer. In "Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reuse" (Queeney et al., 2022), GPI denotes mixture-based sample reuse under trust-region control, and its extension to games is framed through fixed-opponent or slowly varying-opponent updates rather than through successor features.

These neighboring uses suggest that “Game Generalized Policy Improvement” names a family resemblance rather than a single universally fixed construction. In the encyclopedic sense, the most specific and technically distinctive usage is the one in alternating zero-sum Markov games: successor features under linear rewards, one-shot transfer through re-evaluation on $r_2(s,a,b)=-r_1(s,a,b)$2, and robust action selection by

$r_2(s,a,b)=-r_1(s,a,b)$3

That is the formulation for which explicit transfer bounds, implementation details, and pursuer–evader evidence are given (Amatya et al., 29 Jul 2025).