Reevaluating Policy Gradient Methods for Imperfect-Information Games

Abstract: In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR). In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP, DO, and CFR-based DRL approaches. To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for four large games. Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-information games. Over 5600 training runs, FP, DO, and CFR-based approaches fail to outperform generic policy gradient methods. Code is available at https://github.com/nathanlct/IIG-RL-Benchmark and https://github.com/gabrfarina/exp-a-spiel .

• The paper shows that well-tuned generic PG methods match or outperform traditional FP, DO, and CFR approaches using exploitability metrics.
• It leverages over 5600 training runs across four benchmark games to compare DRL strategies in two-player zero-sum imperfect-information settings.
• The study highlights the need for higher entropy regularization than typical in single-agent problems to optimize hyperparameter configurations.

The paper "Reevaluating Policy Gradient Methods for Imperfect-Information Games" posits that generic policy gradient (PG) methods can achieve results on par with or exceeding those of algorithms rooted in fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR) when applied to imperfect-information games (IIGs).

The central hypothesis, referred to as the Policy Gradient Hypothesis, suggests that appropriately tuned generic PG methods are competitive with or superior to model-free deep reinforcement learning (DRL) approaches based on FP, DO, or CFR in two-player zero-sum IIGs. This challenges the prevalent belief that algorithms with established equilibrium convergence guarantees are essential for effective DRL in such games.

To evaluate this hypothesis, the authors implemented exact exploitability computations for four games: 3x3 Dark Hex, Phantom Tic-Tac-Toe, 3x3 Abrupt Dark Hex, and Abrupt Phantom Tic-Tac-Toe. They then performed an exploitability comparison of DRL algorithms for IIGs. The algorithms considered were NFSP, PSRO, ESCHER, R-NaD, MMD, PPO, and PPG.

The results of over 5600 training runs revealed that FP, DO, and CFR-based approaches failed to surpass generic PG methods in terms of exploitability. The authors also found that a larger entropy regularization than is typically used for PG methods in single-agent problems may be required to achieve effective hyperparameter configurations.

Here's a breakdown of the key components and findings:

• Game Formalism: The paper formalizes IIGs as perfect-recall extensive-form games (EFGs), defining the game as a tuple $$\langle \mathbb{S}, \mathbb{A}, \mathbb{O}, \mathbb{I}, \mathcal{R}, \mathcal{T}, \mathcal{O}, \mathcal{C}, t_{\text{max}} \rangle$$, where:
  • $\mathbb{S}$ is the space of game states
  • $\mathbb{A}$ is the space of actions
  • $\mathbb{O}$ is the space of observations
  • $$\mathbb{I} = \cup_t (\mathbb{O} \times \mathbb{A})^t \times \mathbb{O}$$ is the space of information states
  • $$\mathcal{T} \colon \mathbb{S} \times \mathbb{A} \to \Delta(\mathbb{S})$$ is the transition function
  • $$\mathcal{R} \colon \mathbb{S} \times \mathbb{A} \to \mathbb{R}$$ is the reward function for player 1
  • $\mathcal{O} \colon \mathbb{S} \to \mathbb{O}$ is the observation function
  • $\mathcal{C} \colon \mathbb{S} \to \{1, 2\}$ is the player choice function
  • $t_{\text{max}}$ is the time step after which the game terminates
• Exploitability Metric: The paper uses exploitability, defined as $\mathbb{S}$0, as the primary metric for evaluating the performance of different algorithms.
  • $\mathbb{S}$1 is policy
  • $\mathbb{S}$2 is the expected return
• Algorithmic Approaches: The paper discusses various algorithmic approaches to DRL in two-player zero-sum IIGs, including:
  • Naive self-play
  • Best Responses: FP and DO
  • CFR
  • Regularization: MMD
• Magnetic Mirror Descent (MMD): The update rule is defined as:

  $\mathbb{S}$3

  • $\mathbb{S}$4 is the value of action $\mathbb{S}$5
  • $\mathbb{S}$6 is a regularization temperature
  • $\mathbb{S}$7 is a stepsize
  • $\mathbb{S}$8 is a magnet policy
  • $\mathbb{S}$9 is KL divergence
• Benchmarks: The paper introduces four benchmark games: Phantom Tic-Tac-Toe (PTTT), 3x3 Dark Hex (DH3), Abrupt Phantom Tic-Tac-Toe (APTTT), and 3x3 Abrupt Dark Hex (ADH3). These games have $\mathbb{A}$0s of billions of game states.
• Experimental Design: The training runs consisted of a hyperparameter tuning launch followed by a maximization-bias-free evaluation launch.
  • For the hyperparameter tuning launch, each combination of the 4 games and 7 algorithms was tested across 50 hyperparameter configurations for 10 million steps over 3 seeds.
  • For the evaluation launch, the 5 strongest hyperparameter configurations from the first launch were run with 10 fresh seeds for 10 million steps, for each combination of the 4 games and 7 algorithms.
• Key Findings: The experimental results indicated that generic PG methods (MMD, PPO, PPG) either matched or outperformed other algorithms while maintaining rough parity among themselves. R-NaD either approached the performance of generic PG methods or underperformed them, while NFSP and, especially, ESCHER and PSRO were non-competitive.
• Entropy Coefficient Analysis: The analysis suggests that the entropy coefficients with the best average performance are between 0.05 and 0.2, larger than any of the default entropy coefficients for PPO in Stable Baselines, CleanRL, RLlib, OpenSpiel, PufferLib, RL-Games, and Tianshou.

The authors release their exploitability computations in an accessible Python package with OpenSpiel-compatible bindings, called exp-a-spiel, to enable community efforts in the field of DRL algorithms for IIGs.