AlphaZero: Self-Play Reinforcement Learning

Updated 10 July 2026

AlphaZero is a self-play reinforcement-learning algorithm for perfect-information games that combines a deep neural network with Monte Carlo Tree Search.
It iteratively refines its policy by using search-improved action distributions from self-play as the training signal for neural network updates.
Engineering refinements and generalizations have extended its core search-coupled design to applications beyond traditional two-player board games.

AlphaZero is a self-play reinforcement-learning algorithm for two-player, deterministic, zero-sum, perfect-information games. In its canonical form, it learns from scratch, without human examples, by combining a deep neural network with Monte Carlo Tree Search (MCTS), and it has been associated with superhuman performance in chess, shogi, and Go (Liang et al., 2023, Petosa et al., 2019). Subsequent literature has treated AlphaZero both as a specific algorithmic instantiation and, more broadly, as a reusable recipe in which a learned policy-value model and search improve one another through iterative self-play.

1. Canonical formulation

AlphaZero treats a board game as a Markov decision process whose states are board positions and side to move, whose actions are legal moves, whose transition function applies a move to produce a new state, and whose rewards are final game outcomes such as win $+1$ , loss $-1$ , and draw $0$ (Liang et al., 2023). The central approximation is a neural network

$(\mathbf{p}, v) = f_\theta(s),$

where $\mathbf{p}$ is a policy vector over actions and $v \in [-1,1]$ is a scalar value estimate of the eventual outcome from the perspective of the current player (Liang et al., 2023).

Canonical descriptions in the literature use a shared trunk with two heads. The policy head produces move probabilities, while the value head produces a scalar evaluation. Input representations are typically stacks of planes encoding current and past board states together with side-to-move information, although later implementations adapt this representation to the structure of the target domain (Liang et al., 2023, Guo et al., 20 Apr 2025).

A persistent misconception is that AlphaZero is merely a neural-network evaluator. The defining architecture is instead search-coupled: the network is not used as a standalone policy, but as a prior and leaf evaluator inside MCTS, and the search output then becomes the supervision signal for the next round of network updates (Liang et al., 2023).

2. Search-coupled learning procedure

AlphaZero’s planning component is MCTS guided by the policy-value network. At a state $s$ , action selection during search follows a PUCT-style criterion

$a^* = \arg\max_a \left( Q(s,a) + U(s,a) \right),$

with

$U(s,a) = c_{\text{puct}}\, P(s,a)\, \frac{\sqrt{\sum_b N(s,b)}}{1 + N(s,a)},$

where $Q(s,a)$ is the current action-value estimate, $-1$ 0 the visit count, and $-1$ 1 the policy prior from the network (Liang et al., 2023). When search reaches a leaf state, the network evaluates it; the policy initializes child priors and the value replaces random rollouts as the leaf estimate (Liang et al., 2023).

Self-play generates the training set. At each position $-1$ 2, MCTS yields a search-improved policy $-1$ 3, usually derived from visit counts, and after a terminal result the game contributes tuples $-1$ 4 to the replay buffer, with $-1$ 5 the final outcome from the viewpoint of the player to move at $-1$ 6 (Liang et al., 2023). A standard training objective is

$-1$ 7

combining value regression, policy imitation of search, and regularization (Liang et al., 2023).

This search-and-distillation loop is the core of AlphaZero’s policy-improvement mechanism. The network sharpens search by supplying priors and value estimates; search sharpens the network by producing stronger action distributions than the raw policy alone. In practical systems, exploration is induced by stochastic move selection early in games and more deterministic choice later, while the exact temperature and noise schedules depend on the implementation (Liang et al., 2023).

3. Theoretical interpretations and evaluation criteria

One line of work analyzes AlphaZero through Shannon-style information theory. In the Unified Intelligence–Communication Model, two self-playing agents interact through an external channel given by the board and game rules, while each agent also maintains an internal world model that functions as an internal channel (Zhang et al., 2018). Within this framework, intelligence entropy is defined as the amount of information recovered by the agent from the environment, and intelligence capacity as the maximum amount of entropy extractable in a given environment and task (Zhang et al., 2018).

Applied to Go, this perspective gives the bound

$-1$ 8

where $-1$ 9 is interpreted as the channel capacity of the $0$0 board and hence an upper bound on the intelligence capacity available in that environment (Zhang et al., 2018). The same analysis interprets AlphaZero’s self-play and internal world-model refinement as a turbo-like iterative decoder, and proposes extrinsic information and EXIT charts as tools for diagnosing whether learning will approach a global optimum or stall at a local one (Zhang et al., 2018). This suggests an information-theoretic reading of AlphaZero as a capacity-approaching architecture, although that interpretation is explicitly conceptual rather than an empirical computation of AlphaZero’s exact “intelligence entropy” (Zhang et al., 2018).

A separate critique concerns reward design. Standard AlphaZero requires a quantitative reward metric, which forces explicit numerical tradeoffs whenever outcomes include multiple desiderata such as winner, margin, or game length. A modified self-play system replaces the fixed scalar reward by a total ordering over outcomes and constructs rewards through an empirical cumulative distribution function over observed outcomes; on a sample game, it learns optimal play in a comparable amount of time to AlphaZero while avoiding explicit quantitative balancing (Schmidt et al., 2019). This reframes AlphaZero’s reliance on scalar rewards as a design choice rather than a logical necessity.

Because AlphaZero is compute-intensive, especially in self-play, much of the literature concerns ways to improve strength, efficiency, or reproducibility without changing the high-level recipe. Population Based Training (PBT) is one such refinement: on 19×19 Go, a PBT-trained agent reached up to a 74% win rate against ELF OpenGo, compared with 47% for a saturated non-PBT agent under the same circumstances, while also avoiding the need for multiple independent full training runs for static hyperparameter sweeps (Wu et al., 2020).

Search itself has also been modified. Warm-start self-play search enhancements replace the purely cold-start regime by rollout, RAVE, and dynamically weighted combinations with the neural network during the beginning phase of training; across three small board games, most variants improved the baseline player, with especially RAVE-based variants playing strongly (Wang et al., 2020). Monte-Carlo Graph Search generalizes AlphaZero’s search tree to a directed acyclic graph, enabling information flow across different subtrees and reducing memory consumption by 30–70% in the reported experiments (Czech et al., 2020).

A parallel line of work focuses on accessibility. AlphaZero-Edu is a lightweight implementation designed for transparent visualization and single-GPU training; it runs on one NVIDIA RTX 3090 and reports a 3.2-fold self-play speedup with 8 processes, together with a consistently high win rate against human opponents in Gomoku (Guo et al., 20 Apr 2025). At the opposite end of the design spectrum, an AlphaZero-inspired system can omit training-time MCTS entirely and use MCTS only at test time around a TD-trained value function; in Othello, this produced what was reported as the first agent trained on standard hardware to beat Edax up to and including level 7 (Scheiermann et al., 2022).

Not all AlphaZero-style work preserves the original training objective. AlphaZeroES keeps the MCTS algorithm and neural architecture unchanged but replaces planning-loss minimization with direct score maximization via evolution strategies; in single-agent environments, this direct objective outperformed minimizing the usual value-and-policy planning loss (Martin et al., 2024). Taken together, these results show that “AlphaZero-style” research has diversified along both axes: search engineering and optimization objective.

5. Generalizations beyond alternating two-player board games

The literature has repeatedly extended AlphaZero beyond its canonical setting.

Extension	Setting	Principal modification
Multiplayer AlphaZero	3-player deterministic games	Vector value head and max$0$1-style search
Simultaneous AlphaZero	Simultaneous-action Markov games	Matrix-game solving at each node; two policy heads
Search-problem adaptation	SAT and other search problems	Self-reductions as moves; easy-instance solvers as terminals
AlphaZero-UVR	Post-storm crew dispatch	Belief-state planning with stochastic MCTS
Grid topology optimization	Power-grid congestion control	Topology-action search plus learned policy priors

In multiplayer settings, the scalar value head is replaced by a vector $0$2, terminal outcomes become score vectors, and MCTS is modified into an MCTS-max$0$3 procedure in which each player maximizes its own component of the value vector at its turn; on two simple 3-player games, this modified AlphaZero consistently outperformed plain MCTS (Petosa et al., 2019). For simultaneous-action two-player zero-sum deterministic Markov games, Simultaneous AlphaZero models each state as a matrix game over joint actions, uses two policy heads and a distributional value head, and solves local matrix games with a regret-optimal solver under bandit feedback during search (Becker et al., 13 Dec 2025).

Other work departs even further from alternating games. An AlphaZero-inspired approach to SAT and general search problems treats problem instances as states, self-reductions as moves, and easy-instance solvers as terminal evaluators; the resulting system uses an MCTS-like procedure over a symbolic state-transition system rather than over game positions (Dantsin et al., 2022). In post-storm repair crew dispatch, AlphaZero-UVR formulates routing as a sequential stochastic optimization problem with belief-state updates from customer calls and vehicle observations, and combines a neural network with stochastic MCTS to provide lookahead crew-navigation decisions (Shuai et al., 2020). In power systems, an AlphaZero-based topology optimization agent for congestion management uses topology actions as the learned action space and achieved a 60% reduction in the average amount of required redispatching while ranking first in the WCCI 2022 Learning to Run a Power Network competition (Dorfer et al., 2022).

These extensions imply that the core AlphaZero abstraction is not restricted to alternating-move board games. What persists is the planning architecture: learned priors and state evaluation embedded inside search, plus iterative policy improvement from self-generated trajectories.

6. Applications, empirical reach, and research significance

AlphaZero’s empirical reach within games has broadened rapidly. In Gomoku, an AlphaZero-style agent trained from random play and without domain knowledge apart from the rules learned a winning strategy on small boards within a few hours on a commodity GPU, achieved a 100% victory rate as the initiating player in self-play on 6×6 four-in-a-row, and outperformed a pure MCTS baseline across tested simulation counts from 500 to 2500 (Liang et al., 2023). In chess-variant analysis, separately trained AlphaZero agents have been used as instruments for rule evaluation: for 1-second-per-move self-play, torpedo chess showed a White expected score of 56.8% and a draw rate of 71.9%, versus 51.8% and 88.2% in classical chess, respectively (Schulz et al., 2020).

Outside games, AlphaZero-style planning has been used as an in silico design and discovery tool. In quantum control, a tabula rasa AlphaZero variant was applied to three classes of control problems using a single common set of algorithmic hyperparameters, achieving substantial improvements in both the quality and quantity of good solution clusters compared with earlier methods, and learning hidden structure and global symmetry in the solutions (Dalgaard et al., 2019). In power-grid operation and restoration, AlphaZero-style agents have been used for topology optimization and repair dispatch, with the appeal that topology actions are non-costly and carbon-free while redispatching is costly and often emissions-intensive (Dorfer et al., 2022, Shuai et al., 2020).

A recurring misconception is that AlphaZero is identical with a single frozen implementation. The broader literature suggests otherwise. The canonical algorithm remains the reference point—policy-value network, MCTS, self-play, and search-imitation loss—but many later systems alter the value representation, backup rule, optimization objective, or even the ontology of “state” and “action.” This suggests that AlphaZero now functions in research practice as both a named algorithm and a general planning-and-learning template for domains where execution-time search and self-generated supervision are jointly advantageous.