---
title: 'AlphaZero: Self-Play Reinforcement Learning'
url: https://www.emergentmind.com/topics/alphazero
type: topic
---

# AlphaZero: Self-Play Reinforcement Learning

AlphaZero is a self-play reinforcement-learning algorithm for two-player, deterministic, zero-sum, perfect-information games. In its canonical form, it learns from scratch, without human examples, by combining a deep neural network with Monte Carlo Tree Search (MCTS), and it has been associated with superhuman performance in chess, shogi, and Go [2309.01294] [1910.13012]. Subsequent literature has treated AlphaZero both as a specific algorithmic instantiation and, more broadly, as a reusable recipe in which a learned policy-value model and search improve one another through iterative self-play.

## 1. Canonical formulation

AlphaZero treats a board game as a Markov decision process whose states are board positions and side to move, whose actions are legal moves, whose transition function applies a move to produce a new state, and whose rewards are final game outcomes such as win \(+1\), loss \(-1\), and draw \(0\) [2309.01294]. The central approximation is a neural network
\[
(\mathbf{p}, v) = f_\theta(s),
\]
where \(\mathbf{p}\) is a policy vector over actions and \(v \in [-1,1]\) is a scalar value estimate of the eventual outcome from the perspective of the current player [2309.01294].

Canonical descriptions in the literature use a shared trunk with two heads. The policy head produces move probabilities, while the value head produces a scalar evaluation. Input representations are typically stacks of planes encoding current and past board states together with side-to-move information, although later implementations adapt this representation to the structure of the target domain [2309.01294] [2504.14636].

A persistent misconception is that AlphaZero is merely a neural-network evaluator. The defining architecture is instead search-coupled: the network is not used as a standalone policy, but as a prior and leaf evaluator inside MCTS, and the search output then becomes the supervision signal for the next round of network updates [2309.01294].

## 2. Search-coupled learning procedure

AlphaZero’s planning component is MCTS guided by the policy-value network. At a state \(s\), action selection during search follows a PUCT-style criterion
\[
a^* = \arg\max_a \left( Q(s,a) + U(s,a) \right),
\]
with
\[
U(s,a) = c_{\text{puct}}\, P(s,a)\, \frac{\sqrt{\sum_b N(s,b)}}{1 + N(s,a)},
\]
where \(Q(s,a)\) is the current action-value estimate, \(N(s,a)\) the visit count, and \(P(s,a)\) the policy prior from the network [2309.01294]. When search reaches a leaf state, the network evaluates it; the policy initializes child priors and the value replaces random rollouts as the leaf estimate [2309.01294].

Self-play generates the training set. At each position \(s_t\), MCTS yields a search-improved policy \(\boldsymbol{\pi}_t\), usually derived from visit counts, and after a terminal result the game contributes tuples \((s_t,\boldsymbol{\pi}_t,z_t)\) to the replay buffer, with \(z_t\) the final outcome from the viewpoint of the player to move at \(t\) [2309.01294]. A standard training objective is
\[
l(\theta) = (z - v)^2 - \boldsymbol{\pi}^\top \log \mathbf{p} + c\|\theta\|^2,
\]
combining value regression, policy imitation of search, and regularization [2309.01294].

This search-and-distillation loop is the core of AlphaZero’s policy-improvement mechanism. The network sharpens search by supplying priors and value estimates; search sharpens the network by producing stronger action distributions than the raw policy alone. In practical systems, exploration is induced by stochastic move selection early in games and more deterministic choice later, while the exact temperature and noise schedules depend on the implementation [2309.01294].

## 3. Theoretical interpretations and evaluation criteria

One line of work analyzes AlphaZero through Shannon-style information theory. In the Unified Intelligence–Communication Model, two self-playing agents interact through an external channel given by the board and game rules, while each agent also maintains an internal world model that functions as an internal channel [1812.05794]. Within this framework, **intelligence entropy** is defined as the amount of information recovered by the agent from the environment, and **intelligence capacity** as the maximum amount of entropy extractable in a given environment and task [1812.05794].

Applied to Go, this perspective gives the bound
\[
\max(I_{B-A}, I_{A-B}) \le C \le \log_2(361!) \approx 2552,
\]
where \(C\) is interpreted as the channel capacity of the \(19 \times 19\) board and hence an upper bound on the intelligence capacity available in that environment [1812.05794]. The same analysis interprets AlphaZero’s self-play and internal world-model refinement as a turbo-like iterative decoder, and proposes extrinsic information and EXIT charts as tools for diagnosing whether learning will approach a global optimum or stall at a local one [1812.05794]. This suggests an information-theoretic reading of AlphaZero as a capacity-approaching architecture, although that interpretation is explicitly conceptual rather than an empirical computation of AlphaZero’s exact “intelligence entropy” [1812.05794].

A separate critique concerns reward design. Standard AlphaZero requires a quantitative reward metric, which forces explicit numerical tradeoffs whenever outcomes include multiple desiderata such as winner, margin, or game length. A modified self-play system replaces the fixed scalar reward by a total ordering over outcomes and constructs rewards through an empirical cumulative distribution function over observed outcomes; on a sample game, it learns optimal play in a comparable amount of time to AlphaZero while avoiding explicit quantitative balancing [1912.07557]. This reframes AlphaZero’s reliance on scalar rewards as a design choice rather than a logical necessity.

## 4. Engineering refinements and alternative training regimes

Because AlphaZero is compute-intensive, especially in self-play, much of the literature concerns ways to improve strength, efficiency, or reproducibility without changing the high-level recipe. Population Based Training (PBT) is one such refinement: on 19×19 Go, a PBT-trained agent reached up to a 74% win rate against ELF OpenGo, compared with 47% for a saturated non-PBT agent under the same circumstances, while also avoiding the need for multiple independent full training runs for static hyperparameter sweeps [2003.06212].

Search itself has also been modified. Warm-start self-play search enhancements replace the purely cold-start regime by rollout, RAVE, and dynamically weighted combinations with the neural network during the beginning phase of training; across three small board games, most variants improved the baseline player, with especially RAVE-based variants playing strongly [2004.12357]. Monte-Carlo Graph Search generalizes AlphaZero’s search tree to a directed acyclic graph, enabling information flow across different subtrees and reducing memory consumption by 30–70% in the reported experiments [2012.11045].

A parallel line of work focuses on accessibility. AlphaZero-Edu is a lightweight implementation designed for transparent visualization and single-GPU training; it runs on one NVIDIA RTX 3090 and reports a 3.2-fold self-play speedup with 8 processes, together with a consistently high win rate against human opponents in Gomoku [2504.14636]. At the opposite end of the design spectrum, an AlphaZero-inspired system can omit training-time MCTS entirely and use MCTS only at test time around a TD-trained value function; in Othello, this produced what was reported as the first agent trained on standard hardware to beat Edax up to and including level 7 [2204.13307].

Not all AlphaZero-style work preserves the original training objective. AlphaZeroES keeps the MCTS algorithm and neural architecture unchanged but replaces planning-loss minimization with direct score maximization via evolution strategies; in single-agent environments, this direct objective outperformed minimizing the usual value-and-policy planning loss [2406.08687]. Taken together, these results show that “AlphaZero-style” research has diversified along both axes: search engineering and optimization objective.

## 5. Generalizations beyond alternating two-player board games

The literature has repeatedly extended AlphaZero beyond its canonical setting.

| Extension | Setting | Principal modification |
|---|---|---|
| Multiplayer AlphaZero | 3-player deterministic games | Vector value head and max\(^n\)-style search |
| Simultaneous AlphaZero | Simultaneous-action Markov games | Matrix-game solving at each node; two policy heads |
| Search-problem adaptation | SAT and other search problems | Self-reductions as moves; easy-instance solvers as terminals |
| AlphaZero-UVR | Post-storm crew dispatch | Belief-state planning with stochastic MCTS |
| Grid topology optimization | Power-grid congestion control | Topology-action search plus learned policy priors |

In multiplayer settings, the scalar value head is replaced by a vector \(\vec v(s) = (v_1(s),\dots,v_n(s))\), terminal outcomes become score vectors, and MCTS is modified into an MCTS-max\(^n\) procedure in which each player maximizes its own component of the value vector at its turn; on two simple 3-player games, this modified AlphaZero consistently outperformed plain MCTS [1910.13012]. For simultaneous-action two-player zero-sum deterministic Markov games, Simultaneous AlphaZero models each state as a matrix game over joint actions, uses two policy heads and a distributional value head, and solves local matrix games with a regret-optimal solver under bandit feedback during search [2512.12486].

Other work departs even further from alternating games. An AlphaZero-inspired approach to SAT and general search problems treats problem instances as states, self-reductions as moves, and easy-instance solvers as terminal evaluators; the resulting system uses an MCTS-like procedure over a symbolic state-transition system rather than over game positions [2207.00919]. In post-storm repair crew dispatch, AlphaZero-UVR formulates routing as a sequential stochastic optimization problem with belief-state updates from customer calls and vehicle observations, and combines a neural network with stochastic MCTS to provide lookahead crew-navigation decisions [2010.06764]. In power systems, an AlphaZero-based topology optimization agent for congestion management uses topology actions as the learned action space and achieved a 60% reduction in the average amount of required redispatching while ranking first in the WCCI 2022 Learning to Run a Power Network competition [2211.05612].

These extensions imply that the core AlphaZero abstraction is not restricted to alternating-move board games. What persists is the planning architecture: learned priors and state evaluation embedded inside search, plus iterative policy improvement from self-generated trajectories.

## 6. Applications, empirical reach, and research significance

AlphaZero’s empirical reach within games has broadened rapidly. In Gomoku, an AlphaZero-style agent trained from random play and without domain knowledge apart from the rules learned a winning strategy on small boards within a few hours on a commodity GPU, achieved a 100% victory rate as the initiating player in self-play on 6×6 four-in-a-row, and outperformed a pure MCTS baseline across tested simulation counts from 500 to 2500 [2309.01294]. In chess-variant analysis, separately trained AlphaZero agents have been used as instruments for rule evaluation: for 1-second-per-move self-play, torpedo chess showed a White expected score of 56.8% and a draw rate of 71.9%, versus 51.8% and 88.2% in classical chess, respectively [2009.01294].

Outside games, AlphaZero-style planning has been used as an in silico design and discovery tool. In quantum control, a tabula rasa AlphaZero variant was applied to three classes of control problems using a single common set of algorithmic hyperparameters, achieving substantial improvements in both the quality and quantity of good solution clusters compared with earlier methods, and learning hidden structure and global symmetry in the solutions [1907.05672]. In power-grid operation and restoration, AlphaZero-style agents have been used for topology optimization and repair dispatch, with the appeal that topology actions are non-costly and carbon-free while redispatching is costly and often emissions-intensive [2211.05612] [2010.06764].

A recurring misconception is that AlphaZero is identical with a single frozen implementation. The broader literature suggests otherwise. The canonical algorithm remains the reference point—policy-value network, MCTS, self-play, and search-imitation loss—but many later systems alter the value representation, backup rule, optimization objective, or even the ontology of “state” and “action.” This suggests that AlphaZero now functions in research practice as both a named algorithm and a general planning-and-learning template for domains where execution-time search and self-generated supervision are jointly advantageous.

Source: https://www.emergentmind.com/topics/alphazero