Expert Iteration (EI) Overview

• Expert Iteration (EI) is an iterative framework that alternates between a policy-improvement step using expert planning (e.g., MCTS) and a policy-distillation step training an apprentice model on self-play data.
• It has been applied in board games, reasoning, and automated theorem proving, achieving strong empirical results by refining decisions through recursive search and learning.
• Variants such as WED, PER, and opponent-aware planning extend EI by manipulating experience distributions and integrating opponent models for enhanced performance.

Searching arXiv for recent and foundational papers on Expert Iteration to ground the article. Expert Iteration (EI), often abbreviated ExIt, is an iterative learning framework in which an “expert” policy generates improved decisions from states and an “apprentice” policy is trained to imitate those decisions, so that planning and function approximation recursively strengthen one another. In game-playing systems, the expert is typically a search procedure such as Monte-Carlo Tree Search (MCTS), while the apprentice is a parameterized policy or actor-critic network trained on self-play data (Soemers et al., 2020). In later extensions, the same loop has been adapted to online planning without search trees, opponent-conditioned best-response learning, LLM reasoning, and automated theorem proving in LEAN, where models iteratively search for proofs and retrain on the proofs they discover (Anthony et al., 2019, Hernandez et al., 2022, Wu et al., 2024).

1. Core formulation

At its canonical core, EI alternates between a policy-improvement step and a policy-distillation step. The expert step applies search or planning from encountered states to obtain an improved action distribution, while the apprentice step fits a learned policy to that distribution. In board-game ExIt, self-play games are generated between agents running MCTS enhanced by the apprentice policy; for each encountered state $s$, MCTS produces a distribution over actions based on visit counts, and the apprentice is trained to mimic that expert distribution through cross-entropy minimization (Soemers et al., 2020).

A common formalization uses the expert distribution $\mathcal{M}_s(s)$ and apprentice policy $\pi_\theta(s)$, with a policy loss of the form

$$\mathcal{L}_{CE} \approx \frac{1}{N}\sum_{i=1}^N \mathcal{M}_{s_i}(s_i)^\top \log \pi_\theta(s_i).$$

In AlphaZero-style chess systems and related implementations, the expert policy target after search is the normalized visit count distribution,

$\pi(s,a)=\frac{N(s,a)}{\sum_a N(s,a)},$

optionally with temperature transformation, while the apprentice network is trained jointly on policy and value targets extracted from self-play trajectories (V. et al., 2018).

Several papers place EI within broader RL taxonomy. One formulation describes ExIt as a generalization of Approximate Policy Iteration in which policy improvement is performed via multi-step search and the apprentice is trained to approximate the expert’s output (Anthony et al., 2019). Another situates Deep Pepper within the Classification-based Approximate Policy Iteration framework, where MCTS acts as the expert and the network is refit on triplets $(s,\pi(s,a),\nu)$ containing state, search-improved policy, and game outcome (V. et al., 2018). Across these variants, the central invariant is the same: the apprentice guides the next round of expert search, and the expert generates the next round of supervisory signal.

2. Search-based expert construction in games

The standard expert in ExIt is MCTS, usually guided by priors from the apprentice. In Deep Pepper, each node tracks visit counts $N(s,a)$, cumulative values $W(s,a)$, mean values $Q(s,a)$, and policy priors $P(s,a)$, and child selection uses a PUCT-style score $\mathcal{M}_s(s)$0 (V. et al., 2018). This yields a search-improved target policy for each state, after which the apprentice network is trained with mean squared error on value and cross-entropy on policy, plus $\mathcal{M}_s(s)$1 regularization.

Deep Pepper also illustrates a non-tabula-rasa ExIt design. It uses a 353-dimensional custom feature representation inspired by Giraffe rather than AlphaZero’s raw board tensor, and it optionally pretrains on grandmaster games with Stockfish-derived value and policy labels (V. et al., 2018). The system then evaluates each newly trained network against the previous network and retains it if it wins more than 50% of games.

A major methodological extension replaces tree search with Policy Gradient Search (PGS). PGS adapts a neural-network simulation policy online via REINFORCE-style updates during search,

$\mathcal{M}_s(s)$2

thereby avoiding an explicit search tree (Anthony et al., 2019). At the root, action selection still uses a PUCT bandit rule, but the rollout policy is locally adapted for the current search. This design targets regimes where MCTS scales poorly because of very high branching factors or stochastic transitions. Empirically, in Hex, PGS achieved comparable performance to MCTS, and an agent trained using Expert Iteration with PGS defeated MoHex 2.0, the strongest open-source Hex agent, in 9x9 Hex; the reported match score was 375–273 using 800 search rounds per move for PGS-ExIt versus 10,000 iterations for MoHex 2.0 (Anthony et al., 2019).

3. Variants for data distribution and opponent-aware planning

Beyond the basic expert–apprentice loop, ExIt research has repeatedly modified how experience is collected, weighted, and conditioned. One line of work manipulates the self-play distribution used for apprentice updates; another changes the expert itself by injecting opponent models into planning.

The following variants were reported in board-game ExIt and best-response ExIt settings (Soemers et al., 2020, Hernandez et al., 2022):

Variant	Modification	Reported effect
WED	Sample weighting inverse to episode duration	9 of 28 top ranks; average strategy mass 0.304
PER	Sampling prioritized by apprentice–expert disagreement	2 top ranks; average strategy mass 0.118
CEE	Exploratory policy mixed into self-play trajectories	Detrimental on average with IS correction
BRExIt	Opponent-model heads and opponent-conditioned MCTS priors	PoI $\mathcal{M}_s(s)$3 for BRExIt; $\mathcal{M}_s(s)$4 for BRExIt-OMS over ExIt

In “Manipulating the Distributions of Experience used for Self-Play Learning in Expert Iteration,” three mechanisms were studied across fourteen board games: weighting samples based on episode durations (WED), Prioritized Experience Replay (PER), and a trained exploratory policy for trajectory diversification called Cross-Entropy Exploration (CEE) (Soemers et al., 2020). WED stores all states but weights each state inversely to the duration of the episode in which it occurred, so that every episode is “equally important” for training. PER samples experiences non-uniformly according to the total absolute difference between expert and apprentice action distributions. CEE mixes a learned exploratory policy with the expert distribution during self-play and applies importance-sampling corrections. The reported outcome was that WED gave the largest and most consistent improvements, particularly early in training; PER produced modest improvements, mainly in stability; and CEE with importance-sampling correction was detrimental on average (Soemers et al., 2020).

BRExIt modifies ExIt for best-response learning in multi-agent games by adding opponent-model heads to the apprentice and by replacing the apprentice prior at opponent nodes in MCTS with either a learned or ground-truth opponent policy (Hernandez et al., 2022). Its total loss augments the usual policy and value terms with an auxiliary opponent-modeling loss, and its replay tuples include opponent state–action information in addition to the agent’s own training targets. In Connect4 against frozen PPO-based opponents, both BRExIt with ground-truth opponent models and BRExIt-OMS with learned opponent models consistently outperformed ExIt, whereas using opponent models only for feature shaping did not reliably help and could harm performance (Hernandez et al., 2022).

4. Expert Iteration with LLMs

In LLM systems, EI is no longer restricted to game-tree planning. It becomes a self-training loop over reasoning trajectories, proofs, or refusals, with the expert formed by search, ranking, or resampling procedures around the current policy. Two 2024 papers exemplify this transition: Automatic Curriculum Expert Iteration for reasoning alignment and large-scale LEAN theorem proving with policy–critic search (Zhao et al., 2024, Wu et al., 2024).

Auto-CEI uses Expert Iteration to mitigate hallucination and laziness in reasoning tasks by sampling multiple reasoning trajectories from the current policy, scoring them with a reward that depends on correctness and trajectory length, and retraining via supervised fine-tuning on an expert-curated resampling of those trajectories (Zhao et al., 2024). Its reward function is

$\mathcal{M}_s(s)$5

and its curriculum objective is

$\mathcal{M}_s(s)$6

The step count in the chain of thought acts as a proxy for both difficulty and capability boundary. Refusals after enough reasoning steps receive a positive reward, while premature refusals are penalized. On BoardgameQA, MATH, and Blocksworld, Auto-CEI reported precision scores of 84.52%, 55.63%, and 91.53%, respectively, while maintaining refusal rates of 18–36% and improving precision over baselines by 10–24 percentage points (Zhao et al., 2024).

InternLM2.5-StepProver applies EI to formal theorem proving in LEAN on Lean-Workbook-Plus, described as the largest open-source Lean 4 dataset with more than 82,000 problems, using more than 20,000 CPU days of proof search (Wu et al., 2024). The system starts from InternLM2-StepProver and alternates proof search with retraining. Search combines Best-First Search, which expands states with highest cumulative log-probabilities under the policy model, and Critic-Guided Search, which prioritizes states scored by a learned critic. The first pass uses a small search budget of 10 iterations and 50 seconds per problem; solved or disproved statements are removed; later rounds increase the budget up to 2,000 iterations and 1 hour per problem; and proofs found in each round are added to the training set for further fine-tuning of both policy and critic models (Wu et al., 2024).

The critic predicts how close a state is to a completed proof (“no goals”) and is trained with pairwise preference learning using path-pair and sibling-pair strategies. Reported validation accuracy for distinguishing progress along proof paths was 78% (Wu et al., 2024). The paper further reports log-linear trends between solved problem amount, proof length, and CPU usage. The number of problems successfully proved decreases log-linearly as proof length and CPU time increase; only 1.5% of CPU resources led to solving 17% of problems, with the remaining 98.5% spent on unsolved problems; and critic-guided search found deeper proofs than naive best-first search, with average proof lengths of 4.44 versus 1.66 (Wu et al., 2024).

The benchmark results position this EI pipeline as a strong open-source theorem prover. InternLM2.5-StepProver achieved a pass of 65.9% on MiniF2F-test, solved or disproved 17.0% of Lean-Workbook-Plus problems—10,880 proved and 3,195 disproved, up from 9.5% when Lean-Workbook-Plus was released—reached 27.0% pass@256 on ProofNet, and solved 6 out of 640 Putnam problems (Wu et al., 2024).

5. Relation to adjacent iterative schemes

EI is closely related to, but distinct from, other iterative improvement procedures built around foundation models. The clearest contrast in the cited literature is with In-Context Policy Iteration (ICPI), which performs policy iteration entirely through prompt updates rather than gradient-based weight changes (Brooks et al., 2022).

The ICPI paper characterizes classical Expert Iteration as a framework that alternates between imitation or reward learning from an expert policy and improving the apprentice policy, typically via supervised learning or RL gradients (Brooks et al., 2022). By contrast, ICPI keeps the LLM frozen, stores self-generated trajectories in a replay buffer, and changes only the prompt content used for action selection and rollout prediction. Its action choice is

$\mathcal{M}_s(s)$7

with action values estimated by LLM-predicted rollouts conditioned on sampled prompt content from the buffer. The paper’s explicit point of contrast is that EI generally requires expert trajectories or strong off-policy supervision and often updates model weights, whereas ICPI uses no expert demonstrations and no gradients (Brooks et al., 2022).

This comparison clarifies the conceptual boundary of EI. In EI, the expert is externalized as search, planning, or reward-ranked trajectory selection, and the apprentice is updated to imitate or absorb that improved behavior. In ICPI, the prompt buffer itself becomes the locus of adaptation. A plausible implication is that recent LLM work has split the classical EI template into two directions: one that scales search-and-distillation over model weights, as in Auto-CEI and theorem proving, and one that relocates the improvement operator into context management alone (Brooks et al., 2022, Zhao et al., 2024, Wu et al., 2024).

6. Empirical profile, significance, and limitations

Across domains, EI is associated with strong empirical performance when search can generate informative targets and when the apprentice can generalize those targets efficiently. In self-play board games, ExIt was reported as effective for learning game-playing policies from self-play, with experience-distribution manipulations producing major improvements in early training performance in some games and minor improvements averaged over fourteen games (Soemers et al., 2020). In Hex, PGS-ExIt substantially outperformed Approximate Policy Iteration with Monte Carlo Search and approached AlphaZero-style MCTS-ExIt, while also demonstrating that strong planning-based EI need not rely on explicit search trees (Anthony et al., 2019).

In reasoning and theorem proving, EI serves a different but structurally homologous role: it bootstraps stronger supervision from the model’s own search neighborhood. Auto-CEI shows that the expert need not be a game-tree planner; it can be a reward-shaped resampling operator over reasoning trajectories that explicitly controls assertiveness versus conservativeness (Zhao et al., 2024). InternLM2.5-StepProver shows that, in formal mathematics, large-scale EI can be organized over more than 82,000 Lean problems with critic-guided search and iterative retraining, yielding open-source state-of-the-art results on MiniF2F, Lean-Workbook-Plus, ProofNet, and Putnam (Wu et al., 2024).

The limitations reported in the literature are equally instructive. Classical MCTS-based ExIt inherits MCTS scaling difficulties in domains with very high branching factors or stochastic transitions (Anthony et al., 2019). Experience manipulation can help, but not every intervention is beneficial: CEE with importance-sampling correction harmed learning on average (Soemers et al., 2020). Standard EI in reasoning may improve overall accuracy while allowing error rates to grow rapidly with reasoning length, which motivated Auto-CEI’s curriculum-based reward shaping (Zhao et al., 2024). In theorem proving, resource concentration on unsolved problems is severe: the reported proof-search distribution has a heavy tail, and 98.5% of CPU time was spent on unsolved problems (Wu et al., 2024).

The cumulative significance of these results is that EI has become a reusable design pattern rather than a domain-specific trick. Its concrete instantiations differ—MCTS, PGS, opponent-conditioned search, reward-shaped trajectory resampling, critic-guided proof search—but each retains the same operational schema: construct an expert signal that is stronger than the current policy at visited states, then train the policy to approximate that signal so that the next round of expert construction starts from a better base.