---
title: Monte Carlo Permutation Search (MCPS)
url: https://www.emergentmind.com/topics/monte-carlo-permutation-search-mcps
type: topic
---

# Monte Carlo Permutation Search (MCPS)

Monte Carlo Permutation Search (MCPS) is a general-purpose Monte Carlo Tree Search (MCTS) algorithm that improves upon the GRAVE algorithm and is relevant when deep reinforcement learning is not an option, or when the computing power available before play is not substantial, such as in General Game Playing [2510.06381]. Its defining principle is to include in the exploration term of a node the statistics on all the playouts that contain all the moves on the path from the root to the node. MCPS therefore extends the standard MCTS and GRAVE reuse of simulation data by introducing a third statistical source, designed for settings in which permutations of moves are informative because order is not crucial or because permuting moves often reaches the same or a similar state [2510.06381].

## 1. Position within Monte Carlo Tree Search

Standard MCTS learns from statistics of moves played as the first move after a node. In the notation used for MCPS, these are the ordinary move-first statistics \(Q(s,a)\) and \(N(s,a)\), where \(Q(s,a)\) is the average reward of playouts below \(s\) that start with \(a\), and \(N(s,a)\) is the number of such playouts [2510.06381]. UCT uses these quantities to balance exploitation and exploration, but early in the search it can be data-starved.

RAVE and AMAF address this problem by reusing statistics from playouts where a move appears later in the simulation, not only as the first move. GRAVE improves on RAVE by using AMAF statistics not from the node itself, but from an ancestor node chosen by a reference threshold \(ref\) [2510.06381]. MCPS pushes this line of development further. It uses three statistics instead of two: ordinary statistics \(Q(s,a)\), AMAF statistics \(\tilde Q(s,a)\), and permutation statistics \(\hat Q(s,a)\) [2510.06381].

The conceptual motivation is that many games are permutation-friendly. If a set of moves appears in a playout, then any permutation of those moves may correspond to the same state or at least a state that is highly informative. MCPS is meant for exactly these settings: general game playing, combinatorial games, and domains where one cannot rely on offline training or learned priors [2510.06381]. This suggests a view of MCPS as a “search only” method with little or no tuning, rather than as a search procedure augmented by a learned policy or value model.

## 2. Core construction: ordinary, AMAF, and permutation statistics

The central object in MCPS is the path-conditioned permutation statistic. The paper defines a state \(s\) as the sequence of actions from the root,
\[
s = a_0, a_1, a_2, \ldots, a_d,
\]
and a playout \(p\) as
\[
p = p_0, p_1, p_2, \ldots, p_t.
\]
Ordinary statistics are computed from playouts below the current node that start with the candidate move:
\[
Q(s,a) = avg\bigl(\{p \in P(s)\mid p_0 = a\}\bigr), \qquad
N(s,a) = |\{p \in P(s)\mid p_0 = a\}|.
\]
AMAF statistics broaden the reuse criterion to any playout below the node containing the move anywhere:
\[
\tilde Q(s,a) = avg\bigl(\{p \in P(s)\mid a \in p\}\bigr), \qquad
\tilde N(s,a) = |\{p \in P(s)\mid a \in p\}|.
\]
MCPS then adds permutation statistics computed from all playouts from the root that contain both the candidate move and every move on the path to the current node, regardless of order:
\[
\hat Q(s,a) = avg\bigl(\{p \in P(root)\mid a \in p,\ \forall a_i \in s:\ a_i \in p\}\bigr),
\]
\[
\hat N(s,a) = |\{p \in P(root)\mid a \in p,\ \forall a_i \in s:\ a_i \in p\}|.
\]
This condition is stronger than AMAF, because it requires the whole path’s move set to appear in the playout, not just the candidate move [2510.06381].

The consequence is that MCPS treats the root-to-node path as a set-like signature and asks whether a playout contains that entire signature. In domains where move order permutations are meaningful, this gives more data for the exploration term than node-local statistics alone. A plausible implication is that MCPS is best understood not merely as another AMAF variant, but as a path-conditioned reuse mechanism that interpolates between exact local statistics and broader transposition-like regularities.

## 3. Selection rule and weighting formulas

MCPS combines the three statistical sources in a convex mixture:
\[
Q_*(s,a) = \alpha Q(s,a) + \beta \tilde Q(s,a) + \gamma \hat Q(s,a), \qquad \alpha+\beta+\gamma=1.
\]
The paper derives the weights by minimizing mean squared error under an independence assumption, an equal bias assumption,
\[
b=\tilde b=\hat b,
\]
and Bernoulli-style variance approximations,
\[
\sigma^2=\frac{\mu(1-\mu)}{n}, \qquad
\tilde \sigma^2=\frac{\mu(1-\mu)}{\tilde n}, \qquad
\hat \sigma^2=\frac{\mu(1-\mu)}{\hat n},
\]
with
\[
n=N(s,a), \qquad \tilde n=\tilde N(s,a), \qquad \hat n=\hat N(s,a).
\]
The resulting raw optimum is proportional to sample counts:
\[
\alpha^\star = \frac{n}{n+\tilde n+\hat n}, \qquad
\beta^\star = \frac{\tilde n}{n+\tilde n+\hat n}, \qquad
\gamma^\star = \frac{\hat n}{n+\tilde n+\hat n}.
\]
The paper states that this is the minimum-variance combination under the stated assumptions, and also notes that \(Q_*(s,a)\) is a positive convex combination of three estimates, so the minimizer is unique [2510.06381].

The practical algorithm does not use these raw formulas unchanged. The paper says that direct use would differ from RAVE/GRAVE-like behavior for large numbers of playouts, so it introduces a correction factor \(c_1\) on \(\alpha\), chosen to make \(\alpha\) tend to \(\frac{n}{n+\tilde n}\) for large numbers of playouts. This yields
\[
c_1=\frac{\tilde n+\hat n}{\tilde n},
\]
and the implemented weights become
\[
\alpha = \frac{c_1 n}{c_1 n + \tilde n + \hat n}, \qquad
\beta = \frac{\tilde n}{c_1 n + \tilde n + \hat n}, \qquad
\gamma = \frac{\hat n}{c_1 n + \tilde n + \hat n}.
\]
This weighting rule is one of the main distinctions between MCPS and GRAVE. GRAVE uses a \(bias\) hyperparameter in its mixture rule, whereas MCPS no longer uses the bias hyperparameter of GRAVE [2510.06381].

## 4. Reference ancestors, move coding, and implementation tradeoffs

MCPS still uses a reference ancestor node \(s_{\text{ref}}\) as GRAVE does, but the experiments report that MCPS is not sensitive to the choice of \(ref\). The tested values are \(ref \in [50,100,200,400]\), and the paper reports that performance is relatively stable, so a default \(ref = 50\) is fine across games [2510.06381]. This point is central to the paper’s practical positioning: MCPS is meant to reduce dependence on tuning, especially relative to GRAVE.

Efficient computation of AMAF and permutation statistics requires a coding scheme for moves. The paper distinguishes exact codes, defined as a bijection between moves and integers, from abstract codes, which encode only part of a move [2510.06381]. Abstract codes are important because they appear in more playouts, make \(\tilde n\) and \(\hat n\) larger, strengthen the AMAF and permutation contributions, and reduce memory and computational cost.

The paper also identifies a tradeoff. If codes are too specific, they appear too rarely and permutation statistics are weak. If codes are too abstract, they appear in almost all playouts, making the statistics non-discriminative [2510.06381]. Abstract codes are reported as especially useful in the Wargame, Investment Pair Game, and Video Game. For board games, exact and abstract codes are effectively the same because the move representation already matches the needed abstraction [2510.06381]. MCPS uses abstract codes more heavily than GRAVE because both \(\tilde Q\) and \(\hat Q\) benefit from them.

A corresponding limitation is that MCPS requires extra bookkeeping for permutation statistics. Its benefit is also contingent on the domain: if permutations rarely occur in playouts, gains are smaller, and if the permutation set is small, the third source of statistics has limited effect [2510.06381].

## 5. Empirical evaluation across game classes

The experimental comparison is direct: MCPS is evaluated against GRAVE with the same number of playouts. GRAVE uses \(ref = 50\) and \(bias = 10^{-5}\). MCPS is tested with \(ref \in [50,100,200,400]\) and playout budgets \(N = 1000\) and \(5000\) [2510.06381]. For two-player games, the protocol uses 800 games per parameter setting, with 400 games from each first-player assignment. For three-player games, it also uses 800 games, split evenly across starting positions, with seeds 0–799 and 200 parallel processes [2510.06381].

The reported two-player domains are Atarigo 6x6, Breakthrough 8x8, Gomoku 9x9, Hex 7x7, Knightthrough 8x8, Nogo 5x5, a sequential Wargame, Investment Pair Game, and a Video Game inspired by TFT. The multi-player domains are Three-player Nogo 5x5, Three-player Wargame 10x10x10, and Three-player Investment Pair Game 10x10x10 [2510.06381].

Across the six board games, MCPS is better than GRAVE or at least comparable in every case. The aggregate table reports that, for all games at 1000 playouts, MCPS wins around 57–58% depending on \(ref\), and for all games at 5000 playouts, MCPS wins about 62% [2510.06381]. The paper highlights that MCPS improves more than GRAVE as the budget grows.

Several game-specific outcomes are singled out. Hex is the strongest result for MCPS, with 74–77% at 5000 playouts depending on \(ref\); this is the game where permutations are most natural. Nogo is also strong. Knightthrough is weaker, consistent with lower frequency of useful move codes in playouts. Atarigo shows modest but consistent improvement, while Breakthrough and Gomoku show small to moderate gains [2510.06381].

In the Wargame, MCPS beats GRAVE and the gain increases with more playouts. With abstract codes, MCPS performs substantially better than with exact codes, and much better than GRAVE with exact codes. In the Investment Pair Game, MCPS again outperforms GRAVE, especially on the 21x21 version with abstract codes, where MCPS with abstract codes versus GRAVE with abstract codes reaches 65.25% at 5000 playouts. In the Video Game, MCPS and GRAVE are close, but MCPS still improves slightly more with increased playout budget [2510.06381].

For multi-player games, MCPS and GRAVE are roughly balanced. The paper attributes this to the games themselves being inherently balanced through coalition effects or shared outcomes, not because MCPS fails in principle [2510.06381].

## 6. Applicability, strengths, and limitations

The paper characterizes MCPS as most appropriate when deep RL or learned priors are not available, the search budget before play is limited, the domain is a combinatorial game or other general planning problem, and permutations of actions can reveal useful shared structure [2510.06381]. Within that regime, its reported strengths are better performance than GRAVE in all the reported two-player games, no need for the GRAVE \(bias\) parameter, low sensitivity to \(ref\), the ability to exploit abstract move coding for extra gains, and generality across game classes [2510.06381].

The limitations are equally explicit. MCPS requires extra bookkeeping for permutation statistics, is more useful when move-order permutations are meaningful, can suffer from overly abstract codes that make statistics too diffuse, and may show muted benefit in multi-player settings because game dynamics can already balance outcomes [2510.06381]. The paper is also careful to note that MCPS does not guarantee that usual \(Q(s,a)\) dominates asymptotically in the same way as GRAVE’s bias term is designed to make \(\beta \to 0\). Instead, the empirical claim is that, because the usual statistics are included in the mixture, the methods can still converge in practice when one move becomes dominant [2510.06381].

A common misconception is to treat MCPS as a universally stronger replacement for GRAVE. The published evidence supports a narrower conclusion: MCPS is empirically stronger than GRAVE on the paper’s two-player benchmarks, especially in games where move permutations are natural and abstract codes are informative, but the gain is domain-dependent and not presented as a universal asymptotic theorem [2510.06381].

## 7. Related meanings of permutation-based Monte Carlo

The name “Monte Carlo Permutation Search” can be confused with several adjacent research lines, but the literature distinguishes them. The paper “Monte Carlo Search Algorithms Discovering Monte Carlo Tree Search Exploration Terms” searches over mathematical expressions encoded in reverse polish notation and evaluates them empirically on cached search datasets; its core idea is described as very close in spirit to MCPS and other Monte Carlo optimization methods, because it searches over a structured space of candidate formulas rather than over a continuous parameter space [2404.09304]. This is conceptually adjacent to MCPS as Monte Carlo optimization over symbolic discrete space, but it is not the same algorithmic object as the GRAVE-derived MCPS of [2510.06381].

A different neighboring usage appears in “A Multi-Level Monte Carlo Tree Search Method for Configuration Generation in Crystalline Systems,” where substitutional configurations are explored by swap-based rearrangements over a permutation-like state space. The paper states that, if MCPS is understood broadly as Monte Carlo search over permutations, its ML-MCTS method is interpretable as a form of MCPS, but distinguishes its own contribution as a tree-structured and hierarchical variant with level-dependent action constraints [2507.02509].

In statistics, “Monte Carlo permutation” often refers not to game-tree search but to finite-budget permutation testing. “More Permutations Do Not Always Increase Power: Non-monotonicity in Monte Carlo Permutation Tests” shows that increasing the number of sampled permutations does not always increase power, because the rejection cutoff
\[
k_B=\lfloor (B+1)\alpha\rfloor -1
\]
creates a saw-toothed power curve with infinitely many local maxima [2605.03886]. This result is directly relevant to MCPS-style procedures only in the broad sense that the number of sampled permutations is part of the procedure and can have nontrivial statistical consequences.

Two further neighboring traditions concern permutation structure in MCMC and conditional testing. Neal’s “How to view an MCMC simulation as a permutation” reformulates MCMC updates as random permutations on an extended finite space or as volume-preserving one-to-one maps on a continuous extended space [1205.0070]. “Markov Chain Monte Carlo sampling for conditional tests: A link between permutation tests and algebraic statistics” develops an orbit-based MCMC over permutation classes of the conditional sample space and shows that sampling at the orbit level can give a more reliable estimate of the conditional cdf under \(H_0\) than standard permutation sampling [1707.08513]. These works share the central motif that permutation structure can be exploited to improve Monte Carlo procedures, but they address inference and sampling rather than the specific MCTS exploration rule that defines MCPS.

Within this broader landscape, MCPS is most precisely identified as GRAVE plus a third, permutation-based statistical source and a principled reweighting scheme, designed for search without deep learning and for domains in which path-move-set information is useful [2510.06381].

Source: https://www.emergentmind.com/topics/monte-carlo-permutation-search-mcps