Evo-MCTS: Adaptive Evolution in MCTS

• Evo-MCTS is a family of algorithms that integrates evolutionary computation with MCTS by evolving selection policies using a symbolic grammar to replace traditional UCT formulas.
• It employs an online evolutionary strategy with mutation and semantic filtering to adaptively fine-tune exploration-exploitation trade-offs in dynamically structured reward landscapes.
• Empirical results in domains like Carcassonne demonstrate that Evo-MCTS effectively navigates multimodal and deceptive scenarios, often outperforming classic UCT under constrained rollouts.

Evo-MCTS, or Evolutionary Monte Carlo Tree Search, denotes a family of algorithms integrating evolutionary computation with the classical Monte Carlo Tree Search paradigm. This approach primarily targets the automated synthesis and adaptation of the statistical tree selection policy within MCTS, specifically as a replacement or augmentation to the canonical Upper Confidence Bounds for Trees (UCT) formula. The driving motivation is to overcome the rigidity and suboptimality that arises from fixed, hand-tuned UCT selection policies in domains with diverse, deceptive, or dynamically structured reward landscapes, such as combinatorial games, function optimization, or multi-agent environments (Galván et al., 2022, Galván et al., 2021, Ameneyro et al., 2023, Galvan et al., 2023, Panwar et al., 2021).

1. Formalization of UCT and Evolutionary Selection Policies

The UCT policy is the de facto standard for node selection in MCTS, expressed for node $v$ (child of parent $p$) as

$$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$

with $\overline X_v$ as the empirical mean reward at $v$, $N_p$ and $N_v$ the visit counts for $p$ and $v$, respectively, and $C$ a tunable exploration-exploitation trade-off parameter. Its effectiveness relies on careful calibration of $p$0, which can be problem-dependent and nontrivial in complex settings (Galván et al., 2022, Galvan et al., 2023).

Evo-MCTS generalizes this selection policy by evolving expressions using a symbolic grammar inspired by Genetic Programming (GP). Each candidate policy is represented as an expression tree with

• Terminals: $p$1, where $p$2 is the (possibly unnormalized) accumulated reward, $p$3 and $p$4 are parent/child visit counts, and $p$5 a tunable numeric constant.
• Functions: $p$6 with protected semantics to guard against invalid numeric domains (Galván et al., 2022, Ameneyro et al., 2023, Galvan et al., 2023).

The result is a symbolic policy, $p$7, which can take highly non-standard forms, e.g.,

$p$8

2. Evolutionary Algorithm and Semantic Guidance

Evo-MCTS employs an online evolutionary strategy to adapt the selection policy at each MCTS decision point. Typically, this is a $p$9-Evolution Strategy with $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$0 (current parent), $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$1 offspring, and $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$2 generations per move.

The evolutionary workflow encompasses:

• Initialization: Seed parent with canonical UCT.
• Variation: Subtree mutation exclusively (node replacement at internal/leaf), no crossover, max depth constraints.
• Fitness Evaluation: Use each candidate policy as the MCTS selection formula for $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$3 rollouts; average the resultant (empirical) rewards as fitness.
• Semantic Selection: Semantic-inspired variants (e.g., SIEA-MCTS) employ Sampling Semantic Distance (SSD) to prioritize offspring whose behavioral reward profiles ($$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$4) are neither too similar nor too divergent from the parent's, as defined by SSD thresholds $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$5 (typical: $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$6, $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$7). This preserves behavioral diversity and robustness in very small populations (Galván et al., 2022, Ameneyro et al., 2023, Galvan et al., 2023).

Pseudocode for one MCTS selection decision: $\overline X_v$7 (Galván et al., 2022, Ameneyro et al., 2023)

3. Empirical Performance and Evolved Policy Structures

Empirical analyses in domains such as Carcassonne illustrate that Evo-MCTS and SIEA-MCTS produce dynamically adaptive, per-turn expressions. Example evolved formulas span:

• Purely exploitative (e.g., $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$8, $$\mathrm{UCT}(v) = \overline X_v + C \sqrt{\frac{\ln N_p}{N_v}},$$9),
• Modified exploration terms (e.g., using $\overline X_v$0, divisions, or nested roots/logs),
• Elimination or down-weighting of explicit exploration bonuses,
• Combinations tailored to the encountered reward distributions (Galván et al., 2021, Galván et al., 2022).

A summary of tournament results in Carcassonne (using 400 rollouts):

Controller	Points	Win–loss–draw	Avg. Point Diff.
MCTS-UCT ($\overline X_v$1) 2800	109	23–1–0	+646.6
SIEA-MCTS (400+evo)	86	18–5–1	+352.0
MCTS-RAVE (2800)	82	17–7–0	+352.0
EA-MCTS (400+evo)	82	17–7–0	+354.3
EA-p-MCTS (partial)	10	2–22–0	–660.3
Random	0	0–24–0	–1901.2

SIEA-MCTS is not statistically distinguishable from optimally tuned MCTS-UCT ($\overline X_v$2) at 2800 rollouts, and with only 400 rollouts, SIEA-MCTS outperforms all other 400-budget controllers (Galván et al., 2022).

In single-function optimization, Evo-MCTS and SIEA-MCTS demonstrate superior coverage of multimodal or deceptive optima, whereas UCT is reliably optimal in unimodal scenarios, provided $\overline X_v$3 is tuned (Ameneyro et al., 2023, Galvan et al., 2023).

4. Domain-specific Adaptations and Extensions

Evo-MCTS was initially developed for deterministic, two-player domains (e.g., Carcassonne), but the methodology extends to:

• Arbitrary function optimization scenarios, where evolved policies adapt to reward topology (e.g., presence of multiple peaks, deceptive traps) (Ameneyro et al., 2023, Galvan et al., 2023).
• Multi-agent, partially observable games (e.g., Pommerman), where evolutionary operators are instead used to optimize rollout/default policies rather than tree selection formulas (e.g., FEMCTS), yielding significant gains over Rolling Horizon Evolution and competitive performance with classical, well-tuned MCTS (Panwar et al., 2021).

5. Comparative Evaluation and Insights

Analysis across several benchmarks yields the following conclusions:

• Unimodal/benign domains: Classic UCT (with moderate $\overline X_v$4) is simpler and typically outperforms Evo-MCTS.
• Multimodal/deceptive/rugged domains: Evo-MCTS and SIEA-MCTS deliver superior exploration and more robust avoidance of local optima, at the expense of increased per-move computational overhead.
• Semantic guidance: Semantic diversity (SSD filtering) materially increases robustness and consistency, especially critical for small evolutionary populations, by maintaining behavioral variance and mitigating premature convergence (Galván et al., 2022, Galvan et al., 2023).
• Overhead: The cost of per-decision online evolution must be justified by sufficient reward landscape complexity; otherwise, classic UCT should be preferred (Ameneyro et al., 2023, Galvan et al., 2023).

6. Limitations and Prospective Research Directions

• No single, universally optimal evolved policy emerges; the method generates a trajectory of context-specific formulas, which may preclude interpretability and complicate transferability.
• Parameter settings (number of generations, mutation rates, SSD thresholds) can be domain-sensitive and may require adaptation.
• Current methods use fixed arithmetic grammars; inclusion of broader function sets (e.g., $\overline X_v$5, exponentials) or crossover operators may yield richer evolved behaviors.
• Prospective research encompasses:
  • Automatic adaptation of semantic thresholds ($\overline X_v$6),
  • Evaluation in other stochastic and adversarial multi-agent domains,
  • Integration of offline-evolved static formulas with online semantic fine-tuning,
  • Evolution of rollout as well as selection policies (Galván et al., 2022, Panwar et al., 2021, Ameneyro et al., 2023, Galvan et al., 2023).

7. Broader Implications and Practitioner Guidance

Evo-MCTS establishes a principled framework for adaptive search control in tree-based planning methods, offering tangible benefits in domains characterized by complex search topologies or deceptive reward structures. The approach's main advantage lies in automated, per-instance tuning of exploration-exploitation trade-offs without manual calibration. However, standard UCT remains preferable in uniformly smooth domains or under strict resource constraints. A practitioner aiming for robust performance in nontrivial problem classes should consider Evo-MCTS or SIEA-MCTS, especially with semantic EA integration (Galván et al., 2022, Galvan et al., 2023, Ameneyro et al., 2023).