Multiple Policy Value MCTS (MPV-MCTS)

• MPV-MCTS is an extension of PV-MCTS that integrates two policy-value networks to jointly enhance search breadth and evaluative accuracy.
• It interleaves rapid simulations from a fast 'small net' with selective high-fidelity evaluations from a 'large net' using a shared tree framework.
• Empirical studies on 9×9 NoGo and AlphaZero self-play show significant Elo gains and a training acceleration of approximately 2× over conventional methods.

Multiple Policy Value Monte Carlo Tree Search (MPV-MCTS) is an extension of policy value Monte Carlo Tree Search (PV-MCTS) that interleaves simulations using multiple policy-value neural networks (PV-NNs) of differing computational cost and predictive accuracy. In this approach, two PV-NNs—a “small net” ($f_S$) and a “large net” ($f_L$)—are jointly leveraged within a shared tree expansion and backup mechanism. This framework enables the agent to benefit from both high simulation throughput and improved policy/value estimation, offering a principled balance between search breadth and evaluative accuracy. Empirical studies on the 9×9 NoGo domain demonstrate statistically significant gains over conventional PV-MCTS and enhanced efficiency in AlphaZero-style self-play training (Lan et al., 2019).

1. Network Architectures, Training, and Intuition

MPV-MCTS employs two distinct PV-NNs, each grounded in AlphaZero-style residual towers:

• $f_S$ (“small net”):
  • Supervised: 200K NoGo self-play games ($\approx 10^7$ positions).
  • AlphaZero (AZ): 800 simulations, PUCT $c=1.5$, replay buffer 100K, 500K SGD steps on 60 self-play GPUs + 4 training GPUs ($\approx 2$M games).
• $f_L$ (“large net”):

Architecture: 128 filters, 10 residual blocks ($f_L$0). Cost: baseline for one unit of normalized budget. Training: Identical data and AZ regimen with separate weight schedule.

The design rationale is that $f_L$1 enables rapid rollouts and broader search, while $f_L$2 provides more reliable policy priors and state-value estimates. MPV-MCTS fuses their outputs through coordinated search and aggregation.

2. Mathematical Fusion and Shared-Tree Framework

MPV-MCTS grows two parallel trees, $f_L$3 and $f_L$4, sharing priors and value estimates at co-visited nodes. Given state $f_L$5:

• $f_L$6
• $f_L$7

The combined policy $f_L$8 and value $f_L$9 used in selection and backup are: $f_S$0

$f_S$1

where $f_S$2. In the reported experiments, $f_S$3, $f_S$4 (i.e., $f_S$5, $f_S$6).

Selection in each tree follows the PUCT formula: $f_S$7

$f_S$8

Upon evaluation of a leaf node $f_S$9, with result $f_{64,5}$0, standard MCTS back-up is performed: $f_{64,5}$1 Updates from $f_{64,5}$2 and $f_{64,5}$3 reinforce each other through shared $f_{64,5}$4 and $f_{64,5}$5 values.

3. Simulation Scheduling and Algorithmic Details

Given simulation budgets $f_{64,5}$6 for $f_{64,5}$7 and $f_{64,5}$8 respectively, the algorithm interleaves $f_{64,5}$9 simulations with $\approx 1/8$0 and $\approx 1/8$1 with $\approx 1/8$2. The default scheduling samples $\approx 1/8$3 of the $\approx 1/8$4 iterations uniformly at random for the small net, the remainder for the large net.

For $\approx 1/8$5 simulations:

1. SELECT in $\approx 1/8$6 via PUCT to obtain a leaf $\approx 1/8$7.
2. EXPAND & EVALUATE: $\approx 1/8$8.
3. BACKUP in $\approx 1/8$9 and update shared statistics.

For $f_{128,10}$0 simulations:

1. Select unevaluated leaf in $f_{128,10}$1 with highest $f_{128,10}$2 (visit-count from $f_{128,10}$3); fallback to PUCT if no such node.
2. EXPAND & EVALUATE: $f_{128,10}$4.
3. BACKUP in $f_{128,10}$5 and update shared statistics.

The normalized action distribution for play is $f_{128,10}$6.

Alternative scheduling (e.g., round-robin, front-loading $f_{128,10}$7) is permitted, provided budget constraints are respected.

4. Balancing Exploration and High-Fidelity Estimation

Breadth of the search is attributed to $f_{128,10}$8, as its low computational cost leads to extensive rollouts ($f_{128,10}$9), thus constructing large $\approx 10^7$0. Accuracy, in contrast, comes from $\approx 10^7$1 via select, more costly evaluations ($\approx 10^7$2), targeted at the most promising regions as indicated by $\approx 10^7$3 through the $\approx 10^7$4 priority.

Budget allocation is guided by a parameter $\approx 10^7$5: $\approx 10^7$6, $\approx 10^7$7 for total budget $\approx 10^7$8. Empirically, $\approx 10^7$9 yielded optimal performance in NoGo. Mixing weights $c=1.5$0 can be tuned to favor stronger networks (e.g., $c=1.5$1 to bias $c=1.5$2 toward $c=1.5$3).

5. Experimental Results: NoGo and AlphaZero Self-Play

Performance was validated using NoGo and AlphaZero self-play training protocols:

a) Supervised NoGo, 9×9:

• Budgets $c=1.5$4 normalized units.
• $c=1.5$5 alone: peak $c=1.5$6 Elo ($c=1.5$7).
• $c=1.5$8 alone: peak $c=1.5$9 Elo.
• MPV-MCTS ($\approx 2$0): peak $\approx 2$1 Elo ($\approx 2$2 over large-only). Outperforms all intermediate-sized nets.

b) AZ-trained PV-NNs:

• $\approx 2$3 self-play simulations, $\approx 2$4, $\approx 2$5K buffer, $\approx 2$6M games.
• At eight checkpoints, evaluate each $\approx 2$7, and MPV.
• MPV-MCTS consistently exceeds both constituent nets by tens of Elo across all test budgets.

c) AlphaZero training with MPV-MCTS:

• Self-play: $\approx 2$8 per move.
• Baselines: $\approx 2$9-only with $f_L$0 sims/move.
• Equalized “generated-game” budget: $f_L$1 sim $f_L$2 $f_L$3 unit, $f_L$4 sim $f_L$5 unit, MPV $f_L$6 unit.
• After $f_L$7M games:
  • Best $f_L$8: $f_L$9 Elo ($f_L$00-sim test), $f_L$01 Elo ($f_L$02-sim test).
  • MPV: $f_L$03 Elo / $f_L$04 Elo ($f_L$05 / $f_L$06 over baseline).
  • MPV trained for $f_L$07 games outperforms $f_L$08 trained for $f_L$09 games at 51.2–56.6% win-rates—approximately a $f_L$10 acceleration in training.

6. Generalization, Extensions, and Practical Implications

MPV-MCTS is a general multi-teacher extension of PV-MCTS; any set of PV-NNs with differing accuracy/computational cost can be utilized. The scheme permits:

• A curriculum of incrementally larger “support” nets, with meta-learned mixing weights and simulation scheduling.
• Refinement of the $f_L$11 priority function, such as PUCT-based selection, discounted parent visits, or hybrid heuristics.
• Deployment of “micro-nets” (e.g., 32 filters) under tight constraints; ablation indicates the tree search driven by $f_L$12 is critical regardless of individual network strength.
• Applicability to any turn-based game (e.g., chess, shogi, Hex) or continuous-action RL tasks leveraging fast “sampler” alongside a slower “accurate” network.
• Facilitates ensemble approaches or knowledge distillation: smaller nets may acquire high-quality targets from the Q-values of larger nets, potentially accelerating neural convergence.

In sum, MPV-MCTS realizes a theoretically principled and empirically validated division of labor between rapid search and high-fidelity evaluation, substantially enhancing both online performance and the efficiency of reinforcement learning in complex domains (Lan et al., 2019).