Uncertainty-Aware Monte Carlo Tree Search

• Uncertainty-aware MCTS is a family of enhanced tree search algorithms that integrate explicit uncertainty estimates into selection, expansion, simulation, and backup phases.
• These methods employ diverse uncertainty representations—including statistical, structural, and epistemic—to address both cumulative and simple regret in decision making.
• Empirical results across domains like Go, robotics, and reinforcement learning demonstrate that uncertainty-aware MCTS improves robustness and risk-sensitive planning compared to traditional approaches.

Searching arXiv for recent and foundational work on uncertainty-aware MCTS. Uncertainty-aware Monte Carlo Tree Search (MCTS) denotes a family of tree-search methods that augment the classical exploration–exploitation machinery of MCTS with explicit estimates of uncertainty about action quality, model accuracy, subtree structure, return distributions, belief-state ambiguity, or action plausibility. Rather than treating uncertainty as a by-product of visit counts alone, these methods modify selection, expansion, simulation, backup, or final root action choice so that search better reflects finite-budget decision quality, epistemic model error, structural incompleteness, or risk sensitivity (Tolpin et al., 2012, Oren et al., 2022, Dam et al., 2023, Stegmaier et al., 2022).

1. Conceptual foundations and historical development

A foundational distinction in uncertainty-aware MCTS is the separation between cumulative regret and simple regret. In flat bandits and in UCT-derived tree policies, the underlying UCB1 logic is designed to minimize cumulative regret. In many MCTS settings, however, only the final chosen move at a node matters. On that view, the relevant objective is simple regret, formalized as

$$\mathbb{E}\, r = \sum_{j=1}^K \Delta_j \Pr(\overline{X}_j = \max_i \overline{X}_i), \quad \Delta_j = \mu_* - \mu_j,$$

and a sampling rule that directly reduces the probability of choosing a suboptimal final action becomes preferable to UCB1/UCT (Tolpin et al., 2012).

A second foundational line models uncertainty probabilistically at the node level. In Bayesian MCTS, each node maintains a probability distribution over its true expected reward value, with Beta-Bernoulli posteriors at leaves and posterior mean and variance propagated upward through max/min operators. This replaces purely count-derived confidence terms by posterior uncertainty estimates and yields Bayes-UCT variants that use posterior means and posterior standard deviations during selection (Tesauro et al., 2012).

A third line emphasizes structural rather than statistical uncertainty. Standard MCTS tracks local visit counts, but this does not capture how much of the subtree below an action remains unexplored. The resulting “second type of uncertainty” is induced by subtree size, depth variation, and loops; it can make standard MCTS diffuse visits across shallow and deep arms in sparse exploration problems (Moerland et al., 2020). Closely related work on asymmetric trees formalizes this as tree-structure uncertainty and uses it to down-weight exploration toward branches whose subtrees are already effectively exhausted (Moerland et al., 2018).

More recent work generalizes uncertainty propagation beyond counts or scalar posteriors. Wasserstein MCTS models both value nodes and action-value nodes as Gaussian posteriors and backs up value nodes by an optimal-transport barycenter of child posteriors, so that both location and scale are propagated to the root (Dam et al., 2023). In model-based deep RL, EMCTS augments AlphaZero/MuZero-style search with explicit epistemic uncertainty estimation and an optimism-under-uncertainty score, turning search itself into a mechanism for deep exploration (Oren et al., 2022).

2. Sources and representations of uncertainty

Uncertainty-aware MCTS methods differ primarily in what uncertainty signal they represent and how that signal is computed. One family uses concentration-based uncertainty over rollout means. In VOI-aware MCTS, the intrinsic value of information of sampling arm $i$ is the expected decrease in regret compared to selecting the best arm without additional sampling, and the practical policy uses upper bounds on the budgeted VOI of allocating the remaining budget $N$ to each arm (Tolpin et al., 2012).

Another family targets epistemic uncertainty from learned models. EMCTS distinguishes epistemic uncertainty from aleatoric uncertainty and uses local epistemic uncertainty $\eta(s,a)$, leaf value uncertainty from the Uncertainty Bellman Equation, and a backed-up standard deviation $\sigma_q(s,a)$ to form

$q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$

The exploration signal is explicitly the epistemic component rather than the aleatoric component (Oren et al., 2022).

In imperfect-model planning, uncertainty may be attached directly to transitions. UA-MCTS defines transition uncertainty as

$U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$

with the square applied elementwise to the state vector and then summed, and learns an estimator $\hat{U}(s,a)$ from real experience. Search is then directed toward more certain transitions in the state space (Kohankhaki et al., 2023). A related robotics formulation uses the Model Deviation Estimate,

$MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$

where $\sigma^2$ is the posterior predictive variance of a Gaussian Process Regression model. Here the uncertainty signal is explicitly interpreted as a proxy of epistemic uncertainty of the learned dynamics (Faroni et al., 28 Jul 2025).

A different representation appears in structural-uncertainty methods. MCTS-T and MCTS-T+ maintain a per-state scalar $i$0, with $i$1 indicating a completely unexplored subtree and $i$2 indicating a fully enumerated subtree. In deterministic, fully observable environments, loops are treated as a special case of subtree-depth variation by setting $i$3 for repeated states on the same trace (Moerland et al., 2020).

Other systems encode uncertainty at the action or belief level rather than in model dynamics or value posteriors. WebUncertainty introduces Confidence-induced Action Uncertainty (ConActU), computed from action-level confidence scores $i$4 as

$i$5

followed by

$i$6

This explicitly disentangles epistemic uncertainty as lack of knowledge and aleatoric uncertainty as ambiguity among plausible actions (Zhang et al., 20 Apr 2026). In autonomous driving under partial observability, uncertainty is represented by a Gaussian root belief and propagated through Unscented Transform sigma points, after which risk-aware action choice depends on the mean and variance of $i$7 across sigma points (Naghshvar et al., 2018).

3. Modifications to the MCTS pipeline

Uncertainty awareness enters MCTS through concrete changes to the four standard phases. In selection, VOI-aware MCTS replaces UCB-style indices at the first step of each rollout by a direct estimate of expected improvement in final decision quality. With $i$8 the empirically best arm and $i$9 the second-best, it selects

$N$0

where, for example,

$N$1

This focuses sampling where a rank reversal is most plausible under the remaining budget (Tolpin et al., 2012).

In learned-model search, EMCTS replaces $N$2 by $N$3 and plugs it into EUCT or EPUCT:

$N$4

$N$5

The method also backs up an uncertainty estimate alongside returns, with

$N$6

This makes selection optimism depend on epistemic standard deviation rather than only on counts or priors (Oren et al., 2022).

Transition-uncertainty methods alter all four phases. UA-MCTS uses an uncertainty-adapted UCT score

$N$7

with

$N$8

Expansion can delete exactly one newly created child, biased toward higher uncertainty; simulation aggregates rollout returns with weights based on cumulative trajectory uncertainty; and backpropagation weights a child’s value contribution by how certain that child is relative to its siblings (Kohankhaki et al., 2023).

Some robotics planners bias selection and expansion toward reliable model regions without modifying rollout or backup. In liquid handling, the standard UCT score is multiplied by $N$9, where $\eta(s,a)$0 is a softmax of MDE across siblings, and expansion stochastically discards children with above-average MDE using a sigmoid in $\eta(s,a)$1. Simulation and backpropagation remain standard (Faroni et al., 28 Jul 2025). In asymmetric-tree methods, the exploration term is multiplied by $\eta(s,a)$2,

$\eta(s,a)$3

so that arms whose subtrees are largely known lose exploration pressure (Moerland et al., 2020).

Uncertainty may also appear in reward shaping and backup semantics. WebUncertainty keeps a PUCT-style prior

$\eta(s,a)$4

but uses stored $\eta(s,a)$5 and $\eta(s,a)$6 at the edge to modulate simulation rewards: high $\eta(s,a)$7 and high $\eta(s,a)$8 produce $\eta(s,a)$9, high $\sigma_q(s,a)$0 and low $\sigma_q(s,a)$1 produce $\sigma_q(s,a)$2, low $\sigma_q(s,a)$3 and high $\sigma_q(s,a)$4 keep the base score, and low $\sigma_q(s,a)$5 and low $\sigma_q(s,a)$6 produce $\sigma_q(s,a)$7 (Zhang et al., 20 Apr 2026). Reward-Centered ReST-MCTS similarly modifies selection, rollout, and backup with

$\sigma_q(s,a)$8

and prunes infeasible expansions through rule-based validation (Wang, 7 Mar 2025).

Finally, some methods propagate full uncertainty-bearing posteriors through backup. Wasserstein MCTS represents

$\sigma_q(s,a)$9

and computes the $q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$0-posterior as a weighted $q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$1-Wasserstein barycenter of child $q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$2-posteriors, yielding

$q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$3

This gives a generalized-mean backup for both expected value and uncertainty (Dam et al., 2023).

4. Decision criteria, risk, and robustness

A central theme in uncertainty-aware MCTS is that different applications require different decision criteria. VOI-aware MCTS is explicitly motivated by finite-budget best action identification rather than cumulative reward accumulation, and its theoretical analysis supports the use of uncertainty bounds for pure exploration and simple regret minimization (Tolpin et al., 2012). By contrast, many risk-aware planners treat uncertainty as a property of the distribution of returns rather than the correctness of a single action ranking.

In risk-aware and multi-objective reinforcement learning, NLU-MCTS and DMCTS replace expected return optimization by expected utility of the cumulative return from individual policy executions. The relevant objective is

$q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$4

and not $q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$5. DMCTS maintains a bootstrap distribution over expected utility at each chance node and uses Thompson sampling during planning, which makes uncertainty awareness explicit at the utility level rather than at the level of scalar values alone (Hayes et al., 2022).

Risk-sensitive planning under partial observability often uses belief-state sampling plus a final risk metric. In autonomous driving, the Unscented Transform yields sigma points $q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$6 and weights $q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$7, after which the planner computes

$q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$8

and selects

$q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).$9

The modification to root action selection in MCTS is motivated by the observation that UCB can produce noisy $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$0-value estimates under uncertainty and degrade QMDP performance (Naghshvar et al., 2018).

A related but distinct risk-aware construction appears in cooperative continuous-action planning. Here multiple deterministic trees are built from root belief samples, their action values are aggregated by kernel regression, and the final action is selected with either a Kernel Regression Lower Confidence Bound or a complementary CVaR criterion on reward particles derived from the root-sampled trees. The stated effect is consistently safer trajectories in uncertain environments (Stegmaier et al., 2022).

Robustness can also be encoded as a search objective that is neither posterior variance nor return variance. DyRo-MCTS defines a schedule robustness functional

$U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$1

normalizes it into $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$2, and interpolates it with normalized tardiness value:

$U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$3

The planner then uses $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$4 inside a PUCT-style score, thereby steering search toward schedules that are “easily adaptable to future job arrivals” (Chen et al., 26 Sep 2025).

These formulations show that uncertainty-aware MCTS is not tied to a single notion of uncertainty. In some cases uncertainty is used to identify the best arm under a budget; in others it is used to penalize hallucination-prone actions, to prefer reliable model regions, to quantify posterior dispersion, or to compute a risk-adjusted objective at the root. A common misconception is that uncertainty-aware MCTS is always Bayesian. The literature instead includes concentration-based VOI bounds, subtree-completeness measures, GP predictive variance, risk metrics over root-sampled return distributions, and heuristic confidence decompositions (Tolpin et al., 2012, Moerland et al., 2020, Faroni et al., 28 Jul 2025, Stegmaier et al., 2022, Zhang et al., 20 Apr 2026).

5. Empirical domains and performance patterns

Empirical results across domains show that uncertainty-aware modifications are most useful when standard MCTS is misaligned with the dominant failure mode. In random multi-armed bandits with 32 Bernoulli arms and budgets $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$5, VOI-aware sampling was consistently better in simple regret than UCB1, and in 9x9 Computer Go a VOI-aware root policy achieved higher winning rates than UCT for most sample budgets under matched average samples per node (Tolpin et al., 2012).

In sparse-reward exploration with learned models, EMCTS shows the clearest effect of propagating epistemic uncertainty through search. On DeepSea $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$6, E-AZ with RND novelty reached the first goal in $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$7 steps and discovered the goal in $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$8 of 35 seeds, whereas AZ discovered the goal in $U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,$9 of 20 seeds. In the same setting, E-MZ with counts novelty reached $\hat{U}(s,a)$0 steps and discovered the goal in $\hat{U}(s,a)$1 of 23 seeds, while MZ+UBE discovered the goal in $\hat{U}(s,a)$2 of 21 seeds (Oren et al., 2022).

In stochastic and partially observable planning, propagating uncertainty distributions can materially change performance. On Pocman, W-MCTS-TS with $\hat{U}(s,a)$3 achieved $\hat{U}(s,a)$4 discounted total reward at 65536 simulations, whereas UCT achieved $\hat{U}(s,a)$5. W-MCTS-TS also outperformed UCT and D2NG in Rocksample, and W-MCTS-TS was the only algorithm to reliably collect all three passengers in Taxi (Dam et al., 2023).

When the uncertainty is action-level confidence and ambiguity, uncertainty-aware search can improve both effectiveness and runtime. On WebVoyager with Qwen-Max, WebUncertainty reached $\hat{U}(s,a)$6 success rate versus $\hat{U}(s,a)$7 for AgentOccam and $\hat{U}(s,a)$8 for WebPilot, with average time $\hat{U}(s,a)$9 s versus $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$0 s for WebPilot. Removing Action Uncertainty reduced WebVoyager success rate to $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$1 from $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$2 (Zhang et al., 20 Apr 2026).

In robotics with inaccurate learned dynamics, uncertainty-aware search tends to trade a small increase in conservatism for improved reliability. In robotized liquid handling, UA-MCTS-1 achieved $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$3 success on the 40-point and 10-point GPR models, $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$4 on the 20-point and 5-point GPR models, and consistently outperformed standard MCTS and UA-MCTS-0 in success rate under a 0.5 s planning budget (Faroni et al., 28 Jul 2025). In dynamic job shop scheduling, DyRo-MCTS provided consistent additional gains over vanilla MCTS across all scenarios; the reported total improvements reached up to $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$5 and $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$6 for Random priors, while average decision time remained $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$7 s at 100 iterations and $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$8 s at 1000 iterations (Chen et al., 26 Sep 2025).

Evidence from reasoning and manipulation under noisy conditions follows the same pattern. Reward-Centered ReST-MCTS reports a $MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),$9 accuracy improvement over Chain-of-Thought prompting and Vanilla ReST-MCTS while maintaining computational feasibility, with over $\sigma^2$0 accuracy retained under high noise levels in robustness tests (Wang, 7 Mar 2025). In satellite scheduling under cloud cover uncertainty, Average-MCTS was more robust than Max-MCTS on large instances and achieved $\sigma^2$1 expected value on the 15-orbit, 300-task set versus $\sigma^2$2 for the GA baseline, although it trailed the GA baseline on the 500-task set (Norman et al., 2024).

6. Limitations, misconceptions, and open directions

Uncertainty-aware MCTS methods share neither a common uncertainty source nor a common guarantee. Some methods rely on stationary, bounded rewards and concentration inequalities; VOI-aware MCTS explicitly notes that deeper MCTS nodes can be nonstationary and that the stationary-based VOI bounds are used at the first decision step (Tolpin et al., 2012). Others rely on Gaussian posteriors or Gaussian approximation; Wasserstein MCTS notes that heavy-tailed or multimodal returns may be misrepresented under the Gaussianity assumption (Dam et al., 2023). In imperfect-model search, UA-MCTS can be misled when $\sigma^2$3 is inaccurate, and the strongest empirical gains come from selection and expansion rather than from weighted rollouts or backups alone (Kohankhaki et al., 2023).

A second misconception is that uncertainty awareness automatically implies robustness to aleatoric noise. EMCTS explicitly uses epistemic uncertainty and does not use aleatoric randomness as the exploration signal (Oren et al., 2022). WebUncertainty separates $\sigma^2$4 and $\sigma^2$5 and uses them differently: $\sigma^2$6 drives penalties for hallucination-prone trajectories, while $\sigma^2$7 marks ambiguity as an exploration target (Zhang et al., 20 Apr 2026). The liquid-handling planner does not focus on aleatoric uncertainty and does not propagate transition distributions; it uses GP predictive variance as an epistemic proxy (Faroni et al., 28 Jul 2025).

A third misconception is that injecting uncertainty is always bias-free. Reward-centered shaping changes the backed-up return itself and is “not potential-based shaping,” so invariance of the optimal policy is not guaranteed (Wang, 7 Mar 2025). WebUncertainty states that uncertainty-shaped rewards deviate from classical unbiased returns and that standard UCT asymptotic optimality guarantees do not directly apply (Zhang et al., 20 Apr 2026). Mixed-criticality MCTS provides feasibility and safety structure for high-criticality actions, but it does not provide regret bounds and represents uncertainty by optimistic and pessimistic worst-case budgets rather than by a stochastic transition model (Cordeiro et al., 2024).

Open directions in the literature are correspondingly heterogeneous. VOI-based work points to full-rollout metareasoning, deeper-node VOI, and sample reuse across states (Tolpin et al., 2012). EMCTS identifies uncertainty propagation through learned latent transitions in MuZero as non-trivial and uses optimistic approximations in practice (Oren et al., 2022). Robust scheduling work suggests learning better robustness estimators, while continuous-action robotics work explicitly notes extensions via progressive widening and broader epistemic estimators such as ensembles or MC dropout (Chen et al., 26 Sep 2025, Faroni et al., 28 Jul 2025).

Taken together, these studies suggest that uncertainty-aware MCTS is best understood not as a single algorithmic variant but as a design principle: uncertainty should be represented in the form most closely tied to the failure mode that matters at decision time. When the problem is finite-budget action choice, VOI and simple regret are central. When the problem is sparse-reward exploration with learned models, epistemic uncertainty propagation is central. When the problem is inaccurate simulation, transition or model deviation uncertainty is central. When the problem is partial observability or safety, belief-state aggregation and risk metrics are central. When the problem is structurally asymmetric search, subtree uncertainty is central. The diversity of the literature reflects that plurality rather than a settled canonical formulation (Tolpin et al., 2012, Oren et al., 2022, Kohankhaki et al., 2023, Stegmaier et al., 2022, Moerland et al., 2020).