---
title: Uncertainty-Aware Monte Carlo Tree Search
url: https://www.emergentmind.com/topics/uncertainty-aware-monte-carlo-tree-search-mcts
type: topic
---

# Uncertainty-Aware Monte Carlo Tree Search

Searching arXiv for recent and foundational work on uncertainty-aware MCTS.
Uncertainty-aware Monte Carlo Tree Search (MCTS) denotes a family of tree-search methods that augment the classical exploration–exploitation machinery of MCTS with explicit estimates of uncertainty about action quality, model accuracy, subtree structure, return distributions, belief-state ambiguity, or action plausibility. Rather than treating uncertainty as a by-product of visit counts alone, these methods modify selection, expansion, simulation, backup, or final root action choice so that search better reflects finite-budget decision quality, epistemic model error, structural incompleteness, or risk sensitivity [1207.5589] [2210.13455] [2309.10737] [2203.04452].

## 1. Conceptual foundations and historical development

A foundational distinction in uncertainty-aware MCTS is the separation between *cumulative regret* and *simple regret*. In flat bandits and in UCT-derived tree policies, the underlying UCB1 logic is designed to minimize cumulative regret. In many MCTS settings, however, only the final chosen move at a node matters. On that view, the relevant objective is simple regret, formalized as
$$
\mathbb{E}\, r = \sum_{j=1}^K \Delta_j \Pr(\overline{X}_j = \max_i \overline{X}_i),
\quad \Delta_j = \mu_* - \mu_j,
$$
and a sampling rule that directly reduces the probability of choosing a suboptimal final action becomes preferable to UCB1/UCT [1207.5589].

A second foundational line models uncertainty probabilistically at the node level. In Bayesian MCTS, each node maintains a probability distribution over its true expected reward value, with Beta-Bernoulli posteriors at leaves and posterior mean and variance propagated upward through max/min operators. This replaces purely count-derived confidence terms by posterior uncertainty estimates and yields Bayes-UCT variants that use posterior means and posterior standard deviations during selection [1203.3519].

A third line emphasizes *structural* rather than statistical uncertainty. Standard MCTS tracks local visit counts, but this does not capture how much of the subtree below an action remains unexplored. The resulting “second type of uncertainty” is induced by subtree size, depth variation, and loops; it can make standard MCTS diffuse visits across shallow and deep arms in sparse exploration problems [2005.09645]. Closely related work on asymmetric trees formalizes this as tree-structure uncertainty and uses it to down-weight exploration toward branches whose subtrees are already effectively exhausted [1805.09218].

More recent work generalizes uncertainty propagation beyond counts or scalar posteriors. Wasserstein MCTS models both value nodes and action-value nodes as Gaussian posteriors and backs up value nodes by an optimal-transport barycenter of child posteriors, so that both location and scale are propagated to the root [2309.10737]. In model-based deep RL, EMCTS augments AlphaZero/MuZero-style search with explicit epistemic uncertainty estimation and an optimism-under-uncertainty score, turning search itself into a mechanism for deep exploration [2210.13455].

## 2. Sources and representations of uncertainty

Uncertainty-aware MCTS methods differ primarily in what uncertainty signal they represent and how that signal is computed. One family uses concentration-based uncertainty over rollout means. In VOI-aware MCTS, the intrinsic value of information of sampling arm \(i\) is the expected decrease in regret compared to selecting the best arm without additional sampling, and the practical policy uses upper bounds on the budgeted VOI of allocating the remaining budget \(N\) to each arm [1207.5589].

Another family targets epistemic uncertainty from learned models. EMCTS distinguishes epistemic uncertainty from aleatoric uncertainty and uses local epistemic uncertainty \(\eta(s,a)\), leaf value uncertainty from the Uncertainty Bellman Equation, and a backed-up standard deviation \(\sigma_q(s,a)\) to form
$$
q_\beta(s,a) := q(s,a) + \beta \sigma_q(s,a).
$$
The exploration signal is explicitly the epistemic component rather than the aleatoric component [2210.13455].

In imperfect-model planning, uncertainty may be attached directly to transitions. UA-MCTS defines transition uncertainty as
$$
U(s,a) = \big(\hat{M}(s,a) - M(s,a)\big)^2,
$$
with the square applied elementwise to the state vector and then summed, and learns an estimator \(\hat{U}(s,a)\) from real experience. Search is then directed toward more certain transitions in the state space [2312.11348]. A related robotics formulation uses the Model Deviation Estimate,
$$
MDE(x_k, u_k) := \sigma^2([x_k^T, u_k^T]^T),
$$
where \(\sigma^2\) is the posterior predictive variance of a Gaussian Process Regression model. Here the uncertainty signal is explicitly interpreted as a proxy of epistemic uncertainty of the learned dynamics [2507.20861].

A different representation appears in structural-uncertainty methods. MCTS-T and MCTS-T+ maintain a per-state scalar \(\sigma_\tau(s)\in[0,1]\), with \(\sigma_\tau(s)=1\) indicating a completely unexplored subtree and \(\sigma_\tau(s)=0\) indicating a fully enumerated subtree. In deterministic, fully observable environments, loops are treated as a special case of subtree-depth variation by setting \(\sigma_\tau=0\) for repeated states on the same trace [2005.09645].

Other systems encode uncertainty at the action or belief level rather than in model dynamics or value posteriors. WebUncertainty introduces Confidence-induced Action Uncertainty (ConActU), computed from action-level confidence scores \(c_i\) as
$$
p_i = \frac{c_i}{\sum_{j=1}^K c_j}, \qquad
E = \frac{1}{K}\sum_{i=1}^K c_i, \qquad
H_{\mathrm{norm}} = - \frac{1}{\log K} \sum_{i=1}^K p_i \log p_i,
$$
followed by
$$
EU = 1 - E, \qquad AU = H_{\mathrm{norm}} \cdot E.
$$
This explicitly disentangles epistemic uncertainty as lack of knowledge and aleatoric uncertainty as ambiguity among plausible actions [2604.17821]. In autonomous driving under partial observability, uncertainty is represented by a Gaussian root belief and propagated through Unscented Transform sigma points, after which risk-aware action choice depends on the mean and variance of \(Q_{MDP}(x_i,a)\) across sigma points [1812.01254].

## 3. Modifications to the MCTS pipeline

Uncertainty awareness enters MCTS through concrete changes to the four standard phases. In selection, VOI-aware MCTS replaces UCB-style indices at the first step of each rollout by a direct estimate of expected improvement in final decision quality. With \(\alpha\) the empirically best arm and \(\beta\) the second-best, it selects
$$
i^* = \arg\max_i \hat{\Lambda}_i^b,
$$
where, for example,
$$
\hat{\Lambda}_\alpha^b = \frac{2N\,\overline{X}_\beta}{n_\alpha}
\exp\!\left(- 1.37\,(\overline{X}_\alpha - \overline{X}_\beta)^2 n_\alpha\right).
$$
This focuses sampling where a rank reversal is most plausible under the remaining budget [1207.5589].

In learned-model search, EMCTS replaces \(q(s,a)\) by \(q_\beta(s,a)\) and plugs it into EUCT or EPUCT:
$$
a_{EUCT}(s) = \arg\max_a \left[ q_\beta(s,a) + C_{UCT}\sqrt{\frac{2\log(\sum_{a'} N(s,a'))}{N(s,a)}} \right],
$$
$$
a_{EPUCT}(s) = \arg\max_a \left[ q_\beta(s,a) + \pi(a|s) C_{PUCT}\frac{\sqrt{\sum_{a'} N(s,a')}}{1 + N(s,a)} \right].
$$
The method also backs up an uncertainty estimate alongside returns, with
$$
[\nu(s,a)] = \sum_{j=0}^{T-1} \gamma^{2j} [\hat{R}(s_j,a_j)] + \gamma^{2T} [\hat{V}(s_T)].
$$
This makes selection optimism depend on epistemic standard deviation rather than only on counts or priors [2210.13455].

Transition-uncertainty methods alter all four phases. UA-MCTS uses an uncertainty-adapted UCT score
$$
\mathrm{UA\text{-}UCT}(v_i) =
\frac{Q(v_i)}{N(v_i)} +
c \sqrt{\frac{\ln N(v)}{N(v_i)} \cdot (1 - \alpha_i)},
$$
with
$$
\alpha_i =
\frac{\exp(\hat{U}(v_i)/\tau)}
{\sum_{v_j \in Ch(v)} \exp(\hat{U}(v_j)/\tau)}.
$$
Expansion can delete exactly one newly created child, biased toward higher uncertainty; simulation aggregates rollout returns with weights based on cumulative trajectory uncertainty; and backpropagation weights a child’s value contribution by how certain that child is relative to its siblings [2312.11348].

Some robotics planners bias selection and expansion toward reliable model regions without modifying rollout or backup. In liquid handling, the standard UCT score is multiplied by \((1-\delta_i)\), where \(\delta_i\) is a softmax of MDE across siblings, and expansion stochastically discards children with above-average MDE using a sigmoid in \(MDE(v_i)-\theta\). Simulation and backpropagation remain standard [2507.20861]. In asymmetric-tree methods, the exploration term is multiplied by \(\sigma_\tau(s')\),
$$
\pi_{\text{tree}(s)} = \arg\max_a \left[ Q(s,a) + c \cdot \sigma_\tau(s') \cdot \frac{\sqrt{n(s)}}{n(s,a)} \right],
$$
so that arms whose subtrees are largely known lose exploration pressure [2005.09645].

Uncertainty may also appear in reward shaping and backup semantics. WebUncertainty keeps a PUCT-style prior
$$
U(s,a) = w_{puct}\cdot \frac{P_{con}(s,a)\sqrt{\sum_b N(s,b)}}{1+N(s,a)}
$$
but uses stored \(AU\) and \(EU\) at the edge to modulate simulation rewards: high \(EU\) and high \(AU\) produce \(R=-5\), high \(EU\) and low \(AU\) produce \(R=-1\), low \(EU\) and high \(AU\) keep the base score, and low \(EU\) and low \(AU\) produce \(R=0\) [2604.17821]. Reward-Centered ReST-MCTS similarly modifies selection, rollout, and backup with
$$
r'_t = r_t + R_c(s_t), \qquad
R_c(s) = \alpha R_{rule}(s) + \beta R_{heur}(s) + \gamma_n R_{neural}(s),
$$
and prunes infeasible expansions through rule-based validation [2503.05226].

Finally, some methods propagate full uncertainty-bearing posteriors through backup. Wasserstein MCTS represents
$$
Q(s,a)\sim\mathcal{N}(\mu_{s,a},\sigma_{s,a}^2), \qquad
V(s)\sim\mathcal{N}(\mu_s,\sigma_s^2),
$$
and computes the \(V\)-posterior as a weighted \(L^1\)-Wasserstein barycenter of child \(Q\)-posteriors, yielding
$$
\mu_s = \left(\sum_a w_a \mu_{s,a}^p\right)^{1/p}, \qquad
\sigma_s = \left(\sum_a w_a \sigma_{s,a}^p\right)^{1/p}.
$$
This gives a generalized-mean backup for both expected value and uncertainty [2309.10737].

## 4. Decision criteria, risk, and robustness

A central theme in uncertainty-aware MCTS is that different applications require different decision criteria. VOI-aware MCTS is explicitly motivated by finite-budget *best action identification* rather than cumulative reward accumulation, and its theoretical analysis supports the use of uncertainty bounds for pure exploration and simple regret minimization [1207.5589]. By contrast, many risk-aware planners treat uncertainty as a property of the *distribution of returns* rather than the correctness of a single action ranking.

In risk-aware and multi-objective reinforcement learning, NLU-MCTS and DMCTS replace expected return optimization by expected utility of the cumulative return from individual policy executions. The relevant objective is
$$
V_u^\pi = \mathbb{E}\!\left[u\!\left(\sum_{t=0}^{\infty}\gamma^t \vec r_t\right)\right],
$$
and not \(u(\mathbb{E}[\sum_t \gamma^t \vec r_t])\). DMCTS maintains a bootstrap distribution over expected utility at each chance node and uses Thompson sampling during planning, which makes uncertainty awareness explicit at the utility level rather than at the level of scalar values alone [2211.13032].

Risk-sensitive planning under partial observability often uses belief-state sampling plus a final risk metric. In autonomous driving, the Unscented Transform yields sigma points \(x_i\) and weights \(W_i\), after which the planner computes
$$
\hat Q(b(s),a)=\sum_i W_i Q_{MDP}(x_i,a), \qquad
\hat \sigma^2(b(s),a)=\sum_i W_i\big(Q_{MDP}(x_i,a)-\hat Q(b(s),a)\big)^2,
$$
and selects
$$
a^*=\arg\max_{a\in A}\big(\hat Q(b(s),a)-\alpha \hat \sigma^2(b(s),a)\big).
$$
The modification to root action selection in MCTS is motivated by the observation that UCB can produce noisy \(Q\)-value estimates under uncertainty and degrade QMDP performance [1812.01254].

A related but distinct risk-aware construction appears in cooperative continuous-action planning. Here multiple deterministic trees are built from root belief samples, their action values are aggregated by kernel regression, and the final action is selected with either a Kernel Regression Lower Confidence Bound or a complementary CVaR criterion on reward particles derived from the root-sampled trees. The stated effect is consistently safer trajectories in uncertain environments [2203.04452].

Robustness can also be encoded as a search objective that is neither posterior variance nor return variance. DyRo-MCTS defines a schedule robustness functional
$$
\mathcal{R} = \sum_{m \in \mathcal{M}} \int_{0}^{T} w(t)\cdot \mathbb{I}_m(t)\,dt,
\qquad
w(t)=\min\left(0,\frac{t}{\beta}-1\right),
$$
normalizes it into \(\rho(s,a)\), and interpolates it with normalized tardiness value:
$$
\mathcal{E}(s,a)=\alpha q(s,a)+(1-\alpha)\rho(s,a).
$$
The planner then uses \(\mathcal{E}(s,a)\) inside a PUCT-style score, thereby steering search toward schedules that are “easily adaptable to future job arrivals” [2509.21902].

These formulations show that uncertainty-aware MCTS is not tied to a single notion of uncertainty. In some cases uncertainty is used to identify the best arm under a budget; in others it is used to penalize hallucination-prone actions, to prefer reliable model regions, to quantify posterior dispersion, or to compute a risk-adjusted objective at the root. A common misconception is that uncertainty-aware MCTS is always Bayesian. The literature instead includes concentration-based VOI bounds, subtree-completeness measures, GP predictive variance, risk metrics over root-sampled return distributions, and heuristic confidence decompositions [1207.5589] [2005.09645] [2507.20861] [2203.04452] [2604.17821].

## 5. Empirical domains and performance patterns

Empirical results across domains show that uncertainty-aware modifications are most useful when standard MCTS is misaligned with the dominant failure mode. In random multi-armed bandits with 32 Bernoulli arms and budgets \(32..1024\), VOI-aware sampling was consistently better in simple regret than UCB1, and in 9x9 Computer Go a VOI-aware root policy achieved higher winning rates than UCT for most sample budgets under matched average samples per node [1207.5589].

In sparse-reward exploration with learned models, EMCTS shows the clearest effect of propagating epistemic uncertainty through search. On DeepSea \(40\times 40\), E-AZ with RND novelty reached the first goal in \(10539 \pm 9006\) steps and discovered the goal in \(94\%\) of 35 seeds, whereas AZ discovered the goal in \(0\%\) of 20 seeds. In the same setting, E-MZ with counts novelty reached \(14339 \pm 6845\) steps and discovered the goal in \(100\%\) of 23 seeds, while MZ+UBE discovered the goal in \(57\%\) of 21 seeds [2210.13455].

In stochastic and partially observable planning, propagating uncertainty distributions can materially change performance. On Pocman, W-MCTS-TS with \(p=100\) achieved \(77.70 \pm 1.22\) discounted total reward at 65536 simulations, whereas UCT achieved \(28.5 \pm 3.8\). W-MCTS-TS also outperformed UCT and D2NG in Rocksample, and W-MCTS-TS was the only algorithm to reliably collect all three passengers in Taxi [2309.10737].

When the uncertainty is action-level confidence and ambiguity, uncertainty-aware search can improve both effectiveness and runtime. On WebVoyager with Qwen-Max, WebUncertainty reached \(63.6\%\) success rate versus \(58.9\%\) for AgentOccam and \(55.8\%\) for WebPilot, with average time \(351.4\) s versus \(803.7\) s for WebPilot. Removing Action Uncertainty reduced WebVoyager success rate to \(54.3\%\) from \(63.6\%\) [2604.17821].

In robotics with inaccurate learned dynamics, uncertainty-aware search tends to trade a small increase in conservatism for improved reliability. In robotized liquid handling, UA-MCTS-1 achieved \(100\%\) success on the 40-point and 10-point GPR models, \(97\%\) on the 20-point and 5-point GPR models, and consistently outperformed standard MCTS and UA-MCTS-0 in success rate under a 0.5 s planning budget [2507.20861]. In dynamic job shop scheduling, DyRo-MCTS provided consistent additional gains over vanilla MCTS across all scenarios; the reported total improvements reached up to \(57\%\) and \(45\%\) for Random priors, while average decision time remained \(0.021 \pm 0.006\) s at 100 iterations and \(0.212 \pm 0.061\) s at 1000 iterations [2509.21902].

Evidence from reasoning and manipulation under noisy conditions follows the same pattern. Reward-Centered ReST-MCTS reports a \(2\text{–}4\%\) accuracy improvement over Chain-of-Thought prompting and Vanilla ReST-MCTS while maintaining computational feasibility, with over \(85\%\) accuracy retained under high noise levels in robustness tests [2503.05226]. In satellite scheduling under cloud cover uncertainty, Average-MCTS was more robust than Max-MCTS on large instances and achieved \(\approx 658.4\) expected value on the 15-orbit, 300-task set versus \(630.2\) for the GA baseline, although it trailed the GA baseline on the 500-task set [2405.20951].

## 6. Limitations, misconceptions, and open directions

Uncertainty-aware MCTS methods share neither a common uncertainty source nor a common guarantee. Some methods rely on stationary, bounded rewards and concentration inequalities; VOI-aware MCTS explicitly notes that deeper MCTS nodes can be nonstationary and that the stationary-based VOI bounds are used at the first decision step [1207.5589]. Others rely on Gaussian posteriors or Gaussian approximation; Wasserstein MCTS notes that heavy-tailed or multimodal returns may be misrepresented under the Gaussianity assumption [2309.10737]. In imperfect-model search, UA-MCTS can be misled when \(\hat{U}(s,a)\) is inaccurate, and the strongest empirical gains come from selection and expansion rather than from weighted rollouts or backups alone [2312.11348].

A second misconception is that uncertainty awareness automatically implies robustness to aleatoric noise. EMCTS explicitly uses epistemic uncertainty and does not use aleatoric randomness as the exploration signal [2210.13455]. WebUncertainty separates \(EU\) and \(AU\) and uses them differently: \(EU\) drives penalties for hallucination-prone trajectories, while \(AU\) marks ambiguity as an exploration target [2604.17821]. The liquid-handling planner does not focus on aleatoric uncertainty and does not propagate transition distributions; it uses GP predictive variance as an epistemic proxy [2507.20861].

A third misconception is that injecting uncertainty is always bias-free. Reward-centered shaping changes the backed-up return itself and is “not potential-based shaping,” so invariance of the optimal policy is not guaranteed [2503.05226]. WebUncertainty states that uncertainty-shaped rewards deviate from classical unbiased returns and that standard UCT asymptotic optimality guarantees do not directly apply [2604.17821]. Mixed-criticality MCTS provides feasibility and safety structure for high-criticality actions, but it does not provide regret bounds and represents uncertainty by optimistic and pessimistic worst-case budgets rather than by a stochastic transition model [2407.12564].

Open directions in the literature are correspondingly heterogeneous. VOI-based work points to full-rollout metareasoning, deeper-node VOI, and sample reuse across states [1207.5589]. EMCTS identifies uncertainty propagation through learned latent transitions in MuZero as non-trivial and uses optimistic approximations in practice [2210.13455]. Robust scheduling work suggests learning better robustness estimators, while continuous-action robotics work explicitly notes extensions via progressive widening and broader epistemic estimators such as ensembles or MC dropout [2509.21902] [2507.20861].

Taken together, these studies suggest that uncertainty-aware MCTS is best understood not as a single algorithmic variant but as a design principle: uncertainty should be represented in the form most closely tied to the failure mode that matters at decision time. When the problem is finite-budget action choice, VOI and simple regret are central. When the problem is sparse-reward exploration with learned models, epistemic uncertainty propagation is central. When the problem is inaccurate simulation, transition or model deviation uncertainty is central. When the problem is partial observability or safety, belief-state aggregation and risk metrics are central. When the problem is structurally asymmetric search, subtree uncertainty is central. The diversity of the literature reflects that plurality rather than a settled canonical formulation [1207.5589] [2210.13455] [2312.11348] [2203.04452] [2005.09645].

Source: https://www.emergentmind.com/topics/uncertainty-aware-monte-carlo-tree-search-mcts