Alpha-UCT Selection Rule

• Alpha-UCT selection rule is a modified version of classical UCT that uses the maximum observed rollout value to accelerate branch pruning in step-level MCTS.
• It leverages martingale concentration theory to provide rigorous regret guarantees and improve performance in non-iid, open-ended environments.
• Empirical results in GUI-based planning domains show significant gains in action evaluation accuracy and search efficiency through hyperparameter tuning.

Alpha-UCT is a selection rule introduced in the Agent Alpha framework for step-level Monte Carlo Tree Search (MCTS) in computer-use agents. Designed to synergize generation, exploration, and evaluation, Alpha-UCT modifies the exploitation and exploration components of classical UCT to enable proactive identification and pruning of suboptimal branches, effective prefix reuse, and improved empirical performance in open-ended GUI-based planning domains. The approach is grounded in martingale concentration theory and provides theoretical guarantees on regret while demonstrating substantial performance gains on benchmarks such as OSWorld (Tang et al., 3 Feb 2026).

1. Formal Definition and Computational Components

At each non-terminal node $v$ in the search tree, Alpha-UCT maintains:

• The visit count for each action $N(v,a)$;
• The maximal value observed across all completed rollouts through $(v, a)$:

$$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$

where $\mathcal{T}(v, a)$ is the set of all simulated trajectories passing through $(v, a)$ and $V_k \in [-1,1]$ is the comparative judge's score for trajectory $k$;

• The total visits to the parent $N(v) = \sum_{b \in \mathcal{A}(v)} N(v, b)$.

The Alpha-UCT selection rule for choosing an action $a^*$ at node $N(v,a)$0 is: $N(v,a)$1 Here, $N(v,a)$2 is the exploration coefficient. The exploitation term $N(v,a)$3 prefers actions with the highest observed outcome; the exploration term provides a bonus inversely proportional to the exploration frequency, naturally handling dependent samples and maintaining optimism for lesser-explored actions (Tang et al., 3 Feb 2026).

2. Comparison to Classical UCT

Alpha-UCT departs from the classical UCT (Kocsis & Szepesvári, 2006) formulation in two principal ways:

• Exploitation: Classical UCT uses the empirical mean $N(v,a)$4 over all rollouts for an action; Alpha-UCT replaces this with the maximum observed value, $N(v,a)$5, positing that in open-ended tasks, the single best observed trajectory provides stronger evidence for branch pruning and rapid detection of promising directions.
• Exploration: Instead of the standard $N(v,a)$6 form, Alpha-UCT employs $N(v,a)$7, replacing $N(v,a)$8 with the total sibling visits and regularizing the denominator to reflect search effort, as derived from a martingale-based analysis.

These design changes bias the search towards rapid misstep detection and leverage the reduced residual variance in non-iid, reflection-informed evaluations (Tang et al., 3 Feb 2026).

3. Theoretical Properties and Regret Analysis

Alpha-UCT's regret analysis is based on the theory of martingale concentration, reflecting that the agent's samples are not iid due to recursive reflection and comparative evaluation. The analysis uses the conditional residual variance: $N(v,a)$9 where $(v, a)$0 is the reflection-based prior and $(v, a)$1 is the judge's score.

Applying Freedman's inequality for martingales, the confidence radius is of order $(v, a)$2, which yields the regret theorem: $(v, a)$3 where $(v, a)$4 is the gap to the optimal action (Tang et al., 3 Feb 2026).

In the case where $(v, a)$5 is the branching factor and $(v, a)$6, the regret scales as $(v, a)$7. Compared to standard UCT, which scales with the raw variance $(v, a)$8, Alpha-UCT achieves regret reduction proportional to $(v, a)$9, reflecting greater efficiency in scenarios where residual variance is tightly controlled.

4. Integration into Step-Level MCTS

Alpha-UCT is embedded into Agent Alpha’s step-level MCTS loop as follows:

$\mathcal{T}(v, a)$6

Within the loop, Alpha-UCT determines the selection policy at each decision point, employing a max-backup mechanism during back-propagation to propagate the highest observed evaluation up the tree (Tang et al., 3 Feb 2026).

5. Hyperparameter Guidelines and Empirical Observations

The authors provide empirical recommendations for key hyperparameters on the OSWorld benchmark:

• Expansion factor $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$0: Performance rapidly improves as $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$1 rises from 1 to 5, then saturates; $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$2 is recommended for optimal coverage and efficiency.
• Maximum MCTS iterations $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$3: Substantial gains up to $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$4 are observed, followed by a plateau; thus, $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$5 is advised for most applications.
• Action chunking (macro-action length): Chunk size of 5 assists with long-horizon navigation, whereas chunk size 7 degrades performance; moderate chunking ($$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$6) achieves a balance between foresight and recoverability.
• Exploitation strategy ablation: Replacing max-backup with mean-backup causes an 18.8% drop in success rate, underscoring the dominance of the maximum-based exploitation term for step-level search in these environments.
• Exploration coefficient $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$7: Although no explicit grid search is reported, $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$8 is treated analogously to UCT's and may be tuned within $$Q_{\max}(v, a) = \max_{k \in \mathcal{T}(v,a)} V_k,$$9 via standard bandit methodologies (Tang et al., 3 Feb 2026).

Hyperparameter	Empirical Trend	Recommended Value
Expansion factor $\mathcal{T}(v, a)$0	Steep gains up to 5, then saturates	$\mathcal{T}(v, a)$1
Max MCTS iterations $\mathcal{T}(v, a)$2	Strong effect up to 20, then plateau	$\mathcal{T}(v, a)$3
Chunk size	Moderate (≤5) balances performance	$\mathcal{T}(v, a)$4
Max vs. mean backup	Max-backup superior (SR ↑18.8%)	Use max-backup
Exploration coeff $\mathcal{T}(v, a)$5	Standard bandit tuning (not grid-searched)	[0.5, 2.0]

6. Context and Applicability

Alpha-UCT is designed for environments where trajectories are highly non-deterministic, feedback is non-iid, and the capacity for recovery from early errors or leveraging partial solutions is critical. Its application to complex GUI environments, as demonstrated in Agent Alpha, yields state-of-the-art performance, notably outperforming previous trajectory-level sampling baselines under identical computational constraints.

The approach is characterized by its integration of max-backup exploitation and a martingale-derived, data-dependent exploration bonus, offering both a rigorous theoretical foundation and empirically validated design for efficient deliberative planning (Tang et al., 3 Feb 2026).