---
title: 'MCTS-Judge: Code Evaluation via Test-Time MCTS'
url: https://www.emergentmind.com/topics/mcts-judge
type: topic
---

# MCTS-Judge: Code Evaluation via Test-Time MCTS

Searching arXiv for the specified paper and closely related MCTS-as-a-judge work to ground the article in current literature.
MCTS-Judge is a framework for code correctness evaluation within the LLM-as-a-Judge paradigm that introduces test-time computation through Monte Carlo Tree Search (MCTS). It was presented as “the first ‘test-time computation’ framework for LLM-as-a-Judge in code-correctness evaluation” and is explicitly designed for reasoning-intensive judging settings where single-pass prompting is unreliable [2502.12468]. Rather than issuing a single holistic verdict, MCTS-Judge treats the judgment process as a search over partial reasoning trajectories, decomposing the question of whether code is correct into a sequence of simpler sub-evaluations and scoring those trajectories with a unit-test-level reward mechanism [2502.12468]. In this formulation, the framework aligns judge-time deliberation with broader test-time scaling trends in reasoning models, while remaining distinct from earlier MCTS applications in constrained generation and preference-based search [2109.13582] [1807.06286].

## 1. Position within LLM-as-a-Judge research

Traditional LLM-as-a-Judge methods for code evaluation, including ICE-Score and CodeJudge, are described as relying on single-pass, “System-1” prompting, which is fast but susceptible to superficial or biased judgments on programming tasks [2502.12468]. MCTS-Judge was introduced in response to this limitation, motivated by the observation that state-of-the-art reasoning models improve reliability by allocating extra computation at inference time through mechanisms such as beam search, self-consistency, or search [2502.12468].

The central premise is that judging code correctness is itself a structured reasoning problem. MCTS-Judge therefore reformulates evaluation as sequential decision-making over subtasks such as logic checks, boundary-case checks, and complexity checks, using MCTS to organize exploration of alternative reasoning orders and partial analyses [2502.12468]. This distinguishes it from ordinary judge prompting, where the entire evaluative burden is compressed into a single response.

A useful comparison is with earlier MCTS-based language-model methods. “PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding” formalizes constrained generation as tree exploration in which a discriminator supplies long-term rewards [2109.13582]. “Preference-Based Monte Carlo Tree Search” replaces numeric rewards with pairwise preference signals in sequential search [1807.06286]. MCTS-Judge differs in immediate objective—it is not decoding task outputs but evaluating candidate code—but it shares the broader design pattern of using tree search to compensate for the limitations of local greedy decisions. This suggests that MCTS-Judge is best understood as a judge-specific adaptation of test-time search rather than a generic prompting heuristic.

## 2. Formalization of the judgment process

In MCTS-Judge, each node in the search tree corresponds to a partial reasoning trajectory over the original code evaluation problem \(x=(c,p)\), where \(c\) is code and \(p\) is the programming problem [2502.12468]. A state \(s\) is defined as a sequence of previously executed sub-evaluations \((s_1,\dots,s_{i-1})\) together with the original problem. An action \(a\) is to choose one remaining subtask, such as checking null-pointer handling, or to choose the null action for early termination [2502.12468].

The transition operator is implemented by the language model itself:

\[
s' = LLM(x, s, a)
\]

where the model executes the chosen subtask on the code and problem, producing either a binary outcome or a short analysis snippet that is then binarized [2502.12468]. This makes the search tree a tree over evaluative reasoning, not over program executions or output candidates.

The framework is parameterized by a maximum depth \(K\), a number of rollouts \(R\), a UCT exploration constant \(c\), an LLM self-assessment weight \(w_l\), and a UCT weight \(w_u\) [2502.12468]. The depth controls how many sub-evaluations can appear in a trajectory, while the rollout count determines how extensively the search explores alternatives. The paper also states that each rollout costs \(O(K)\) LLM calls and that \(R\) rollouts cost \(O(R \cdot K)\), making the computational profile explicit [2502.12468].

This state-action design operationalizes the idea that code evaluation can be factorized into orthogonal diagnostic checks. A plausible implication is that the method gains robustness not merely by “thinking longer,” but by enforcing structural diversity in the reasoning path.

## 3. Search procedure and node-selection rule

MCTS-Judge follows the standard MCTS phases of selection, expansion, simulation, and backpropagation, but introduces a hybrid node-selection rule that combines global rollout statistics with local self-assessment [2502.12468].

During selection, for each child \(u\) of a node \(v\), the method computes

\[
UCT(u)=Q(u)/N(u) + c\cdot\sqrt{\ln N(v)/N(u)}
\]

and

\[
S(u)=LLM\text{-}SelfAssess(u \mid current\text{-}trajectory)\in\{0,1\}
\]

which are combined as

\[
Score(u)=w_u\cdot UCT(u)+w_l\cdot S(u)
\]

with ties broken at random [2502.12468]. The paper also gives the UCT term in conventional notation,

\[
UCT(i)=\bar X_i + c\sqrt{\frac{\ln N}{n_i}}
\]

where \(\bar X_i=Q(i)/n_i\), \(n_i=N(i)\), and \(N\) is the visit count of the parent; it states that the paper used \(\alpha=3\) as the exploration constant [2502.12468].

The self-assessment term is binary. The LLM is prompted with a question of the form “Will adding subtask \(a\) now improve overall evaluation?”; “Yes” maps to 1 and “No” maps to 0 [2502.12468]. This is not a conventional MCTS component. The authors describe it as the crucial “global-local” twist: UCT captures long-run value from prior rollouts, while self-assessment biases search toward locally promising next checks within the current reasoning trajectory [2502.12468].

Expansion samples one unused non-null subtask, executes it via the LLM, and records the resulting binary outcome [2502.12468]. Simulation then continues by sampling non-null subtasks randomly until depth \(K\) is reached [2502.12468]. Backpropagation increments both visit counts and cumulative reward along the root-to-leaf path [2502.12468]. The final output is the trajectory among all rollouts with the highest cumulative reward [2502.12468].

Relative to prior MCTS formulations, MCTS-Judge’s selection rule is more judge-centric than token-centric. PPL-MCTS uses a PUCT term biased by next-token probabilities from the base LM [2109.13582], whereas MCTS-Judge uses an explicit meta-evaluative query to the same LLM. This suggests a shift from language-model prior guidance to judgment-process guidance.

## 4. Reward design and line-by-line analysis

A defining feature of MCTS-Judge is its “high-precision, unit-test-level reward mechanism,” designed to encourage line-by-line analysis rather than superficial summarization [2502.12468]. At the end of a trajectory \(t\), the framework computes two binary quantities. The first is the trajectory’s verdict \(f(t,g)\in\{Yes,No\}\), described as the binary verdict of the trajectory based on majority vote of subtasks plus a global decision [2502.12468]. The second is a simulated execution result \(h(x)\in\{Yes,No\}\), obtained by sampling \(\gamma\) test cases, simulating each \(\delta\) times, taking a majority vote on each case, and returning “Yes” only if all \(\gamma\) pass [2502.12468].

The terminal reward is

\[
R(t)=\epsilon \mathbf{1}\{f(t,g)=h(x)\},\quad \epsilon>1
\]

and each visited node’s cumulative value is incremented by this reward during backpropagation [2502.12468]. The simulation-guided reward is explicitly intended to steer the judge toward detailed, execution-aware reasoning [2502.12468].

This reward structure differs from earlier MCTS-based language-model systems. In PPL-MCTS, the reward is a weighted combination of language-model likelihood and discriminator score over a completed sequence [2109.13582]. In MCTS-Judge, by contrast, the reward is tied to agreement between the judge’s evaluative trajectory and simulated execution outcomes [2502.12468]. The result is a search process whose objective is not stylistic or semantic conformity, but alignment between reasoning-derived judgment and execution-based evidence.

The paper’s description of the reward also clarifies the framework’s intended precision. Because the simulated execution result is aggregated at unit-test level, the judge is incentivized to attend to concrete code behavior rather than broad plausibility. A plausible implication is that this reduces the common failure mode in LLM judging where persuasive but underspecified analysis is mistaken for correctness.

## 5. Test-time scaling law

A major claim of MCTS-Judge is that it reveals a test-time scaling law for the LLM-as-a-Judge paradigm [2502.12468]. The paper reports empirical sweeps over four “compute knobs”: the number of test cases \(\alpha\), the number of simulations \(\delta\) per case, tree depth \(K\), and the number of rollouts \(R\) [2502.12468]. On APPS, increasing \(\alpha\) from \(1 \to 5 \to 10\) raises accuracy from approximately \(65\% \to 75\% \to 78\%\); increasing \(\delta\) from \(1 \to 7\) adds approximately \(3\)–\(5\%\); extending depth from \(5 \to 9 \to 12\) adds approximately \(2\)–\(4\%\); and increasing rollouts from \(R=4 \to 8 \to 16\) yields positive but diminishing gains [2502.12468].

The authors summarize these trends as monotonically increasing accuracy under increased inference-time budget and propose an approximate sub-power-law relation between evaluation accuracy \(A\) and total test-time budget \(B\):

\[
A(B)\approx 1 - \alpha B^{-\beta},\quad \beta\in(0,1)
\]

while noting that no closed-form exponent is fit in the paper [2502.12468]. They characterize the observed improvements as consistent and concave-up [2502.12468].

The same source also states that empirically the method reaches stable high accuracy with \(R=8, K=9\), and that doubling \(R\) yields only marginal gains, which is presented as evidence of rapid convergence [2502.12468]. In conjunction with the \(O(R\cdot K)\) cost profile, this positions MCTS-Judge as a resource-aware search method rather than a brute-force deliberation scheme.

The broader significance is methodological. Much of the earlier discussion of scaling in LLM systems focused on train-time scaling. MCTS-Judge frames judgment quality itself as something that can scale at inference through structured search [2502.12468]. This suggests a conceptual shift in evaluation research: judge reliability may be improved not only by better judge models, but also by more capable judge-time inference procedures.

## 6. Benchmarks, models, and empirical performance

The paper evaluates MCTS-Judge on three benchmarks: BigCodeBench, consisting of 1,140 real-world Python tasks; APPS, with 100 competition-level Python tasks; and HumanEval-X, with 164 tasks across Python, Java, C++, JavaScript, and Go [2502.12468]. The evaluated base models include DeepSeek-Coder-V2-16B, Qwen2.5-Coder-14B, Mistral-Codestral-22B, Llama-3.1-8B, and GPT-4o-mini; comparisons also include o1-preview, o1-mini, and Qwen-QwQ-32B [2502.12468].

On APPS with DeepSeek-Coder-V2-16B, the reported results are as follows:

| Method | Accuracy / Tokens |
|---|---|
| Vanilla | 41.0% |
| ICE-Score | 48.0% |
| CodeJudge | 62.0% |
| MCTS-Judge | 80.0% |
| o1-preview | 75.0%, uses \(\sim 5631\) reasoning tokens |
| MCTS-Judge w/ DeepSeek | 80.0% using only 2065 reasoning tokens and a 16 B model |

These figures are reported directly in the paper summary [2502.12468]. The paper also states that, on BigCodeBench and HumanEval-X, MCTS-Judge lifts all bases by approximately 15–30 percentage points and often outperforms o1-series models and Qwen-QwQ-32B by 5–20 percentage points while using fewer tokens and smaller models [2502.12468].

The authors further evaluate reasoning trajectory quality using GPT-4o as a side-by-side judge over four dimensions: Thoroughness, Logic, Analysis, and Overall quality [2502.12468]. Against o1-preview, o1-mini, and Qwen-QwQ, MCTS-Judge trajectories reportedly win 60–80% of the time on Thoroughness and Analysis, and approximately 50–60% on Logic and Overall [2502.12468]. The intended interpretation is that search-based judging produces more complete and less superficial justifications.

These results position MCTS-Judge as both an accuracy improvement and a trajectory-quality improvement. However, the trajectory-quality assessment itself is conducted by another model judge, GPT-4o [2502.12468]. This does not invalidate the result, but it indicates that some of the evidence concerns comparative quality of explanations as judged by an LLM rather than by direct human annotation.

## 7. Relation to adjacent MCTS-based judge frameworks

MCTS-Judge belongs to a broader movement that uses tree search to strengthen judgment or control mechanisms around language models, but its role is specific. PPL-MCTS applies MCTS to constrained text generation, where nodes correspond to partial token sequences and a discriminator provides terminal rewards [2109.13582]. Preference-Based MCTS addresses cases where feedback is ordinal rather than numeric, maintaining pairwise win statistics and using Relative UCB to select competing actions [1807.06286]. Judge-MCTS, introduced in multimodal judging, uses MCTS to generate pairwise reasoning trajectories of varying correctness and length for data construction and judge-model training rather than for direct test-time verdict selection [2603.00546].

A concise comparison is useful.

| Framework | Primary objective | Feedback signal |
|---|---|---|
| MCTS-Judge | Code correctness evaluation at test time | Simulated unit-test-level reward |
| PPL-MCTS | Constrained textual generation | Discriminator-guided sequence reward |
| Preference-Based MCTS | Sequential search with ordinal feedback | Pairwise preferences |
| Judge-MCTS | Multimodal judge-data generation | Correctness and length-based scalar reward |

MCTS-Judge is therefore distinguished by two properties. First, it operates directly in the judge loop at inference time, rather than in generation or offline data synthesis [2502.12468]. Second, it targets code correctness evaluation specifically, where simulated execution can serve as a reward source [2502.12468]. Judge-MCTS in multimodal evaluation also searches over partial reasoning states, but it is used to generate supervisory pairs such as SC, SE, LC, and LE for subsequent SFT and RL training of M-Judger, not to adjudicate one instance online [2603.00546].

This comparison also helps delimit common misconceptions. MCTS-Judge is not merely “chain-of-thought plus search,” nor is it identical to generic UCT prompting. Its distinctive contribution is the integration of subtask decomposition, a hybrid self-assessment-plus-UCT node policy, and a simulated-execution reward tailored to code evaluation [2502.12468]. A plausible implication is that transferring the framework to domains without an analogous high-precision reward source may require substantial redesign, even if the search skeleton remains applicable.

Source: https://www.emergentmind.com/topics/mcts-judge