MCTS-Judge: Code Evaluation via Test-Time MCTS

• MCTS-Judge is a framework that applies Monte Carlo Tree Search to decompose code evaluation into sequential sub-tasks.
• It employs a hybrid node-selection rule that combines UCT rollout statistics and LLM self-assessment for accurate judgments.
• The method demonstrates notable accuracy improvements and token efficiency, establishing a test-time scaling law for LLM evaluators.

Searching arXiv for the specified paper and closely related MCTS-as-a-judge work to ground the article in current literature. MCTS-Judge is a framework for code correctness evaluation within the LLM-as-a-Judge paradigm that introduces test-time computation through Monte Carlo Tree Search (MCTS). It was presented as “the first ‘test-time computation’ framework for LLM-as-a-Judge in code-correctness evaluation” and is explicitly designed for reasoning-intensive judging settings where single-pass prompting is unreliable (Wang et al., 18 Feb 2025). Rather than issuing a single holistic verdict, MCTS-Judge treats the judgment process as a search over partial reasoning trajectories, decomposing the question of whether code is correct into a sequence of simpler sub-evaluations and scoring those trajectories with a unit-test-level reward mechanism (Wang et al., 18 Feb 2025). In this formulation, the framework aligns judge-time deliberation with broader test-time scaling trends in reasoning models, while remaining distinct from earlier MCTS applications in constrained generation and preference-based search (Chaffin et al., 2021, Joppen et al., 2018).

1. Position within LLM-as-a-Judge research

Traditional LLM-as-a-Judge methods for code evaluation, including ICE-Score and CodeJudge, are described as relying on single-pass, “System-1” prompting, which is fast but susceptible to superficial or biased judgments on programming tasks (Wang et al., 18 Feb 2025). MCTS-Judge was introduced in response to this limitation, motivated by the observation that state-of-the-art reasoning models improve reliability by allocating extra computation at inference time through mechanisms such as beam search, self-consistency, or search (Wang et al., 18 Feb 2025).

The central premise is that judging code correctness is itself a structured reasoning problem. MCTS-Judge therefore reformulates evaluation as sequential decision-making over subtasks such as logic checks, boundary-case checks, and complexity checks, using MCTS to organize exploration of alternative reasoning orders and partial analyses (Wang et al., 18 Feb 2025). This distinguishes it from ordinary judge prompting, where the entire evaluative burden is compressed into a single response.

A useful comparison is with earlier MCTS-based language-model methods. “PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding” formalizes constrained generation as tree exploration in which a discriminator supplies long-term rewards (Chaffin et al., 2021). “Preference-Based Monte Carlo Tree Search” replaces numeric rewards with pairwise preference signals in sequential search (Joppen et al., 2018). MCTS-Judge differs in immediate objective—it is not decoding task outputs but evaluating candidate code—but it shares the broader design pattern of using tree search to compensate for the limitations of local greedy decisions. This suggests that MCTS-Judge is best understood as a judge-specific adaptation of test-time search rather than a generic prompting heuristic.

2. Formalization of the judgment process

In MCTS-Judge, each node in the search tree corresponds to a partial reasoning trajectory over the original code evaluation problem $x=(c,p)$, where $c$ is code and $p$ is the programming problem (Wang et al., 18 Feb 2025). A state $s$ is defined as a sequence of previously executed sub-evaluations $(s_1,\dots,s_{i-1})$ together with the original problem. An action $a$ is to choose one remaining subtask, such as checking null-pointer handling, or to choose the null action for early termination (Wang et al., 18 Feb 2025).

The transition operator is implemented by the LLM itself:

$s' = LLM(x, s, a)$

where the model executes the chosen subtask on the code and problem, producing either a binary outcome or a short analysis snippet that is then binarized (Wang et al., 18 Feb 2025). This makes the search tree a tree over evaluative reasoning, not over program executions or output candidates.

The framework is parameterized by a maximum depth $K$, a number of rollouts $R$, a UCT exploration constant $c$, an LLM self-assessment weight $c$0, and a UCT weight $c$1 (Wang et al., 18 Feb 2025). The depth controls how many sub-evaluations can appear in a trajectory, while the rollout count determines how extensively the search explores alternatives. The paper also states that each rollout costs $c$2 LLM calls and that $c$3 rollouts cost $c$4, making the computational profile explicit (Wang et al., 18 Feb 2025).

This state-action design operationalizes the idea that code evaluation can be factorized into orthogonal diagnostic checks. A plausible implication is that the method gains robustness not merely by “thinking longer,” but by enforcing structural diversity in the reasoning path.

3. Search procedure and node-selection rule

MCTS-Judge follows the standard MCTS phases of selection, expansion, simulation, and backpropagation, but introduces a hybrid node-selection rule that combines global rollout statistics with local self-assessment (Wang et al., 18 Feb 2025).

During selection, for each child $c$5 of a node $c$6, the method computes

$c$7

and

$c$8

which are combined as

$c$9

with ties broken at random (Wang et al., 18 Feb 2025). The paper also gives the UCT term in conventional notation,

$p$0

where $p$1, $p$2, and $p$3 is the visit count of the parent; it states that the paper used $p$4 as the exploration constant (Wang et al., 18 Feb 2025).

The self-assessment term is binary. The LLM is prompted with a question of the form “Will adding subtask $p$5 now improve overall evaluation?”; “Yes” maps to 1 and “No” maps to 0 (Wang et al., 18 Feb 2025). This is not a conventional MCTS component. The authors describe it as the crucial “global-local” twist: UCT captures long-run value from prior rollouts, while self-assessment biases search toward locally promising next checks within the current reasoning trajectory (Wang et al., 18 Feb 2025).

Expansion samples one unused non-null subtask, executes it via the LLM, and records the resulting binary outcome (Wang et al., 18 Feb 2025). Simulation then continues by sampling non-null subtasks randomly until depth $p$6 is reached (Wang et al., 18 Feb 2025). Backpropagation increments both visit counts and cumulative reward along the root-to-leaf path (Wang et al., 18 Feb 2025). The final output is the trajectory among all rollouts with the highest cumulative reward (Wang et al., 18 Feb 2025).

Relative to prior MCTS formulations, MCTS-Judge’s selection rule is more judge-centric than token-centric. PPL-MCTS uses a PUCT term biased by next-token probabilities from the base LM (Chaffin et al., 2021), whereas MCTS-Judge uses an explicit meta-evaluative query to the same LLM. This suggests a shift from language-model prior guidance to judgment-process guidance.

4. Reward design and line-by-line analysis

A defining feature of MCTS-Judge is its “high-precision, unit-test-level reward mechanism,” designed to encourage line-by-line analysis rather than superficial summarization (Wang et al., 18 Feb 2025). At the end of a trajectory $p$7, the framework computes two binary quantities. The first is the trajectory’s verdict $p$8, described as the binary verdict of the trajectory based on majority vote of subtasks plus a global decision (Wang et al., 18 Feb 2025). The second is a simulated execution result $p$9, obtained by sampling $s$0 test cases, simulating each $s$1 times, taking a majority vote on each case, and returning “Yes” only if all $s$2 pass (Wang et al., 18 Feb 2025).

The terminal reward is

$s$3

and each visited node’s cumulative value is incremented by this reward during backpropagation (Wang et al., 18 Feb 2025). The simulation-guided reward is explicitly intended to steer the judge toward detailed, execution-aware reasoning (Wang et al., 18 Feb 2025).

This reward structure differs from earlier MCTS-based language-model systems. In PPL-MCTS, the reward is a weighted combination of language-model likelihood and discriminator score over a completed sequence (Chaffin et al., 2021). In MCTS-Judge, by contrast, the reward is tied to agreement between the judge’s evaluative trajectory and simulated execution outcomes (Wang et al., 18 Feb 2025). The result is a search process whose objective is not stylistic or semantic conformity, but alignment between reasoning-derived judgment and execution-based evidence.

The paper’s description of the reward also clarifies the framework’s intended precision. Because the simulated execution result is aggregated at unit-test level, the judge is incentivized to attend to concrete code behavior rather than broad plausibility. A plausible implication is that this reduces the common failure mode in LLM judging where persuasive but underspecified analysis is mistaken for correctness.

5. Test-time scaling law

A major claim of MCTS-Judge is that it reveals a test-time scaling law for the LLM-as-a-Judge paradigm (Wang et al., 18 Feb 2025). The paper reports empirical sweeps over four “compute knobs”: the number of test cases $s$4, the number of simulations $s$5 per case, tree depth $s$6, and the number of rollouts $s$7 (Wang et al., 18 Feb 2025). On APPS, increasing $s$8 from $s$9 raises accuracy from approximately $(s_1,\dots,s_{i-1})$0; increasing $(s_1,\dots,s_{i-1})$1 from $(s_1,\dots,s_{i-1})$2 adds approximately $(s_1,\dots,s_{i-1})$3–$(s_1,\dots,s_{i-1})$4; extending depth from $(s_1,\dots,s_{i-1})$5 adds approximately $(s_1,\dots,s_{i-1})$6–$(s_1,\dots,s_{i-1})$7; and increasing rollouts from $(s_1,\dots,s_{i-1})$8 yields positive but diminishing gains (Wang et al., 18 Feb 2025).

The authors summarize these trends as monotonically increasing accuracy under increased inference-time budget and propose an approximate sub-power-law relation between evaluation accuracy $(s_1,\dots,s_{i-1})$9 and total test-time budget $a$0:

$a$1

while noting that no closed-form exponent is fit in the paper (Wang et al., 18 Feb 2025). They characterize the observed improvements as consistent and concave-up (Wang et al., 18 Feb 2025).

The same source also states that empirically the method reaches stable high accuracy with $a$2, and that doubling $a$3 yields only marginal gains, which is presented as evidence of rapid convergence (Wang et al., 18 Feb 2025). In conjunction with the $a$4 cost profile, this positions MCTS-Judge as a resource-aware search method rather than a brute-force deliberation scheme.

The broader significance is methodological. Much of the earlier discussion of scaling in LLM systems focused on train-time scaling. MCTS-Judge frames judgment quality itself as something that can scale at inference through structured search (Wang et al., 18 Feb 2025). This suggests a conceptual shift in evaluation research: judge reliability may be improved not only by better judge models, but also by more capable judge-time inference procedures.

6. Benchmarks, models, and empirical performance

The paper evaluates MCTS-Judge on three benchmarks: BigCodeBench, consisting of 1,140 real-world Python tasks; APPS, with 100 competition-level Python tasks; and HumanEval-X, with 164 tasks across Python, Java, C++, JavaScript, and Go (Wang et al., 18 Feb 2025). The evaluated base models include DeepSeek-Coder-V2-16B, Qwen2.5-Coder-14B, Mistral-Codestral-22B, Llama-3.1-8B, and GPT-4o-mini; comparisons also include o1-preview, o1-mini, and Qwen-QwQ-32B (Wang et al., 18 Feb 2025).

On APPS with DeepSeek-Coder-V2-16B, the reported results are as follows:

Method	Accuracy / Tokens
Vanilla	41.0%
ICE-Score	48.0%
CodeJudge	62.0%
MCTS-Judge	80.0%
o1-preview	75.0%, uses $a$5 reasoning tokens
MCTS-Judge w/ DeepSeek	80.0% using only 2065 reasoning tokens and a 16 B model

These figures are reported directly in the paper summary (Wang et al., 18 Feb 2025). The paper also states that, on BigCodeBench and HumanEval-X, MCTS-Judge lifts all bases by approximately 15–30 percentage points and often outperforms o1-series models and Qwen-QwQ-32B by 5–20 percentage points while using fewer tokens and smaller models (Wang et al., 18 Feb 2025).

The authors further evaluate reasoning trajectory quality using GPT-4o as a side-by-side judge over four dimensions: Thoroughness, Logic, Analysis, and Overall quality (Wang et al., 18 Feb 2025). Against o1-preview, o1-mini, and Qwen-QwQ, MCTS-Judge trajectories reportedly win 60–80% of the time on Thoroughness and Analysis, and approximately 50–60% on Logic and Overall (Wang et al., 18 Feb 2025). The intended interpretation is that search-based judging produces more complete and less superficial justifications.

These results position MCTS-Judge as both an accuracy improvement and a trajectory-quality improvement. However, the trajectory-quality assessment itself is conducted by another model judge, GPT-4o (Wang et al., 18 Feb 2025). This does not invalidate the result, but it indicates that some of the evidence concerns comparative quality of explanations as judged by an LLM rather than by direct human annotation.

7. Relation to adjacent MCTS-based judge frameworks

MCTS-Judge belongs to a broader movement that uses tree search to strengthen judgment or control mechanisms around LLMs, but its role is specific. PPL-MCTS applies MCTS to constrained text generation, where nodes correspond to partial token sequences and a discriminator provides terminal rewards (Chaffin et al., 2021). Preference-Based MCTS addresses cases where feedback is ordinal rather than numeric, maintaining pairwise win statistics and using Relative UCB to select competing actions (Joppen et al., 2018). Judge-MCTS, introduced in multimodal judging, uses MCTS to generate pairwise reasoning trajectories of varying correctness and length for data construction and judge-model training rather than for direct test-time verdict selection (Chen et al., 28 Feb 2026).

A concise comparison is useful.

Framework	Primary objective	Feedback signal
MCTS-Judge	Code correctness evaluation at test time	Simulated unit-test-level reward
PPL-MCTS	Constrained textual generation	Discriminator-guided sequence reward
Preference-Based MCTS	Sequential search with ordinal feedback	Pairwise preferences
Judge-MCTS	Multimodal judge-data generation	Correctness and length-based scalar reward

MCTS-Judge is therefore distinguished by two properties. First, it operates directly in the judge loop at inference time, rather than in generation or offline data synthesis (Wang et al., 18 Feb 2025). Second, it targets code correctness evaluation specifically, where simulated execution can serve as a reward source (Wang et al., 18 Feb 2025). Judge-MCTS in multimodal evaluation also searches over partial reasoning states, but it is used to generate supervisory pairs such as SC, SE, LC, and LE for subsequent SFT and RL training of M-Judger, not to adjudicate one instance online (Chen et al., 28 Feb 2026).

This comparison also helps delimit common misconceptions. MCTS-Judge is not merely “chain-of-thought plus search,” nor is it identical to generic UCT prompting. Its distinctive contribution is the integration of subtask decomposition, a hybrid self-assessment-plus-UCT node policy, and a simulated-execution reward tailored to code evaluation (Wang et al., 18 Feb 2025). A plausible implication is that transferring the framework to domains without an analogous high-precision reward source may require substantial redesign, even if the search skeleton remains applicable.