SR-MCTS: Tree Search with Self-Refinement

• SR-MCTS is an advanced algorithm that combines Monte Carlo Tree Search with iterative self-refinement using LLM feedback for robust multi-step decision making.
• It incrementally decomposes complex tasks into intermediate candidate solutions, optimizing exploration–exploitation balance and enabling automated error correction.
• Empirical evaluations demonstrate that SR-MCTS outperforms baselines by achieving higher pass rates in code generation, improved SQL execution accuracy, and superior symbolic regression recovery.

Monte Carlo Tree Search with Self-Refinement (SR-MCTS) is an advanced algorithmic paradigm that combines the adaptive exploration-exploitation strategy of Monte Carlo Tree Search (MCTS) with iterative self-evaluation and refinement, typically orchestrated by LLMs. This closed-loop system is designed to address the limitations of conventional LLM-driven reasoning, planning, and program synthesis—especially on tasks where error-prone multi-step decomposition and irrevocable mistakes undermine single-shot or Chain-of-Thought (CoT) solutions. SR-MCTS operates across symbolic regression, code generation, software engineering, and complex query translation by structuring search as trees of intermediate reasoning or solution steps, with each node subject to autonomous critique and improvement.

1. Conceptual Foundations and Motivation

SR-MCTS extends MCTS by incorporating an internal feedback loop where the agent (usually an LLM or LLM ensemble) assigns reward signals and critiques to partial solutions during search. The core limitation in vanilla LLM inference—lack of mid-search error correction, limited diversity, and irreversible errors—is mitigated in SR-MCTS through:

• Incremental Decomposition: Solutions are constructed via diverse, stepwise plans or candidate actions, represented as paths in a search tree. Each branch explores alternative continuations from preliminary reasoning states.
• Self-Refinement Cycle: Each intermediate node is autonomously critiqued and assigned a quality/reward score. Erroneous or low-value nodes trigger reflection, refinement, and branching—distinguishing SR-MCTS from static CoT or self-consistency protocols (Xu et al., 17 Nov 2024, Yuan et al., 28 Jan 2025).
• Exploration–Exploitation Optimization: The search space is traversed under UCB/PUCT policies, integrating both current reward estimates and the statistical promise of underexplored paths (Li et al., 24 Jan 2024, Antoniades et al., 26 Oct 2024).

This architecture explicitly addresses the need for multi-step reasoning, resilience to error propagation, and diverse solution generation in disruptive planning domains.

2. Formal SR-MCTS Workflow

The standard SR-MCTS algorithm unfolds over four canonical phases, each specialized according to the domain (code, SQL, symbolic expression, repository-level reasoning):

1. Selection: Navigate from the root node toward a leaf node by recursively selecting child nodes maximizing an upper confidence bound, typically of the form:

$\mathrm{UCB} = r_i + c \sqrt{2 \ln N / n_i}$

or

$$\mathrm{UCT}(a) = P(a) + c \sqrt{\frac{\ln N(\mathrm{parent}(a)) + 1}{N(a) + \epsilon}}$$

where $r_i$ and $P(a)$ are reward/value estimates, $N$ and $n_i$ are parent/child visit counts, $c$ is exploration constant.

2. Expansion: For each selected leaf node, the agent proposes up to $B$ new candidate steps or refinements via LLM prompting or model-guided policy vectors:

$$\text{state}_{d+1}^{i} = \text{concatenate}(S_d, A_d),\quad A_{d+1}^i = \text{LLM}(S_{d+1}^i; R)$$

Duplicates are pruned, so only novel continuations propagate (Xu et al., 17 Nov 2024).

3. Simulation (Evaluation & Reflection): The agent assigns scalar reward scores based on correctness, coherence, completeness, or domain-specific execution:

$$r_{d+1} = \text{LLM}_\text{score}(S_{d+1} + A_{d+1}), \quad R = \begin{cases} \langle\text{end}\rangle & \text{solved} \ \text{guidance} & \text{otherwise} \end{cases}$$

In domains such as SQL or symbolic regression, evaluation is via execution accuracy (e.g., result on real database (Yuan et al., 28 Jan 2025), regression fit (Li et al., 24 Jan 2024)), with rewards ranging from hard negatives (syntax error) to full credit.

4. Backpropagation: Reward increments are weighted across visits for robust value estimation:

$$r_{\text{inc}} = \frac{\sum_{i=1}^B \text{Visits}_i \cdot r_i}{\sum_{i=1}^B \text{Visits}_i}$$

with parent updating:

$$r_\text{parent} \leftarrow \alpha r_\text{parent} + (1-\alpha) r_{\text{inc}}$$

or via domain-specific smoothing formulas:

$$P(a) = \tfrac{1}{2}(\min_i r_a^i + \tfrac{1}{|r_a|} \sum_i r_a^i),\quad P'(a) = \tfrac{1}{2}(P(a) + \max_{i \in \mathrm{Children}(a)} P(i))$$

Tree traversal repeats until either a terminal solution is found, maximum iteration/wall-clock constraints are met, or quality thresholds are satisfied. Final outputs are typically extracted from the best-reward branch (code plan, query, formula, or patch).

3. Domain-Specific Instantiations

SR-MCTS has been deployed in several research areas with concrete instantiation variations:

Code Generation

SRA-MCTS (Xu et al., 17 Nov 2024) operates by generating (problem description, natural-language plan, final code) triples for tasks such as LeetCode and MBPP+. The search tree encodes natural-language reasoning only—code synthesis is deferred until the best plan is extracted. Post-hoc unit tests validate output code and inform selective retraining.

Text-to-SQL Translation

MCTS-SQL (Yuan et al., 28 Jan 2025) extends SR-MCTS to SQL query synthesis, incorporating three LLM roles—Critiquer (diagnoses errors), Refiner (suggests corrections), and Evaluator (assigns execution-based and semantic rewards). The tree structure explores candidate queries via iterative self-refinement, guided by execution error feedback and LLM-verification.

Symbolic Regression

SR-GPT (Li et al., 24 Jan 2024) couples MCTS with a GPT policy to guide symbolic formula assembly. The policy and value networks optimize via MCTS rollouts, storing encountered state-policy-reward tuples for continual retraining. The objective function is a composite of value regression, cross-entropy against improved MCTS visit-distributions, entropy regularization, and L2 weight decay.

Repository-Level Software Engineering

SWE-Search (Antoniades et al., 26 Oct 2024) treats software engineering as a non-Markovian planning space with git-backed states, hierarchical actions, and hybrid value estimation (numerical and qualitative LLM feedback). The framework integrates Action, Value, and Discriminator Agents, enabling iterative solution edits, testing, debate, and intelligent backtracking as driven by LLM feedback.

4. Self-Refinement Mechanisms

The self-refinement loop is the defining extension of SR-MCTS:

• Autonomous Critique and Rewarding: Intermediate steps/partial solutions are scored for correctness, completeness, and coherence. In the presence of execution errors or flagged coverage gaps, the agent generates new branches via targeted reflection prompts. For symbolic regression, policy entropy and value regression regularize self-improvement (Li et al., 24 Jan 2024).
• Hindsight-Based Re-Expansion: Nodes with unsatisfactory reward or flagged limitations are refined by injecting critique feedback into expansion prompts, driving further tree growth at critical points (as in SWE-Search's loop which prioritizes coverage and diversity (Antoniades et al., 26 Oct 2024)).
• Qualitative and Quantitative Feedback: Hybrid value functions combine numeric test-passing rates with language-derived bonuses/penalties, allowing comprehensive trajectory assessment and prioritization.

A plausible implication is that SR-MCTS guides models toward more human-like iterative planning and error correction, surpassing static search protocols in robustness and reliability.

5. Empirical Evaluation and Ablation Studies

SR-MCTS exhibits substantial gains over vanilla LLM and MCTS baselines:

Domain	SR-MCTS Benchmark	Metric	SR-MCTS Performance	Baseline Performance	Effect of Self-Refinement
Code Generation	Human-Eval+, MBPP+ (Xu et al., 17 Nov 2024)	pass@10	+5.5 pts (8B), +3.3 (14B)	Instruct baseline	Remove plans: −13 pts pass@10 (14B), −7 (8B)
Text-to-SQL	BIRD, Spider (Yuan et al., 28 Jan 2025)	Execution Accuracy	69.40% (GPT-4o, dev)	63.36% vanilla MCTS	Ablation: 57.96% without MCTS-Refiner
Symbolic Regression	Nguyen, others (Li et al., 24 Jan 2024)	Full-recovery rate	74.9% (SR-GPT)	68.5% (DGSR-MCTS), 63.1% (SPL)	Remove entropy/constr: drops recovery to ≈10%
SWE Tasks	SWE-bench Lite (Antoniades et al., 26 Oct 2024)	Pass@1/Pass@5	+23% mean relative gain	Open-source agent baseline	Value Agent alone: 73% correct leaf, +11% w/ Discrim

Ablation studies confirm that natural-language reasoning steps, explicit critique/refinement, entropy regularization, and domain-specific reward functions are critical for SR-MCTS stability and diversity.

6. Algorithmic and Architectural Details

SR-MCTS implementations typically employ the following components:

• Expansion fan-out: $B=3$ or domain-specific, balancing diversity and computational cost.
• Exploration/exploitation constants: $c=0.5$ (code), $C_{puct}=1.0$ (SR-GPT), backprop weights $\alpha=0.7$.
• Training regime: Fine-tuning on context-plan-solution triples, replay buffers (SR-GPT), or LoRA adapters (Xu et al., 17 Nov 2024).
• Modular agent architecture: Specialized LLM agents for action selection, value estimation, multi-agent debate, and critique.
• Flexible state representation: Git commit pointers for software engineering (SWE-Search), symbolic trees for regression, plan chains for code.

Termination criteria include explicit $\langle\text{end}\rangle$ tokens, max iterations, or reward thresholds.

7. Variants, Limitations, and Prospects

SR-MCTS’s key strengths are structured diversity, iterative self-correction, and superior handling of error-prone multi-step planning. The approach remains robust when single-shot CoT or self-consistency methods degrade. Limitations include increased inference-time compute and potential sensitivity to model-generated reward signals or critique quality. Scaling the depth of search and integrating multi-agent ensembles (as in SWE-Search debates) yields additive improvements.

This suggests that further research into search-guided training, hybrid reward functions, and trajectory-aware feedback in LLMs could substantially enhance the reliability and expressivity of automated reasoning and synthesis in complex tasks. SR-MCTS now provides a framework for principled, model-driven feedback integration in sequential decision-making, with empirical superiority confirmed across benchmarks in code generation, SQL synthesis, symbolic regression, and adaptive software engineering.