Empirical-MCTS: Continuous Agent Evolution via Dual-Experience Monte Carlo Tree Search

Abstract: Inference-time scaling strategies, particularly Monte Carlo Tree Search (MCTS), have significantly enhanced the reasoning capabilities of LLMs. However, current approaches remain predominantly stateless, discarding successful reasoning patterns after each problem instance and failing to mimic the empirical accumulation of wisdom characteristic of human problem-solving. To bridge this gap, we introduce Empirical-MCTS, a dual-loop framework that transforms stateless search into a continuous, non-parametric learning process. The framework unifies local exploration with global memory optimization through two novel mechanisms: Pairwise-Experience-Evolutionary Meta-Prompting (PE-EMP) and a Memory Optimization Agent. PE-EMP functions as a reflexive optimizer within the local search, utilizing pairwise feedback to dynamically synthesize adaptive criteria and evolve meta-prompts (system prompts) in real-time. Simultaneously, the Memory Optimization Agent manages a global repository as a dynamic policy prior, employing atomic operations to distill high-quality insights across problems. Extensive evaluations on complex reasoning benchmarks, including AIME25, ARC-AGI-2, and MathArena Apex, demonstrate that Empirical-MCTS significantly outperforms both stateless MCTS strategies and standalone experience-driven agents. These results underscore the critical necessity of coupling structured search with empirical accumulation for mastering complex, open-ended reasoning tasks.

• The paper introduces a dual-loop MCTS system that integrates local experiential adaptation with global memory optimization to evolve reasoning strategies.
• It employs pairwise comparative meta-prompting and robust reward fusion to enhance accuracy and cost-performance on benchmarks like AIME25.
• Empirical evaluations show that the framework outperforms stateless models by leveraging cumulative experience for continuous agent evolution.

Empirical-MCTS: Continuous Agent Evolution via Dual-Experience Monte Carlo Tree Search

Introduction and Motivation

Empirical-MCTS presents a conceptual and architectural shift in leveraging Monte Carlo Tree Search (MCTS) for LLM (LM) inference. Recognizing the critical deficiency in existing inference-time scaling and tree search approaches—the lack of cross-instance empirical accumulation—the framework introduces a non-parametric, dual-loop system that unifies structured search with online memory, allowing LMs to "remember" and reuse high-value reasoning strategies across diverse problem distributions. This bridges the divide between the stateless, myopic agentic behaviors typical of contemporary LLM search and the empirical, wisdom-driven problem-solving characteristic of human experts.

Framework Overview: Dual-Loop Architecture

The core innovation of Empirical-MCTS lies in its integration of local, short-term experiential adaptation with global, long-term memory optimization via two main modules:

1. Pairwise-Experience-Evolutionary Meta-Prompting (PE-EMP): As opposed to standard static prompts per tree node, PE-EMP employs a reflexive, seven-stage optimizer at each expansion. By conducting fine-grained pairwise evaluation between candidate and baseline node expansions, it dynamically synthesizes domain- and context-specific criteria, generates scalar scores, and evolves a meta-prompt for future expansion. This ensures that the search does not merely exploit immediate reward, but actively adapts generative policies in response to localized feedback.
2. Memory Optimization Agent: Beyond RAG or static libraries, the global memory repository acts as a living policy prior. At each search iteration, distilled insights are inserted, merged, modified, or pruned by an optimizer, continuously shaping the action distribution for future inference. The repository is taxonomically indexed and selectivity-gated, preventing semantic drift and ensuring relevance.

   Figure 1: Concrete Instantiation of Empirical-MCTS Framework.

The architecture is explicitly designed for seamless, memory-backed reasoning policy evolution, translating successful search traces into global knowledge and progressive meta-prompt sophistication as search proceeds.

Methodological Breakdown

Pairwise-Experience-Evolutionary Meta-Prompting (PE-EMP)

PE-EMP extends Self-Principled Critique Tuning (SPCT) to a bidirectional, recursive process. At node expansion, the agent generates fine-grained criteria specific to the local problem, performs a comparative chain-of-thought critique, allocates dynamic weights, and delivers precise scoring. Crucially, the process incorporates historical priors—curated from the global memory—filtered for alignment by automatic task taxonomy. The evolved meta-prompt, encoding newly-discovered principles, is then iteratively embedded into the generative workflow, fostering contextually relevant, self-consistent prompt evolution.

Active Experience Memory Optimization

The memory module applies Training-Free Group Relative Policy Optimization principles, employing atomic add/modify/merge/delete operations to distill, refine, and compress high-fidelity reasoning experiences. This goes beyond mere RAG: by treating search-derived experience as a direct policy prior (analogous to non-parametric EM in RL), the system shifts its output distribution towards more accurate reasoning over search iterations, all while remaining model-agnostic and parameter-free.

Figure 2: Growth of empirical memory repository during search. The number of distilled experiences increases monotonically with rollout count ($0 \rightarrow 8$), enabling progressive policy refinement across problem instances.

Robust Preference-Based Value Estimation

Reward signals, the backbone of search guidance, are fused from explicit, self-principled scores and global search history via the Enhanced Borda Count, addressing non-transitivity and cyclic preference graph artifacts. This hybridization mitigates decision instability and supports more reliable reward propagation in high-complexity search spaces.

Rollout Strategy and Value Backpropagation

The search procedure employs UCB-based node selection and decay-based soft backpropagation for $Q$-value updates, crucial for efficiently leveraging the sporadic, high-value discoveries mined by the memory optimization agent.

Experimental Evaluation

Mathematical Reasoning Benchmarks

Empirical-MCTS is benchmarked on AIME25, MathArena Apex, and ARC-AGI-2 using multiple SOTA and open LLMs, with detailed cost, rollout, and accuracy analyses.

Key empirical findings:

• On AIME25, Empirical-MCTS achieves 73.3%, exceeding stateless LLaMA-Berry (63.3%) and memory-augmented FLEX (66.6%) under identical model capacity.
• On MathArena Apex, the method delivers nonzero accuracy (4.17%) where DeepSeek-V3.1-Terminus and Repeated Sampling both fail, indicating the synthesis of nontrivial reasoning paths from accumulated empirical memory.

  Figure 3: Cost-Performance Pareto Frontier Analysis. We plot accuracy against inference cost for various models on MathArena Apex and ARC-AGI-2. The dashed blue line indicates the Pareto frontier, representing the optimal trade-off between cost and performance.

Crucially, Empirical-MCTS with mid-tier models (Gemini 3 Flash) dominates in the cost-accuracy Pareto frontier, delivering frontier-level accuracy at markedly lower inference cost—enabling practical deployment where computational constraints matter.

Ablation and Scalability Studies

Ablations reveal that removing PE-EMP or meta-prompting significantly reduces final accuracy and impedes performance scaling with additional rollouts. Statistically, the full framework reaches 76.7% on AIME25 with 8 rollouts; variants plateau ~66.7%, and repeated sampling stagnates at 56.7%.

Figure 4: Ablation study on AIME25 using gpt-oss-120b.

Figure 5: Performance vs. accumulated experiences on AIME25. The full framework (red) demonstrates a strong positive correlation between experience count and accuracy, while ablated variants show limited improvement (yellow). Baseline performance shown as horizontal reference line (56.7\%).

Memory repository growth is strictly monotonic, and performance correlates strongly with cumulative experience count—a direct validation of the framework’s continual wisdom accumulation hypothesis.

Qualitative Analysis: Prompt Evolution

Meta-prompts show marked evolution over search: from generic guidance to highly specific, domain-corrective constraints—for example, shifting towards domain-specific verification subgoals in geometry problems. This adaptive specification is synchronized with empirical experience accrual and is absent from frameworks lacking the dual-loop architecture.

Theoretical and Practical Implications

Empirical-MCTS operationalizes continuous, non-parametric agent evolution at inference time, eliminating the need for retraining or explicit fine-tuning. The dual-loop structure supports rapid policy adaptation and domain transfer, positioning the framework as a core backbone candidate for generalist agentic reasoning in LMs.

Two major implications arise:

• Theoretical: The framework empirically validates that intertwined search and experience modules produce synergetic gains, and that robust policy evolution can be realized outside of parametric weight updates—echoing trends in rapid RL and evolutionary programming.
• Practical: Empirical-MCTS enables cost-effective, scalable deployment of reasoning-intensive LLMs, widening accessibility where compute or budget constraints exist. The memory-centric paradigm further opens new research into safeguards against toxic experience integration and the detection of hallucinated memories.

Future Directions

Key open challenges include reliably filtering low-quality or hallucinated experience—particularly under tasks exceeding model verification boundaries—and extending the memory-accumulation principles to longer-horizon planning and persistent dialog settings. Advanced consistency checks, uncertainty quantification, and adversarial memory pruning may become necessary to ensure robust long-term performance in open domains.

Conclusion

Empirical-MCTS advances the capability of LLMs to solve complex reasoning tasks by unifying Monte Carlo Tree Search with continual, dual-experience-driven agent evolution. The empirical results demonstrate state-of-the-art efficiency and accuracy, especially under inference-time compute constraints. By endowing inference-time agents with structured memory accrual and meta-prompt evolution, Empirical-MCTS points towards a new class of “memory-competent” nonparametric reasoning systems that adapt, refine, and persist beyond isolated problem boundaries (2602.04248).