Self-Improving Language Models with Bidirectional Evolutionary Search

Abstract: Search has been proposed as an effective method for self-improving LLMs and agentic systems, both for post-training sample generation and for inference. However, widely used methods such as best-of-N sampling and tree search face two fundamental limitations: they are guided by sparse verification signals, and they construct candidates primarily through autoregressive expansion, restricting exploration to regions with substantial model probability mass. To address these, we propose Bidirectional Evolutionary Search (BES), a search framework that couples forward candidate evolution with backward goal decomposition. In the forward search, BES augments standard expansion with evolution operators that recombine partial trajectories to generate candidates that are difficult to obtain from a single model rollout. In the backward search, BES recursively decomposes the original task into checkable subgoals, producing dense intermediate feedback that guides forward search. We provide theoretical motivation showing that candidates generated by expansion-only search are confined to a narrow entropy shell while evolutionary operators can escape it, and that backward search can exponentially reduce the number of required samples to find a correct answer. Experiments show that on challenging post-training tasks where mainstream post-training algorithms fail to improve, BES enables consistent gains, and on three open problem solving benchmarks at inference time, BES outperforms existing open-source frameworks in both average and best-case performance. Code and trained models are available at https://github.com/Embodied-Minds-Lab/BES.

• The paper introduces BES, a framework that combines forward evolutionary operators with backward goal decomposition to overcome limited exploration in language models.
• BES employs recombination techniques such as combination, deletion, translocation, and crossover to explore low-probability solution spaces beyond traditional autoregressive methods.
• Empirical results demonstrate that BES improves multi-hop reasoning and optimization tasks by reducing sample complexity and increasing interpretability.

Self-Improving LLMs with Bidirectional Evolutionary Search

Motivation and Context

Recent advances in LLMs and agentic systems have highlighted two fundamental limitations of dominant search strategies for post-training and inference: the sparsity of verification signals and confinement to the model’s own probability distribution. Conventional approaches, such as best-of-$N$ sampling and tree search, generate candidates exclusively through autoregressive expansion, resulting in exploration restricted to regions of high policy probability mass and a reliance on sparse, coarse-grained verification. This restricts the capacity of LLMs and agents to discover correct solutions in low-probability regions and hinders sample-efficient self-improvement on frontier tasks.

Bidirectional Evolutionary Search (BES) is introduced as a principled solution. BES couples forward candidate evolution—including both expansion and four recombinative evolution operators—with backward goal decomposition, providing dense and interpretable intermediate feedback. Theoretical results establish that expansion-only search is restricted to a narrow entropy shell (a bounded typical set in log-probability space), while evolutionary recombination escapes this shell; the backward search decomposes global goals into explicit sub-goals, exponentially reducing the sample complexity required to reach correct answers on hard tasks.

Figure 1: Comparison of tree search (left) and BES (right); BES escapes the entropy shell via recombination and leverages backward decomposition for dense feedback.

BES Framework: Algorithmic and Theoretical Foundations

Forward Search: Expansion and Evolutionary Operators

BES maintains a candidate pool of partial trajectories, each representing stepwise reasoning or agentic actions. Beyond standard expansion, BES employs four evolution operators inspired by biological recombination:

1. Combination: Concatenates distinct suffixes of two parent trajectories sharing a common prefix;
2. Deletion: Removes an interior reasoning segment to condense a trajectory;
3. Translocation: Replaces a step in one trajectory with a step from another;
4. Crossover: Splices the prefix of one trajectory onto the tail of another.

These operators permit exploration of solution space beyond the native distribution of $\pi_\theta$, enabling the construction of candidates unattainable by sequential expansion alone.

Figure 2: The five forward search operators. Combination, translocation, crossover, and deletion recombine or restructure trajectories to augment the solution space.

Backward Search: Goal Decomposition and Dense Verification

BES integrates backward search by prompting the model to recursively decompose the global goal into fine-grained, verifiable sub-goals, constructing a hierarchical goal tree. Each sub-goal is paired with a local verifier, which can be instantiated as a rule-based checker, embedding similarity model, or LLM judger, providing continuous intermediate feedback. This facilitates granular credit assignment and informed parent selection.

The backward scoring function melds parent and children contributions, recursively blending verification at each goal level. For two-parent operators, the pair score is computed as the maximal coverage across sub-goals, favoring recombination among candidates with complementary strengths.

Figure 3: BES case study on a multi-hop reasoning problem showing expansion, recombination via translocation, and backward goal tree decomposition.

Theoretical Guarantees

The core theoretical claims are:

1. Entropy Shell Confinement (Expansion-only Search):

With mild assumptions (bounded per-step surprise, decaying step dependence, linear block total correlation), candidates generated by sequential expansion are confined to the typical set of size $\exp(H_T + \epsilon T)$, where $H_T$ is trajectory entropy and $T$ is horizon length.

2. Shell Escape (Evolution Operators):

Evolutionary recombination increases trajectory-level surprise by breaking inter-block dependence: the expected log-probability (surprise) of evolution-produced candidates is $H_T + \gamma T$, with $\gamma > \epsilon$ as the block total correlation. A non-negligible fraction of evolution candidates escapes the shell, spanning low-probability regions where correct solutions may reside.

3. Sample Efficiency via Backward Search:

Backward decomposition exponentially reduces required samples to hit all sub-goals (from $\Omega(1/\prod_i p_i)$ to $O(p_{\min}^{-1}\log(m/\delta))$), transforming multiplicative events into efficiently tractable collection problems for $m$ sub-goals.

Empirical Results

Post-Training: Logical and Multi-Hop Reasoning

On the Knights-and-Knaves logical reasoning benchmark (Gemma-3-1B-it), BES demonstrates consistent improvement in validation accuracy, outperforming GRPO and MaxRL which stagnate on hard samples. The case study confirms that evolution operators, especially translocation, enable recombination of partial reasoning chains, constructing correct solutions from diverse branches.

For multi-hop reasoning (MuSiQue, Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct), BES yields substantial gains in accuracy (up to +3.8% over baselines), with higher numbers of valid search actions and finish ratios, indicating agents learn active search strategies rather than degenerate guessing.

Inference: Open Problem Solving

On open scientific optimization tasks (Circle Packing, Heilbronn Convex), BES, integrated with ShinkaEvolve and GPT-5 backbone, consistently outperforms open-source frameworks (OpenEvolve, GEPA, ShinkaEvolve) on both mean and best objective values, with reduced variance and modest API cost overheads. BES’s evolution operators are realized via LLM-driven program rewrites when direct concatenation is infeasible.

Ablation and Cost Analysis

Ablation studies confirm the necessity of both evolution operators and backward decomposition for maximal sample efficiency and effective training. Cost analysis shows that BES adds less than 30% search overhead compared to Tree-GRPO, with significant accuracy benefits.

Implications and Future Directions

Practical Implications:

BES demonstrates robust self-improvement for LLMs and agents in both post-training and inference regimes, enabling efficient sample generation beyond the native policy distribution and facilitating interpretability via explicit goal decomposition. The dense feedback mechanism addresses sparse reward issues and discourages reward hacking.

Theoretical Insights:

The entropy shell theory formalizes the limitations of sequential search and motivates the necessity of recombinative operators for distributional exploration. Backward decomposition transforms high-dimensional multiplicative hitting into tractable sub-goal collection, scaling sample efficiency exponentially with problem complexity.

Future Directions:

Potential avenues include adaptation of BES to subjective evaluation domains, leveraging explicit backward decomposition for more explainable agentic systems, and investigation of advanced evolutionary operators for program synthesis and scientific discovery. Scaling BES for large models and longer horizons, and integrating meta-search strategies, are promising directions.

Conclusion

BES systematically addresses fundamental constraints of search in self-improving LLMs and agents, coupling evolutionary recombination with structured backward goal decomposition. Theoretical and empirical results establish BES as a principled and effective framework for both post-training and inference, facilitating discovery of high-quality solutions on frontier tasks through dense feedback and distributional exploration. The framework is poised to inform future developments in search-based self-improvement and reasoning for AI systems.