EvoToolkit: LLM-Driven Evolutionary Policy Synthesis

• EvoToolkit is a modular evolutionary computation framework that combines LLM-driven code mutation with classical search strategies to synthesize verifiable control policies.
• It employs a five-stage process including population initialization, LLM-based variation (via FunSearch, EoH, and EvoEngineer), fitness evaluation, selection, and reproduction.
• The framework achieves superior interpretability and sample efficiency by producing compact, human-readable policies that are formally verifiable in simulated environments like LunarLander-v3.

EvoToolkit is a modular evolutionary computation framework designed for the synthesis and optimization of interpretable control policies using LLMs as generative operators for program evolution. Targeted at autonomous system policy design, EvoToolkit integrates LLM-driven code mutation and crossover with classic evolutionary search strategies and customizable fitness evaluation. By combining LLMs’ ability to propose semantically meaningful code variants with systematic selection and evaluation, EvoToolkit enables the automated discovery of compact, human-readable policies that can be formally inspected and verified, addressing the limitations of both black-box reinforcement learning and manual policy engineering (Guo et al., 11 Jan 2026).

1. Modular Architecture and Workflow

EvoToolkit is composed of five interlocking modules, each mapping to a canonical stage in population-based evolutionary algorithms:

1. Population Initialization: The framework uses an LLM prompt to generate an initial population $P = \{\pi_1, ..., \pi_n\}$ of candidate Python functions, each of which maps an 8-dimensional state vector to one of four discrete actions. This supports direct seeding of the search space with valid policy stubs.
2. LLM-Based Variation Operators: Mutation and crossover are mediated via prompt-driven LLM calls, supporting multiple strategies:
   • FunSearch: Sequence continuation for policy exploration.
   • EoH (Evolution of Heuristics): Specialized prompts for distinct operator classes (exploration, guided crossover, structural mutation, parameter mutation, re-injection).
   • EvoEngineer: Uses structured prompts that incorporate episode statistics, detected failure modes, and domain-specific hints. The design allows the specification of operator probabilities, e.g., $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$.
3. Fitness Evaluation: Each policy candidate $\pi$ is evaluated in a task-specific simulation environment (e.g., Gymnasium LunarLander-v3). Code is executed within a safety-restricted Python sandbox to mitigate risks from untrusted code.
4. Selection: Supports tournament and fitness-proportional (roulette) selection. For roulette, selection probability is parameterized by

$$P(\pi_i) = \frac{f(\pi_i)^\alpha}{\sum_j f(\pi_j)^\alpha},\quad \alpha \ge 1.$$

1. Reproduction: Next-generation populations are formed via elitist or generational replacement, retaining top-performing policies and filling remaining slots with offspring.

The workflow is cyclic: initialization → evaluation → selection → LLM-based variation → replacement → repeat (Guo et al., 11 Jan 2026).

2. Formal Algorithmic Structure

EvoToolkit’s evolutionary process can be formalized as follows:

Algorithm 1: LLM-Driven Policy Evolution

• Input: Environment $\mathcal{E}$, population size $n$, generations $G$, episodes per evaluation $K$.
• Steps:
  • Evaluate each $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$1: $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$2, where

  $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$3 - Select $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$4 using the selection algorithm. - Generate $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$5 by LLM-driven variation. - Form new population $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$6 via replacement strategy. 3. Return the best policy $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$7.

Mutation and crossover rates, tournament size, and other evolutionary parameters are configurable ((Guo et al., 11 Jan 2026), Section 2).

3. LLM-Driven Code Variation

EvoToolkit employs a prompt engineering methodology for code variation:

• Mutation Prompts: Elicit concrete, localized changes in policy logic. Example:

  "Given the following Python policy code that obtained average reward $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$8: ... Identify one logical threshold or control-flow block that may improve performance when modified. Show only the mutated function in full, explain in two sentences why you changed it, and ensure code length ≤ 60 lines. Avoid syntax errors."

• Crossover Prompts: Instruct the LLM to combine semantically meaningful segments (e.g., "Extract the Phase 1 altitude control logic from πₐ and the Phase 3 touchdown logic from π_b... Merge ... into a single coherent Python function...").

• Syntax Enforcement: The prompt explicitly requests code totals ≤60 lines; a post-processing step parses LLM outputs, rejects non-ast.parse()-able code, and re-prompts if necessary (up to two retries).

This approach provides semantic-rich, targeted policy edits that remain compatible with downstream automated evaluation and formal verification (Guo et al., 11 Jan 2026).

4. Policy Representation and Evaluation

Each individual is encoded as a standalone Python function: $$P(\pi_i) = \frac{f(\pi_i)^\alpha}{\sum_j f(\pi_j)^\alpha},\quad \alpha \ge 1.$$3

• Policies are stored as files per generation and seed, dynamically loaded for simulation.

• Execution is within a Gymnasium LunarLander-v3 loop, under restricted built-ins to prevent file I/O.

Fitness metrics include:

• Average cumulative reward over $p_{\mathrm{mut}} = 0.2, p_{\mathrm{xor}} = 0.8$9 episodes.

• Crash penalty: if the agent crashes, $\pi$0 for that episode.

• Fuel penalty (inherent to the environment).

• Success fraction: $\pi$1 (Guo et al., 11 Jan 2026).

5. Implementation Details

• Hyperparameter configuration: population size ($\pi$2), generations ($\pi$3), episodes per evaluation ($\pi$4), LLM (GPT-4o), temperature $\pi$5, mutation and crossover rates ($\pi$6), tournament size ($\pi$7).

• Infrastructure:

  • Fitness evaluation is parallelized using Python’s multiprocessing.Pool.
  • ASTs for all policies are cached to avoid repeated parsing.
  • LLM calls are batched (where model API supports) to reduce latency.
  • LLM integration is abstracted via LLMClient.call_model, supporting automatic rate-limit backoff and retries.
• Codebase organization:
  • Key modules: init.py (population), variation.py (LLM-based operators), selection.py, evaluate.py (sandbox execution & metrics), reproduce.py (replacement), end-to-end scripts.
  • Open-source release at https://github.com/pgg3/EvoControl ((Guo et al., 11 Jan 2026), Section 8).

6. Empirical Results and Evaluation

Benchmark: Gymnasium LunarLander-v3.

Method	Avg Reward	Success %	LLM Calls	LoC
Random	$\pi$8	0 %	–	N/A
PPO (1M steps)	$\pi$9	60 %	–	N/A
FunSearch	$$P(\pi_i) = \frac{f(\pi_i)^\alpha}{\sum_j f(\pi_j)^\alpha},\quad \alpha \ge 1.$$0	22 %	~45	20–35
EoH	$$P(\pi_i) = \frac{f(\pi_i)^\alpha}{\sum_j f(\pi_j)^\alpha},\quad \alpha \ge 1.$$1	16 %	~45	25–40
EvoEngineer	$$P(\pi_i) = \frac{f(\pi_i)^\alpha}{\sum_j f(\pi_j)^\alpha},\quad \alpha \ge 1.$$2	40 %	~45	30–50
EvoEngineer+	143.6	70 %	~200	59

• EvoEngineer exceeds 70% success rate, outperforming FunSearch and EoH in average reward and robustness.
• Evolved policies are interpretable and phase-decomposed (altitude-based logic), with cyclomatic complexity (≈12) orders of magnitude simpler than PPO networks (~10K parameters).
• Removing failure-mode context in prompts degrades performance significantly, which highlights the importance of diagnostic feedback to LLM operators (Guo et al., 11 Jan 2026).

7. Reproducibility and Research Extensions

• Reproduction instructions include environment setup, dependency installation, API key provisioning, and training script execution, with built-in plotting of convergence.
• EvoToolkit's modularity allows adaptation to different environments and reward structures by customizing simulator invocation and LLM prompt templates.
• The framework’s approach demonstrates that LLM-driven evolutionary search can produce trustworthy, verifiable, and sample-efficient policy code, with direct inspectability for safety-critical control tasks—a property not achievable via black-box neural methods (Guo et al., 11 Jan 2026).

---

EvoToolkit represents a hybrid methodology bridging evolutionary programming and LLMs for autonomous system policy synthesis, combining interpretable policy evolution, flexible LLM-based search operators, and rigorous empirical benchmarking. Its open-source release supports further investigation into scalable, interpretable, and safety-critical policy optimization.