EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning

Abstract: In actor-critic reinforcement learning, network architectures are typically manually designed. Automating this design is challenging because each candidate must be trained before evaluation, and the design space is open-ended. To address these challenges, we introduce EVOM, an agentic meta-evolution framework for discovering high-performance actor-critic architectures. We frame architecture search as a bi-level optimization: an inner loop trains weights via the low-fidelity proximal policy optimization (PPO), while an outer loop drives meta-evolution by iteratively refining architecture programs. Crucially, this outer loop is powered by an LLM-based design agent that operates purely as an architecture designer, completely decoupled from policy execution and environment control. Experiments reveal that EVOM outperforms the manually designed baseline, an LLM-guided random search, and the state-of-the-art LLM-guided programmatic policy search method MLES, delivering superior performance on Ant-v4 and HalfCheetah-v4. Ablation studies validate that both the meta-evolution loop and the LLM Design Agent are indispensable for final performance.

• The paper introduces a novel bi-level meta-evolution framework that leverages LLMs to programmatically design and refine actor-critic architectures.
• It employs an evolutionary outer loop with mutation and crossover operators alongside low-budget PPO for inner-loop weight adaptation.
• Experimental results on continuous-control tasks reveal significant performance gains over hand-designed and random architectures, validating the method's efficacy.

Agentic Meta-Evolution for Actor-Critic Architectures: The EVOM Framework

Introduction and Motivation

Automating network architecture discovery in reinforcement learning (RL), particularly for actor-critic methods, remains a critical open problem due to the expensive nature of candidate evaluation and an unbounded architecture design space. Existing approaches often fix network topologies, disregarding their influence on optimization dynamics and final policy performance. "EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning" (2606.26327) addresses this challenge by introducing a hierarchical optimization paradigm, leveraging LLMs as programmatic design agents operating within an evolutionary search framework. The paradigm decouples architecture innovation from parameter learning: the LLM agent proposes and edits executable actor-critic architectures, while PPO is employed as an inner-loop optimizer for weight adaptation.

Methodology

Bi-Level Architecture Search via Programmatic Evolution

EVOM casts architecture search as a bi-level optimization problem. The outer loop performs meta-evolution by maintaining and iteratively refining a population of architecture programs. The inner loop evaluates each candidate by training its actor-critic weights using low-budget PPO. This division enables systematic empirical fitness estimation while allowing open-ended design in the architectural space.

Figure 1: EVOM operationalizes a bi-level optimization, using an evolutionary LLM-based agent for architectural proposals at the top level, wrapped around a PPO-driven parameter training and evaluation inner loop.

LLM-Guided Design Operators and Evolution

The key differentiator for EVOM lies in its LLM-powered outer loop. The LLM is instantiated as a stateful architecture designer, operating on descriptive and programmatic representations:

• Initialization: Given environment/task context, the LLM synthesizes a diverse initial population of actor-critic programs, semantically structured as modular Python code.
• Mutation: The LLM receives an elite architecture, and edits its components—layers, normalization, activations, residual structures—subject to task- and interface-specific constraints.
• Crossover: The LLM is given two parents and instructed to fuse their patterns, producing hybrid offspring architectures.

All design actions are performed via templated natural-language prompts contextualized with the current elite population.

Figure 2: Prompt templates orchestrate LLM behavior across initialization, mutation, and crossover; the agent receives contextualized design data, ensuring iterative and reusable architecture innovation.

Invalid or divergent architectures are identified and eliminated by a penalty-based evaluation protocol, ensuring the empirical inner loop only propagates plausible designs.

Evaluation Loop and Program Representation

Architectures are programmatically defined, specifying all relevant topological and functional properties (depth, width, normalization, activation, residual links, etc.) within a strict actor-critic interface. For each generation, candidates are trained for a low-fidelity budget of 100k PPO steps and evaluated over multiple deterministic episodes. Top performers are selected as the next elite population; after all generations, the final elite is retrained for 5 million steps to assess true performance.

Experimental Analysis

Benchmarks

Evaluation is conducted on classic continuous-control tasks, Ant-v4 (high-dimensional legs) and HalfCheetah-v4 (planar locomotion), using standardized state and action dimensionalities with normalization.

Performance Results

On Ant-v4, EVOM achieves a mean return of $4652.10 \pm 746.91$, a substantial improvement over the manually designed MLP baseline ($2517.06 \pm 366.24$), random LLM search ($988.15 \pm 2.81$), and the programmatic policy search baseline MLES ($1185.26 \pm 84.15$). Similarly, on HalfCheetah-v4, EVOM obtains $5381.07 \pm 1357.70$, outperforming all baselines, with random search delivering higher variance but lower reliability than the evolutionary process.

Evolutionary progression is reflected in consistent improvements to both historical-best and elite-population mean rewards. The selected architectures frequently employ deeper, LayerNorm-rich, or residual patterns, highlighting the LLM's capacity to construct advanced design motifs beyond initial heuristic guesses.

Figure 3: On Ant-v4, both historical-best and population-mean rewards increase over generations; final architectures incorporate residual and normalization elements not found in conventional hand-designed or naively sampled models.

Figure 4: On HalfCheetah-v4, EVOM evolves a deeper LayerNorm-SiLU MLP outperforming manual designs; PPO learning curves verify stable and high-rewarding policies traced to evolved architectural merits.

Ablations and Operator Decomposition

Operator-level ablation reveals that mixed mutation and crossover significantly surpass single operator variants in final mean reward (e.g., $5381.07$ vs $3455.77$ for mutation-only on HalfCheetah-v4). Increasing the population size fails to monotonically improve performance, being sensitive to the fidelity of proxy evaluation and genetic drift. Changing the LLM backbone produces marked shifts in the returned architectural distributions, suggesting robustness and room for further improvement by scaling LLM capabilities or tailoring prompts.

Theoretical and Practical Implications

EVOM formalizes meta-evolution as architectural program optimization mediated by agentic LLMs, pushing the envelope of program-level NAS in RL. Practically, it demonstrates that LLMs can serve as reusable, context-sensitive, and empirically grounded design engines, systematically outperforming both hand-crafted and independent-sample (i.i.d.) LLM designs under compute constraints.

Theoretically, EVOM answers whether open-ended, human-in-the-loop architectural design can be replicated and even superseded by automated, agentic design operators, provided with the right evaluation and feedback interface. By separating the search over architectural priors from policy/value parameter adaptation, the method enables explainable and reproducible architectures, amendable to further inspection and transfer.

Limitations and Future Directions

The main limitations are (1) proxy evaluation bias—low-budget PPO may favor overfit or fast-starter architectures, (2) waste due to invalid LLM generations, and (3) limited environment scope (Ant-v4 and HalfCheetah-v4). Addressing these gaps calls for adaptive evaluation budgeting, integrating program repair, broader task classes, and investigating the transferability of evolved modules.

Future research could explore tighter LLM-environment feedback loops, multi-agent architecture co-evolution, task-adaptive prompt engineering, and the intersection with automated hyperparameter search.

Conclusion

EVOM establishes a new foundation for agentic, program-level architecture search in the RL domain by leveraging LLMs as design operators within a meta-evolutionary process. Experimental evidence on continuous-control tasks verifies that this approach yields high-performing, semantically rich architectures not reproducible by manual design or simplistic LLM sampling, affirming the natural compatibility between programmatic LLM reasoning and open-ended neural architecture search. The framework offers both a methodological baseline and extensible code base for future advances in automated RL system synthesis.