Discovering Reinforcement Learning Interfaces with Large Language Models

Abstract: Reinforcement learning systems rely on environment interfaces that specify observations and reward functions, yet constructing these interfaces for new tasks often requires substantial manual effort. While recent work has automated reward design using LLMs, these approaches assume fixed observations and do not address the broader challenge of synthesizing complete task interfaces. We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated. We propose LIMEN (Code available at https://github.com/Lossfunk/LIMEN), a LLM guided evolutionary framework that produces candidate interfaces as executable programs and iteratively refines them using policy training feedback. Across novel discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, joint evolution of observations and rewards discovers effective interfaces given only a trajectory-level success metric, while optimizing either component alone fails on at least one domain. These results demonstrate that automatic construction of RL interfaces from raw state can substantially reduce manual engineering and that observation and reward components often benefit from co-design, as single-component optimization fails catastrophically on at least one domain in our evaluation suite.

• The paper presents LIMEN, which automates the design of RL interfaces by jointly evolving observation and reward functions through LLM-guided evolutionary search.
• It employs a bilevel search with a MAP-Elites archive to evaluate candidate interfaces, achieving near-optimal success on gridworld compositional tasks and robotics challenges.
• Empirical results show that joint optimization of observation and reward drastically outperforms single-component approaches, reducing manual design effort and improving agent performance.

Automated RL Interface Discovery via LLM-Guided Evolution

Motivation and Formulation

Reinforcement learning (RL) system development is bottlenecked by the manual design of environment interfaces—specifically, the selection of observation mappings and reward functions. The majority of recent automation efforts in RL have focused primarily on reward synthesis, often using LLMs [llr1, llr2, llr3]. However, these approaches are fundamentally limited by their assumption of fixed observation spaces, neglecting the critical impact of observation structure on agent learning dynamics, task decomposition, and policy efficiency.

The paper introduces the problem of RL interface discovery as the joint search for observation function $\phi$ and reward function $R$ that together induce an effective MDP from raw simulator state. Interface optimization is formulated as a bilevel search: the outer evolutionary loop proposes candidate interfaces, and the inner loop evaluates each candidate by training the RL agent and assessing performance via a trajectory-level task success metric. Notably, the search space comprises executable programs for both $\phi$ and $R$, resulting in interpretable, transferable interface artifacts.

The LIMEN Framework

LIMEN (Learning Interfaces via MDP-guided Evolution) operationalizes RL interface discovery by leveraging LLMs as mutation operators within a quality-diversity (MAP-Elites) evolutionary search. Figure 1 details the system’s structure.

Figure 1: Overview of LIMEN: LLM-guided evolutionary search generates and evaluates candidate interface programs for observation and reward, optimizing for trajectory-level task success.

At each iteration, LIMEN selects a parent interface from the archive, mutates the code using an LLM prompt (incorporating feedback and structural guidance), and evaluates the candidate by training a PPO agent. Interface fitness is determined by episodic task completion rather than instantaneous reward, ensuring alignment with true task objectives.

MAP-Elites archive maintains behavioral diversity via axes of observation dimensionality and reward AST complexity, preventing premature convergence to a single interface strategy and promoting structurally novel solutions.

Experimental Evaluation and Domains

LIMEN is empirically validated across five tasks spanning discrete and continuous domains: three gridworld compositional reasoning tasks and two MuJoCo robotic control tasks (See Figure 2).

Figure 2: Evaluation environments: (a-c) XLand-MiniGrid compositional gridworld tasks, (d) quadruped push recovery, (e) manipulator trajectory tracking.

These tasks are carefully engineered to expose distinct failure modes of interface construction: gridworld tasks are observation-limited, while robotics tasks are reward-limited. Default baselines with raw simulator state and sparse reward collapse on all but the simplest problems, underscoring the necessity of automated interface synthesis.

Quantitative Results and Ablation Analysis

Joint evolution of observation and reward consistently outperforms both observation-only and reward-only search. LIMEN achieves near-optimal performance on XLand-MiniGrid (Easy/Medium: 99%, Hard: 85% success), and superior results on MuJoCo robotics (Panda: 45%, Go1: 48%), substantially exceeding sparse baselines and ablations (Figure 3).

Figure 3: Learning curves for LIMEN and ablations: joint interface discovery achieves consistently higher task success than single-component optimization.

Reward-only ablations collapse on compositional gridworld tasks (e.g., Medium: 19%, Hard: 1%), while observation-only search fails entirely on Panda tracking (0%). Each single-component optimization fails catastrophically in at least one domain, validating the necessity of joint search. Independent LLM sampling (no evolution) produces inferior interfaces compared to evolved solutions, with best-of-30 samples failing to match LIMEN’s performance in all evaluation tasks (Figure 4).

Figure 4: Success rates from independent LLM samples (no evolutionary refinement) versus best evolved interfaces; iterative refinement is essential for effective interface discovery.

Repeated evolution runs reveal modest variance in XLand tasks and higher variance in more complex environments (Figure 5), suggesting the benefit of longer search horizons and more advanced selection strategies.

Figure 5: Evolution progress across five seeds in XLand-MiniGrid: diverse candidate interfaces and gradual improvements in task success.

Structural Interface Patterns and Discovery Dynamics

Program-level analysis of evolved interfaces reveals recurring structural motifs: observation programs implement geometric and relational features, explicit phase indicators, and multi-scale encoding. Reward functions employ potential-based shaping, milestone bonuses, and phase-dependent guidance. Such patterns reflect established RL engineering best practices, now rediscovered through iterative LLM-driven code synthesis.

Evolution modifies reward structure for efficiency—for example, in Go1 push recovery, gating position rewards by uprightness is removed to provide continuous gradients, substantially improving agent success rates.

Robustness, Generalization, and Practical Implications

LIMEN interfaces exhibit robustness under moderate distributional shift in robot tasks (see Figure "robustness_curves.pdf" in supplementary material), maintaining non-trivial performance even with perturbed dynamics. Importantly, interface transfer (rather than policy transfer) yields stable learning under domain changes, suggesting that evolved interfaces capture task-relevant structure beyond nominal parameters.

The practical implications are substantial: LIMEN reduces the manual engineering burden, automates interface design from raw simulator state, and constructs interpretable programs for both agent input and reward.

Limitations and Future Directions

Limitations include the reliance on structured simulator state, computational cost of RL training in the evaluation loop, and dependence on access to accurate trajectory-level task metrics. Extending interface discovery to vision-based environments and real-world domains with restricted state access or partially observable dynamics represents a future research direction. Enhancing evolutionary strategies, increasing search horizon, and using frontier LLMs for higher-quality code mutation may further improve efficiency and generalization.

Conclusion

LIMEN establishes a rigorous benchmark for RL interface discovery, demonstrating—via diverse compositional reasoning and robotics tasks—that joint programmatic search over observation and reward functions is necessary for consistent task success. The framework’s results validate that co-design of observation and reward is critical, and that automated LLM-guided evolution substantially mitigates the bottleneck of manual interface engineering in RL. LIMEN paves the way for future directions in scalable RL interface discovery, especially in settings with complex, unstructured state spaces.