LIMEN: Joint RL Interface Synthesis

• LIMEN is a reinforcement learning framework that jointly synthesizes observation mappings and reward functions from raw simulator states using only a trajectory success metric.
• It employs an evolutionary algorithm with LLM-guided mutations and MAP-Elites archive selection to optimize interfaces in both gridworld and continuous control tasks.
• Experimental results demonstrate that joint evolution significantly outperforms single-component methods, achieving high success rates in diverse RL domains.

Searching arXiv for the LIMEN paper and closely related RL interface/reward-design work. LIMEN, expanded in the source as Learning Interfaces via MDP-guided EvolutioN, is a framework for RL task interface discovery from raw simulator state, in which both the observation mapping and the reward function are synthesized automatically rather than hand-specified. The method treats an interface as a pair of executable programs, jointly evolves these programs with LLM guidance, and evaluates them through policy training feedback, using only a trajectory-level success metric such as whether the agent reached the goal. In the reported experiments, LIMEN is applied to both discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, where joint evolution of observations and rewards is reported to discover effective interfaces in settings where optimizing either component alone fails on at least one domain (Jaswal et al., 5 May 2026).

1. Problem setting and conceptual scope

Reinforcement-learning systems interact with environments through an interface that specifies what the agent observes and how it is rewarded. In the formulation used here, the interface has two components: an observation mapping $\phi : S \to O$, which extracts features from the simulator’s raw state $s \in S$, and a reward function $R : S \times A \times S \to \mathbb{R}$, which assigns per-step rewards based on $(s, a, s')$ (Jaswal et al., 5 May 2026).

The motivation for LIMEN is the claim that most prior work either assumes $\phi$ is fixed and hand-tuned or focuses solely on $R$, including LLM-based reward synthesis. The paper argues that a fixed $\phi$ can omit or poorly structure information critical for learning, while reward-only methods fail when the observations are insufficient. Manually designing both $\phi$ and $R$ is described as laborious, domain-specific, and error-prone. LIMEN addresses this by jointly discovering both components from raw simulator state, conditioned only on a success metric $F$ defined over trajectories.

This framing makes LIMEN a method for task interface synthesis, not merely reward shaping or policy optimization. A plausible implication is that the framework shifts some of the RL design burden from environment engineering to program search over observation-reward pairs.

2. Core algorithmic architecture

At a high level, LIMEN interleaves an outer evolutionary loop over candidate interfaces $s \in S$0 with an inner RL training loop that evaluates each candidate by training a policy $s \in S$1 and measuring its task success $s \in S$2 (Jaswal et al., 5 May 2026).

The algorithm begins by initializing an empty MAP-Elites archive indexed by two descriptors: observation dimension $s \in S$3 and reward complexity, defined by AST node count. An initial interface is sampled from the LLM mutation model, evaluated by training a policy in the induced MDP, scored by expected success over seeds, and inserted into the archive cell corresponding to its descriptors.

Subsequent iterations select a parent interface from the archive with a mixed strategy: 70% global, fitness-proportional and 30% local island, uniform. A mutation prompt is then built from the task description, the parent code and its performance, examples of top interfaces so far, and examples and error traces of recent failed programs. The LLM returns two Python functions, get_observation and compute_reward. If the resulting interface passes validation, including syntax, shape checks, and the requirement that reward is not constant, the system trains a policy on the induced MDP, computes fitness as expected success over seeds, and inserts the result into the archive.

The archive is used not only as a storage structure but as a quality-diversity mechanism. The paper states that it prevents collapse to a single $s \in S$4 pair, and that an island model with $s \in S$5 partitions is used for parallelism and diversity.

3. Mathematical formulation of interfaces

LIMEN starts from a simulator MDP

$s \in S$6

where $s \in S$7 is the simulator state space. An interface is defined by two executable programs:

$s \in S$8

and

$s \in S$9

Applying $R : S \times A \times S \to \mathbb{R}$0 to $R : S \times A \times S \to \mathbb{R}$1 induces a learning MDP

$R : S \times A \times S \to \mathbb{R}$2

where $R : S \times A \times S \to \mathbb{R}$3 is the agent’s observation space and $R : S \times A \times S \to \mathbb{R}$4 is the push-forward of transitions through $R : S \times A \times S \to \mathbb{R}$5 (Jaswal et al., 5 May 2026).

The paper formulates the objective as a bilevel optimization problem:

$R : S \times A \times S \to \mathbb{R}$6

subject to

$R : S \times A \times S \to \mathbb{R}$7

Here, $R : S \times A \times S \to \mathbb{R}$8 is the trajectory-level success metric. This makes the search target the downstream trainability of the interface rather than any static proxy for interface quality.

The interface representation is deliberately programmatic. Both $R : S \times A \times S \to \mathbb{R}$9 and $(s, a, s')$0 are written as short JAX-compatible Python functions using numerical operations, concatenations, norms, and differentiable conditionals. The observation mapping returns a fixed-length jnp.ndarray with max length 512, and the reward function returns a scalar jnp.float32. This executable-program representation is central to the framework: it allows the LLM to propose structured mutations and the evaluator to reject degenerate or malformed candidates automatically.

4. LLM-guided evolution and policy-training feedback

The mutation mechanism uses Claude Sonnet 4.6 with temperature 0.7. Each prompt contains the task description, parent code and fitness, examples of top interfaces, and examples plus error traces of recent failed programs. The paper attributes diversity partly to stochastic decoding, with additional post-processing filters to remove syntax errors, shape mismatches, and degenerate cases such as constant reward or invalid shapes (Jaswal et al., 5 May 2026).

The MAP-Elites archive is a 2D grid whose cells are binned by observation dimension ranges and reward AST node count ranges. The observation dimension bins are described as 1–50, 51–100, … up to 512. Each cell stores the interface with the highest fitness within that niche. The island model partitions the archive into $(s, a, s')$1 islands that occasionally exchange their best solutions.

Evaluation uses a cascade filter. First, a short-budget run is used to discard weak candidates early. The paper gives 500K–3M steps in gridworld as the example range and states that candidates below a minimal success of 1–5% are discarded. Surviving candidates then receive a full multi-seed evaluation (3 seeds), with per-seed success $(s, a, s')$2 and overall fitness

$(s, a, s')$3

This scalar fitness is fed back into archive insertion and drives selection pressure for the next generation.

The inner-loop RL algorithm is Proximal Policy Optimization (PPO) with fixed hyperparameters. A plausible implication is that LIMEN’s claims are intended to isolate interface quality rather than gains from bespoke RL tuning.

5. Experimental domains, baselines, and empirical findings

The experiments cover two classes of tasks. In XLand-MiniGrid, the paper evaluates three gridworld settings: Easy, in which the agent picks up a specified object among distractors on a 9×9 grid with 80 steps; Medium, in which one object must be placed adjacent to another, again on a 9×9 grid with 80 steps; and Hard, involving four rooms, a multi-step rule chain, a 13×13 grid, and 400 steps. The default observation is a flattened 7×7 egocentric grid, and the default reward is +1 only on success. In continuous control, the domains are quadruped push-recovery (Go1), where the agent must remain upright and near the origin for 500 steps, and Panda manipulator tracking, where a 7-DoF arm tracks a Lissajous trajectory for 500 steps (Jaswal et al., 5 May 2026).

The baselines and ablations are: Sparse, using raw $(s, a, s')$4 and success/binary reward; Obs-Only, evolving $(s, a, s')$5 while keeping $(s, a, s')$6 fixed to sparse; and Reward-Only, evolving $(s, a, s')$7 while keeping $(s, a, s')$8 fixed to raw state. The paper reports that joint $(s, a, s')$9 evolution consistently outperformed all ablations, with success rates of 99% on Easy, 99% on Medium, 85% on Hard, 45% on Panda, and 48% on Go1. The Sparse baseline failed all but the easiest task. Reward-only collapsed on Medium (19%) and Hard (1%). Obs-only failed entirely on Panda (0%).

A further ablation sampled 30 interfaces independently from the LLM without evolution. This independent LLM-sampling ablation achieved <3% on Medium/Hard and <25% on robotics, which the paper states is far below LIMEN. The reported interpretation is that search pressure and archive structure are not incidental implementation details but integral to performance.

The paper also identifies recurring structural motifs in discovered interfaces. In observations, these include relative geometric features, normalized distances, directional one-hots, multi-scale encodings, and phase indicators. In rewards, the recurring motifs are potential-based shaping, milestone bonuses gating by phase, and smoothness and torque penalties in continuous domains. In a Go1 case study, an early reward gated position shaping behind uprightness, while a later reward removed that gating to allow continuous small corrections, boosting success from 32% to 55%.

6. Interpretation, failure modes, and terminological ambiguity

The paper’s central conclusion is that LIMEN is the first method to jointly synthesize $\phi$0 and $\phi$1 from raw state, guided only by a trajectory success metric (Jaswal et al., 5 May 2026). Within the reported evaluation suite, this conclusion is tied directly to the observation that single-component optimization catastrophically failed on at least one domain each. The failure analysis is specific: observation-only on Panda failed because the raw sparse reward signal had no gradient, while reward-only on Medium/Hard failed because the default $\phi$2 lacked relational structure, leaving shaped rewards with “nowhere to operate.”

A seed-variance study over 5 independent runs reported consistent convergence to >90% on Easy and Medium, but high variance on Hard, where some runs stalled <10% and some reached ~75%. This suggests that the framework is robust on simpler domains but still search-sensitive in more compositional tasks.

The limitations are explicit. LIMEN relies on a clean success signal $\phi$3, which is stated to be harder to obtain in continuous or real-world tasks without an oracle. Its computational cost is high, measured in tens of GPU-hours per run. It assumes access to structured state, described as privileged simulator info. The paper states that scaling to pixel/vision inputs or real hardware remains open. It also proposes several future directions: more advanced evolutionary strategies, including batch proposals and surrogate models; stronger LLMs for improved efficiency; and penalizing $\phi$4’s dimensionality in fitness to discourage overly large observation vectors.

The name LIMEN should also be distinguished from LIM-Net, the “Lightweight Interactive Network for 3D Medical Image Segmentation with Multi-Round Result Fusion”, which is a compact CNN-based system for interactive 3D medical image segmentation rather than RL interface discovery (Shen et al., 2024). The similarity in names can create bibliographic confusion, but the two methods address different problem classes, use different architectures, and operate in unrelated application domains.