LevelEnv: Adaptive Curriculum for RL and LLM Training

• LevelEnv is a formal abstraction for adaptive, difficulty-aligned task generation, integrating procedural environment design with agent learning objectives.
• It employs explicit parameterization and dynamic environment policies to continually adjust task difficulty based on metrics like empirical success rates.
• Applications in TerrainRLSim and GenEnv demonstrate enhanced learning efficiency and significant empirical gains in reinforcement learning and LLM training.

LevelEnv is a formal abstraction for adaptive, difficulty-aligned task generation frameworks, unifying methodologies for procedurally generated environments in reinforcement learning and co-evolutionary LLM training loops. It is characterized by explicit control over the space of environment configurations, systematic measurement and adaptation of difficulty, and integration with agent learning objectives. LevelEnv has been central to frameworks such as TerrainRLSim (for continuous-control RL benchmarks) (Berseth et al., 2018) and GenEnv (for LLM agent–simulator co-evolution) (Guo et al., 22 Dec 2025), where it operationalizes an environment policy that continuously evolves the task distribution in response to agent competence.

1. Formal MDP Structure of LevelEnv

Let a LevelEnv be defined by the tuple $(S, A, \Theta, P_\theta, P, r, \gamma)$, where $S$ is the state space (concatenating terrain and agent features in locomotion, or world and goal state in more general settings), $A$ is the action space, $\Theta$ is the space of environment generation parameters, $P_\theta$ is the mapping from parameters to a concrete environment ("level"), $P$ is the transition kernel, $r$ is the reward function, and $\gamma$ is the discount factor. In TerrainRLSim, for example, $s = [s_t; s_a]$ with terrain features $s_t \in \mathbb{R}^d$ and agent features $s_a \in \mathbb{R}^k$, and $A \subset \mathbb{R}^m$ is typically continuous (e.g., torques or muscle activations) (Berseth et al., 2018).

For a fixed level instantiated by $\theta \in \Theta$, the MDP is $$M(\theta) = (S, A, P(\cdot | \cdot, \cdot, \theta), r, \gamma)$$. The environment is thus indexable by $\theta$.

2. Parameterization and Procedural Generation

A core feature of LevelEnv is the explicit parameterization of the environment or task-generation process by a vector $\theta$, whose elements control structural features of the environment (e.g., obstacle spacing, heights, slopes in terrain; compositional properties in LLM-simulated tasks). The procedural generator samples each $\theta_j$ from a specified range, commonly via uniform or Beta distributions:

$$\theta_j \sim \mathrm{Uniform}(\theta_{j,\min}, \theta_{j,\max})$$

or

$$\theta_j \sim \theta_{j,\min} + (\theta_{j,\max} - \theta_{j,\min}) \cdot \mathrm{Beta}(\alpha_t, \beta_t).$$

In TerrainRLSim, the procedural terrain is constructed by stitching together segments (gaps, walls, steps, slopes) according to the realized $\theta$ (Berseth et al., 2018). In GenEnv, the simulator LLM dynamically generates datum-level tasks, with $\theta$ encompassing latent aspects of task structure and required skill (Guo et al., 22 Dec 2025).

3. Difficulty Measurement, Control, and Adaptive Curriculum

LevelEnv supports principled difficulty metrics and adaptive curricula through two axes:

• Parameter range control: Expanding parameter intervals increases expected difficulty (e.g., larger obstacle heights; more complex task constraints). A scalar difficulty can be constructed as

$$D(\theta) = \sum_j w_j \cdot (\theta_{j,\max} - \theta_{j,\min})$$

or other monotonic surrogates.

• Empirical agent performance: For policy $\pi$, define empirical success rate over $N$ levels as

$$S(\pi) = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\pi \text{ solves } \theta^{(i)}\}$$

The LevelEnv distribution $P(\theta)$ is then adapted such that $S(\pi)$ tracks a target band (e.g., $80\%$ in RL, or parameter $\alpha$ in GenEnv) (Berseth et al., 2018, Guo et al., 22 Dec 2025).

In GenEnv, difficulty alignment is operationalized via the $\alpha$-Curriculum Reward,

$$R_\alpha(\hat{p}) = \exp\left(-\beta (\hat{p} - \alpha)^2\right)$$

where $\hat{p}$ is the mini-batch empirical success rate and $\beta$ modulates sensitivity to matching the ZPD. Only batches with $|\hat{p} - \alpha| \leq k_\text{min}$ participate in simulator updates.

Mechanistically, this guarantees that the agent is presented with tasks at intermediate difficulty, maximizing the policy-gradient norm and learning efficiency (see Proposition 3.1 in (Guo et al., 22 Dec 2025)).

4. Agent–Environment Co-evolution and Training Algorithm

LevelEnv enables dual-policy, co-evolutionary training regimes. The environment simulator (LevelEnv policy $\pi_\text{env}$) generates a batch of tasks, the agent ($\pi_\text{agent}$) attempts these tasks, and outcomes are measured. Both policies are updated in tandem: the agent maximizes task rewards, and LevelEnv maximizes the $\alpha$-Curriculum Reward. Key algorithmic steps are:

1. Task generation via $\pi_\text{env}$, yielding a batch $\{\tau_i\}$.
2. Agent rollout over each task, computing per-instance rewards and empirical success $\hat{p}$.
3. Curriculum scoring via $R_\alpha(\hat{p})$.
4. Selective simulator fine-tuning using reward-weighted regression, conditioned on proximity to $\alpha$.
5. Data aggregation into separate pools for next iteration agent and environment updates (Guo et al., 22 Dec 2025).

This loop ensures the environment continually calibrates difficulty to match the agent's skill, producing an emergent curriculum that remains closely coupled to learning progress.

5. Reward Structures and Transition Kernels

LevelEnv allows for flexible, task-specific reward shaping and transition dynamics. In RL, a canonical reward includes velocity, survival, control-energy, goal-reaching, and fall penalties:

$$r(s_t,a_t,s_{t+1}) = w_v v_x(s_{t+1}) + w_\text{surv} \mathbf{1}\{ \text{upright} \} - w_u \| a_t \|^2 + w_\text{goal} \mathbf{1}\{\text{goal}\} - w_\text{fall} \mathbf{1}\{\text{fallen}\}$$

Transition kernels are typically deterministic via environment physics, $s_{t+1} = f_\text{physics}(s_t, a_t; \theta)$, but may incorporate stochasticity for sensor noise or disturbances (Berseth et al., 2018).

In LLM settings, reward is computed by explicit comparison of the agent's output $a'_i$ to ground truth $a_i$, with per-instance $R_\text{agent}(a'_i,a_i) \in [0,1]$.

6. Implementation Interfaces and Generalization

LevelEnv-compatible frameworks expose APIs for environment instantiation, rollout, and reproducibility, exemplified by the Gym-style interface in TerrainRLSim:

Method	Description	Reference
getEnvsList()	List registered environment names	(Berseth et al., 2018)
getEnv(...)	Instantiate environment by name and config	(Berseth et al., 2018)
env.reset()	Sample $\theta \sim P(\theta)$ and reset environment	(Berseth et al., 2018)
env.step(a)	Advance MDP, returning $(s', r, \text{done}, \text{info})$	(Berseth et al., 2018)
setRandomSeed(seed)	Reproducible terrain draws	(Berseth et al., 2018)

For new morphologies (e.g., changing agent embodiment), only the agent-feature portion of $S$ and the action space $A$ need modification. The LevelEnv mechanism and procedural generator remain invariant, facilitating transfer and comparative experiments (Berseth et al., 2018).

This suggests that LevelEnv is broadly applicable to any training regime requiring systematic, adaptive task generation, spanning embodied RL, LLM-based agents, and other learning domains.

7. Empirical Outcomes and Theoretical Guarantees

Empirical results across both RL and LLM domains demonstrate significant gains in learning efficiency and final agent performance. In GenEnv, LevelEnv improved agent success on ALFWorld (from $14.2\%$ to $54.5\%$), API-Bank ($61.6\% \rightarrow 79.1\%$), BFCL ($7.0\% \rightarrow 41.8\%$), and surpassed much larger models with greater data efficiency (Guo et al., 22 Dec 2025). In TerrainRLSim, LevelEnv enables a systematic sweep from trivial to highly challenging environments entirely via parameter tuning (Berseth et al., 2018).

Theoretical analysis confirms that maintaining the agent's empirical success near the center of the ZPD (e.g., $p \approx 0.5$) maximizes expected learning gradients, and that empirical difficulty estimates are sufficient to guide reliable environment curriculum ranking given enough samples (Guo et al., 22 Dec 2025).

In sum, LevelEnv formalizes a data-evolving paradigm wherein environment design is a learnable, adaptive, and difficulty-aligned process, yielding faster convergence and superior final agent capability relative to static or untargeted curricula (Berseth et al., 2018, Guo et al., 22 Dec 2025).