GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators (2512.19682v2)

Abstract: Training capable LLM agents is critically bottlenecked by the high cost and static nature of real-world interaction data. We address this by introducing GenEnv, a framework that establishes a difficulty-aligned co-evolutionary game between an agent and a scalable, generative environment simulator. Unlike traditional methods that evolve models on static datasets, GenEnv instantiates a dataevolving: the simulator acts as a dynamic curriculum policy, continuously generating tasks specifically tailored to the agent's ``zone of proximal development''. This process is guided by a simple but effective $α$-Curriculum Reward, which aligns task difficulty with the agent's current capabilities. We evaluate GenEnv on five benchmarks, including API-Bank, ALFWorld, BFCL, Bamboogle, and TravelPlanner. Across these tasks, GenEnv improves agent performance by up to \textbf{+40.3\%} over 7B baselines and matches or exceeds the average performance of larger models. Compared to Gemini 2.5 Pro-based offline data augmentation, GenEnv achieves better performance while using 3.3$\times$ less data. By shifting from static supervision to adaptive simulation, GenEnv provides a data-efficient pathway for scaling agent capabilities.

• The paper proposes a difficulty-aligned co-evolutionary curriculum where an environment LLM adapts tasks to the agent’s capabilities, enhancing learning.
• The framework achieves up to +40.3% improvement on benchmarks, matching larger models with fewer synthetic data samples.
• The adaptive curriculum stabilizes training by calibrating task difficulty to a target success rate, ensuring accelerated learning and robust generalization.

GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators

Motivation and Challenge

LLM agents exhibit considerable capabilities in complex, interactive tasks such as tool use, web navigation, and embodied planning. However, current approaches are constrained by static, pre-collected expert datasets that fail to capture the diversity and adaptivity required for robust open-ended problem-solving. Such static corpora result in high data curation costs, poor generalization to environment variation, and brittle agent behaviors. Prior work on synthetic and augmented data generation demonstrates only limited improvements, largely due to non-adaptive, untargeted task curation. The persistent dependence on scale and on static teacher-generated datasets signals the need for a data-evolving training paradigm.

GenEnv Framework and Data-Evolving Paradigm

GenEnv introduces a difficulty-aligned co-evolutionary curriculum between two interacting LLM policies: an Agent Policy ($\pi_\text{agent}$) and an Environment Policy ($\pi_\text{env}$). Rather than optimizing the agent on a fixed dataset, GenEnv implements a data-evolving pipeline in which the environment LLM generates tasks adaptively calibrated to the agent's current capability, specifically targeting the agent’s "zone of proximal development". The system is structured as a two-player game: the agent maximizes per-task reward, while the environment maximizes a differentiable curriculum reward that aligns empirical task success rates with a target value $\alpha$ (typically $0.5$), yielding maximal learning signal.

Figure 2: Contrasting traditional static-data training (top) with adaptive GenEnv training (bottom), highlighting the co-evolutionary loop and curriculum emergence.

Figure 1: Schematic of GenEnv’s interaction: environment generates, agent attempts, each updates on their respective reward, closing the online adaptive loop.

The agent receives reward for correct task completion or similarity for unstructured responses, and is trained with Group Relative Policy Optimization (GRPO). The environment receives a shaped curriculum reward $$R_\text{env}(\hat{p}) = \exp(-\beta(\hat{p}-\alpha)^2)$$, where $\hat{p}$ is the empirical success rate from rollouts. Training proceeds as a joint online process with separate datasets: an evolving pool of agent trajectories (for on-policy agent updates and rehearsal) and a weighted set of environment task proposals (for reward-weighted regression). This yields stable, monotonic improvements in agent reward, accuracy, and generalization.

Theoretical Analysis

The proposed $\alpha$-Curriculum Reward ensures that agent tasks of intermediate difficulty (as measured by target success rates near $\alpha$) maximize the variance term $p(1-p)$ of the policy gradient, thus supplying the strongest learning signal. A formal proposition demonstrates that this regime optimally balances successful and failed experiences, unlike tasks that are too easy or too hard. Furthermore, the ranking induced by $R_\text{env}$ is shown to be statistically consistent: as sample budget increases, the environment LLM reliably upweights task types closest in difficulty to the agent, as validated by sharp concentration bound analysis.

Experimental Results

Experiments span five challenging benchmarks: API-Bank, ALFWorld, BFCL, Bamboogle, and TravelPlanner. All evaluations use a uniform tool-calling setup with 7B LLM backbones and standardized evaluation metrics.

• Performance: GenEnv achieves up to +40.3% improvement on ALFWorld and substantial gains across all benchmarks versus strong 7B baselines (e.g., ReSearch, Qwen2.5-7B, SearchR1).
• Model scaling: GenEnv matches or surpasses the average performance of several 14B–72B-scale models, underlining the criticality of task alignment over merely scaling model parameters or offline data volume.

  Figure 3: GenEnv’s performance and data efficiency: outperforming larger-scale models and static data augmentation even with substantially fewer synthetic samples.

• Training stability: Learning curves exhibit monotonic improvement with no reward collapse or instability.

  Figure 4: Training stability is evident: reward and accuracy increase smoothly over epochs, without divergence.

• Curriculum emergence: The mean task complexity (as measured by response length) and the agent’s reasoning chain both increase during training, while agent success rate stays within the target calibration band, confirming self-organized curriculum generation.

  Figure 5: The simulator steadily increases task complexity; agent reasoning complexity co-evolves, with success rates tightly controlled.

Data Efficiency and Simulation Analysis

GenEnv's co-evolutionary alignment yields major data-efficiency improvements. Compared to strong Gemini 2.5 Pro-based static augmentation baselines (even with 3.3x more offline samples), GenEnv attains higher validation accuracy with much less synthetic data. Random or untargeted online simulators underperform difficulty-aligned GenEnv by 12.3%.

Figure 6: Comparison of methods shows that difficulty-aligned simulation yields higher agent performance than static or untargeted data scaling.

• Difficulty calibration: The agent's empirical success stabilizes precisely around the designed target $\alpha$ (e.g., 0.5) as training proceeds, empirically verifying the self-calibrated, dynamic curriculum opposed to the collapse seen in random or static baselines.

  Figure 7: Success rates on simulator-generated tasks converge to the target $\alpha$-band, empirically verifying the theoretical curriculum reward properties.

• Accelerated learning: Difficulty-alignment accelerates the elimination of unsolved failure modes relative to random generation baselines.

  Figure 8: GenEnv substantially increases solved task proportion and reduces failure rates faster than unguided online simulation.

Implications and Future Directions

GenEnv provides strong evidence that agent capability scaling is more efficiently realized through difficulty-aligned, adaptive simulation than simply increasing model or static dataset size. This offers a pathway for robust agent training where real-world interaction is costly or infeasible, and motivates future research on generalizing environment-LLM simulators, co-evolutionary frameworks, and scalable automatic curriculum learning in multi-modal and multi-agent scenarios.

The approach extends naturally to scenarios requiring continual learning, domain adaptation, or generalization to out-of-distribution tasks. Integrating further signals (e.g., safety, compositional reasoning, or feedback from human-in-the-loop evaluation) may deliver even stronger agent robustness. Theoretical analysis lays the groundwork for deeper study into curriculum shaping, self-play, and societal simulations for emergent agent skills.

Conclusion

GenEnv establishes a joint optimization framework that transitions LLM agent training from static data dependencies to on-policy, difficulty-aligned co-evolution with an environment LLM. Through a theoretically and empirically validated curriculum mechanism, GenEnv attains substantial improvements in both capability and data efficiency, highlighting the critical value of adaptive simulation as a foundation for the next generation of LLM agents.