AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

Abstract: Building generalist agents that can handle diverse tasks and evolve themselves across different environments is a long-term goal in the AI community. LLMs are considered a promising foundation to build such agents due to their generalized capabilities. Current approaches either have LLM-based agents imitate expert-provided trajectories step-by-step, requiring human supervision, which is hard to scale and limits environmental exploration; or they let agents explore and learn in isolated environments, resulting in specialist agents with limited generalization. In this paper, we take the first step towards building generally-capable LLM-based agents with self-evolution ability. We identify a trinity of ingredients: 1) diverse environments for agent exploration and learning, 2) a trajectory set to equip agents with basic capabilities and prior knowledge, and 3) an effective and scalable evolution method. We propose AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration. AgentGym also includes a database with expanded instructions, a benchmark suite, and high-quality trajectories across environments. Next, we propose a novel method, AgentEvol, to investigate the potential of agent self-evolution beyond previously seen data across tasks and environments. Experimental results show that the evolved agents can achieve results comparable to SOTA models. We release the AgentGym suite, including the platform, dataset, benchmark, checkpoints, and algorithm implementations. The AgentGym suite is available on https://github.com/WooooDyy/AgentGym.

• The paper introduces AgentGym, a framework that enables LLM-based agents to autonomously evolve via 89 tasks across 14 diverse environments.
• It employs behavioral cloning paired with interactive training to improve agent generalization and adaptability, with strong benchmarks on WebShop and ALFWorld.
• Empirical results demonstrate enhanced success rates and efficiency, marking a step forward in scalable, minimally supervised agent training.

"AgentGym: Evolving LLM-based Agents across Diverse Environments"

The paper "AgentGym: Evolving LLM-based Agents across Diverse Environments" introduces a framework called AgentGym to facilitate the development and assessment of generally-capable agents driven by LLMs. This framework addresses the challenges of building agents that can evolve their abilities autonomously in diverse environments and tasks.

Framework Description

AgentGym encompasses a wide variety of tools and environments engineered to test and enhance the capabilities of LLM-based agents. It integrates a platform with 14 unique environments covering 89 distinct tasks. The environments span various categories such as web navigation, household tasks, and digital simulation environments like games and programming tasks.

Figure 1: Overview of the AgentGym framework, facilitating agent evolution and evaluation across diverse environments.

Key Components

1. Environments: AgentGym includes an extensive set of environments deployed as HTTP services, allowing agents to interact with them via well-defined APIs. The diversity of tasks ensures that agents develop robust and generalized capabilities.
2. AgentTraj and AgentEval: AgentGym provides a trajectory dataset named AgentTraj for training a base agent and a trajectory dataset named AgentTraj-L for testing the upper limits of behavioral cloning (BC). The AgentEval benchmark suite allows comprehensive evaluation of the agents.
3. Interactive Training: The platform supports real-time feedback and multi-round interactions, essential for the evolutionary process of the agents.

Methodology

The paper proposes a novel self-evolution method, AgentEvol, which allows agents to explore and learn from diverse environments.

Evolution Approach

1. Behavioral Cloning (BC): Initially, the agents undergo a behavioral cloning phase using AgentTraj to acquire basic skills and knowledge.
2. Exploration and Learning:
   • Exploration Step: Agents interact with various environments based on an enlarged instruction set, collecting updated trajectories.
   • Learning Step: The collected data, combined with BC data, is used to adjust the agent's policy by maximizing a weighted objective function, thereby enhancing agent performance across previously unseen tasks.

     Figure 2: An illustration of self-evolution for generally-capable LLM-based agents, showing the transition from initial knowledge acquisition to autonomous exploration and learning.

Implementation and Results

The paper presents empirical results demonstrating the framework and methodology. AgentEvol agents outperform other state-of-the-art (SOTA) models, showcasing improvements particularly in generalization and adaptability across tasks like WebShop, ALFWorld, and BabyAI.

Performance Metrics

• Success Rate: Measures the percentage of tasks where the agent achieves the given objectives.
• Interactive Rounds: Indicates the efficiency of agents, with fewer rounds often correlating with higher performance.

Impact and Future Directions

AgentEvol signifies an important step toward designing LLM-based agents that require minimal human supervision while achieving high competence across varied tasks. It addresses scalability challenges by leveraging environment feedback effectively, promoting a more sustainable approach to training data acquisition.

Future Developments

• Incorporation of varied reward structures: Adapting agents to environments with complex reward architectures could enhance flexibility.
• Testing on more complex tasks: Extending the types of tasks within AgentGym to further challenge the generalization capabilities of LLM-based agents.

  Figure 3: Case study of WebShop showing the agent's enhanced decision-making capability after evolution, effectively utilizing feedback and task requirements.

Conclusion

AgentGym provides a robust framework that lays crucial groundwork for building generally-capable, self-evolving agents. Through structured environments, comprehensive data sets, and innovative methods like AgentEvol, it paves the way for future research and advancements in autonomous LLM-based agents.