Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents (2403.02502v2)

Abstract: LLMs have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO. This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.

• The paper introduces Exploration-Based Trajectory Optimization (ETO) to iteratively improve LLM agents by learning from failure trajectories.
• It employs a two-phase approach—exploration and contrastive Direct Policy Optimization—to refine decision-making in various task domains.
• Experimental results on tasks like WebShop, ScienceWorld, and ALFWorld demonstrate significant performance improvements and better generalization over traditional methods.

Enhancement of LLM Agents through Exploration-Based Trajectory Optimization

Introduction

LLMs have transformed the landscape of AI research, offering significant advancements in natural language understanding and generation. This paper introduces a novel method, Exploration-Based Trajectory Optimization (ETO), aimed at enhancing the capabilities of LLM agents in complex task environments. In contrast to conventional approaches that primarily focus on successful expert-driven trajectories, ETO emphasizes the importance of learning from exploratory failures. This paradigm shift allows LLM agents to iteratively refine their strategies through a constructive feedback loop, resulting in substantial performance improvements across diverse task domains.

Methodology

ETO is grounded in a two-phase learning cycle: exploration and training. Initially, a baseline agent is constructed through Supervised Fine-Tuning (SFT) on expert trajectories, serving as the foundation for subsequent optimization. During exploration, this agent interacts with the environment, generating a set of failure trajectories which are then contrastively paired with existing expert successes. The core innovation lies in the second phase, where the agent updates its policy by employing a contrastive learning approach, specifically leveraging Direct Policy Optimization (DPO). This cycle of exploration and refinement fosters a continuous learning process, enabling the agent to develop more sophisticated decision-making capabilities over time.

Experimental Validation

The effectiveness of ETO is thoroughly evaluated through rigorous experiments conducted on three challenging interactive tasks: WebShop, ScienceWorld, and ALFWorld. These datasets encompass various application domains, including web navigation, simulated science experiments, and embodied household tasks. The results unequivocally demonstrate ETO's superiority over SFT and other baseline methodologies, showcasing substantial performance gains. Notably, ETO exhibits remarkable proficiency in out-of-distribution generalization scenarios, evidenced by its significant improvements in unseen task environments. Such findings underscore the method's robustness and versatility in adapting to novel challenges.

Implications and Future Directions

The introduction of ETO represents a significant stride towards realizing more autonomous, efficient, and adaptable LLM agents. By harnessing the learning potential from failure trajectories, ETO unlocks new possibilities for agent development beyond the confines of expert demonstration datasets. This research not only contributes a powerful tool for enhancing agent performance but also sets a conceptual foundation for future explorations into iterative, exploration-based learning paradigms. Looking forward, the versatility and scalability of ETO open avenues for its application across a broader spectrum of AI tasks, potentially revolutionizing how autonomous agents learn and interact with complex environments.

Conclusion

Exploration-Based Trajectory Optimization (ETO) offers a novel and effective strategy for enhancing the capabilities of LLMs as agents in complex interactive environments. By prioritizing the learning opportunities inherent in failure trajectories, ETO advances the state of the art in agent optimization, showcasing remarkable performance improvements and strong generalization capabilities. This research not only elevates the performance benchmarks for LLM agents but also contributes valuable insights into iterative learning dynamics, paving the way for future developments in AI and autonomous systems.