BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents (2308.05960v1)

Abstract: The massive successes of LLMs encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs). An LAA is able to generate actions with its core LLM and interact with environments, which facilitates the ability to resolve complex tasks by conditioning on past interactions such as observations and actions. Since the investigation of LAA is still very recent, limited explorations are available. Therefore, we provide a comprehensive comparison of LAA in terms of both agent architectures and LLM backbones. Additionally, we propose a new strategy to orchestrate multiple LAAs such that each labor LAA focuses on one type of action, \textit{i.e.} BOLAA, where a controller manages the communication among multiple agents. We conduct simulations on both decision-making and multi-step reasoning environments, which comprehensively justify the capacity of LAAs. Our performance results provide quantitative suggestions for designing LAA architectures and the optimal choice of LLMs, as well as the compatibility of both. We release our implementation code of LAAs to the public at \url{https://github.com/salesforce/BOLAA}.

Introduction to LLM-Augmented Autonomous Agents

Autonomous agents empowered by LLMs are reshaping the landscape of machine intelligence. This paper presents a comprehensive analysis of the LLM-augmented Autonomous Agents (LAAs) and introduces a new method named BOLAA for orchestrating multiple LAAs to efficiently address complex tasks.

LAA Architectures and LLM Integration

The development of LAAs involves two crucial components - the agent architecture and the LLM backbone. Agent architectures determine the interaction strategy with the environment, while LLMs provide the necessary language understanding and generation capabilities. While previously the focus was either on agent design or LLM capabilities, this paper proposes an integrative approach to assess them together. Several architectures like Zeroshot LAA (ZS-LAA), ZeroshotThink LAA (ZST-LAA), and others that incorporate planning and self-think flows are rigorously benchmarked.

Multi-Agent Orchestration with BOLAA

As tasks grow in complexity, employing single LAAs becomes less efficient. The proposed BOLAA architecture comes into play here, featuring a controller module that coherently manages communication between specialized LAAs focused on discrete action types. The paper suggests that the BOLAA approach could help balance computational resources and performance more effectively compared to a single powerful LAA.

Implications and Future Directions

The paper's findings indicate that a well-chosen combination of LAA architecture and LLM can achieve superior performance for web navigation and knowledge reasoning tasks. Experiments reveal that multi-agent strategies like BOLAA often lead to remarkable improvements, notably when comprising lesser LLMs. These insights open doors to fine-tuning smaller, specialized LAAs instead of a single, large, general LAA.

The paper paves the way for future research, including refining BOLAA to handle compounding actions and further exploration of LAA architectures. The resulting codebase for these LAAs is open for access, allowing for community involvement and continuous improvement.