BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents

Abstract: The massive successes of LLMs encourage the emerging exploration of LLM-augmented Autonomous Agents (LAAs). An LAA is able to generate actions with its core LLM and interact with environments, which facilitates the ability to resolve complex tasks by conditioning on past interactions such as observations and actions. Since the investigation of LAA is still very recent, limited explorations are available. Therefore, we provide a comprehensive comparison of LAA in terms of both agent architectures and LLM backbones. Additionally, we propose a new strategy to orchestrate multiple LAAs such that each labor LAA focuses on one type of action, \textit{i.e.} BOLAA, where a controller manages the communication among multiple agents. We conduct simulations on both decision-making and multi-step reasoning environments, which comprehensively justify the capacity of LAAs. Our performance results provide quantitative suggestions for designing LAA architectures and the optimal choice of LLMs, as well as the compatibility of both. We release our implementation code of LAAs to the public at \url{https://github.com/salesforce/BOLAA}.

• The paper introduces BOLAA, a novel method for orchestrating multiple LLM-augmented agents to optimize performance on complex tasks.
• It rigorously benchmarks various agent architectures, including ZS-LAA and ZST-LAA, to demonstrate effective multi-agent collaboration.
• Results reveal that integrating specialized LAAs via BOLAA can outperform single large models in tasks like web navigation and knowledge reasoning.

Introduction to LLM-Augmented Autonomous Agents

Autonomous agents empowered by LLMs are reshaping the landscape of machine intelligence. This paper presents a comprehensive analysis of the LLM-augmented Autonomous Agents (LAAs) and introduces a new method named BOLAA for orchestrating multiple LAAs to efficiently address complex tasks.

LAA Architectures and LLM Integration

The development of LAAs involves two crucial components - the agent architecture and the LLM backbone. Agent architectures determine the interaction strategy with the environment, while LLMs provide the necessary language understanding and generation capabilities. While previously the focus was either on agent design or LLM capabilities, this study proposes an integrative approach to assess them together. Several architectures like Zeroshot LAA (ZS-LAA), ZeroshotThink LAA (ZST-LAA), and others that incorporate planning and self-think flows are rigorously benchmarked.

Multi-Agent Orchestration with BOLAA

As tasks grow in complexity, employing single LAAs becomes less efficient. The proposed BOLAA architecture comes into play here, featuring a controller module that coherently manages communication between specialized LAAs focused on discrete action types. The paper suggests that the BOLAA approach could help balance computational resources and performance more effectively compared to a single powerful LAA.

Implications and Future Directions

The study's findings indicate that a well-chosen combination of LAA architecture and LLM can achieve superior performance for web navigation and knowledge reasoning tasks. Experiments reveal that multi-agent strategies like BOLAA often lead to remarkable improvements, notably when comprising lesser LLMs. These insights open doors to fine-tuning smaller, specialized LAAs instead of a single, large, general LAA.

The paper paves the way for future research, including refining BOLAA to handle compounding actions and further exploration of LAA architectures. The resulting codebase for these LAAs is open for access, allowing for community involvement and continuous improvement.