ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems (2408.02248v2)

Abstract: Recently, there has been increasing interest in using LLMs to construct complex multi-agent systems to perform tasks such as compiling literature reviews, drafting consumer reports, and planning vacations. Many tools and libraries exist for helping create such systems, however none support recursive multi-agent systems -- where the models themselves flexibly decide when to delegate tasks and how to organize their delegation structure. In this work, we introduce ReDel: a toolkit for recursive multi-agent systems that supports custom tool-use, delegation schemes, event-based logging, and interactive replay in an easy-to-use web interface. We show that, using ReDel, we are able to easily identify potential areas of improvements through the visualization and debugging tools. Our code, documentation, and PyPI package are open-source and free to use under the MIT license at https://github.com/zhudotexe/redel.

• The paper introduces a robust toolkit that enables LLMs to dynamically decompose and delegate complex tasks in recursive multi-agent systems.
• It integrates Python-based tools, custom delegation schemes, and interactive event logging via a user-friendly web interface for enhanced system debugging and analysis.
• Experimental evaluations, particularly with GPT-4o, demonstrate significant performance improvements over single-agent baselines on benchmarks like FanOutQA and TravelPlanner.

ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems

The paper "ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems" introduces ReDel, an advanced toolkit designed to extend the capabilities of LLMs in creating recursive multi-agent systems. This toolkit offers an innovative approach where agents can dynamically delegate tasks in a recursive manner, which is a departure from the traditional static systems where human-defined layouts are indispensable.

Overview

ReDel emphasizes flexibility and autonomy by allowing a root agent in the system to decompose complex tasks into smaller subtasks and delegate them to newly spawned sub-agents. This recursive delegation provides an environment where agents can decide which tasks need further breakdown or if they are manageable enough to be completed independently. The toolkit supports custom tool-use, delegation schemes, event-based logging, and interactive replay, all accessible through a user-friendly web interface.

Specifically, ReDel advances previous methods of task decomposition seen in works by Lee and Kim (2023), Khot et al. (2023), and others by utilizing modern LLMs' native tool use capability to perform zero-shot task decomposition. This allows ReDel to be used flexibly across different domains without requiring hand-crafted examples or fine-tuning.

System Capabilities

The ReDel system is designed with the practicalities of academic research and experimental investigation in mind. Unlike many existing frameworks, ReDel is fully open-source and built to be highly modular, promoting easy experimentation and system iteration. Key features include:

• Tool Usage: Agents in ReDel are equipped with Python-based tools, allowing them to interact with different external resources or environments, such as web browsing or email functions.
• Delegation Schemes: The toolkit includes predefined schemes such as DelegateOne and DelegateWait, which allow for synchronous and asynchronous task handling, respectively. These schemes provide a foundation for users to build more complex delegation behaviors.
• Event Logging: An event-driven approach ensures that all interactions within the system are logged, aiding in post-hoc analysis and system debugging. Special attention is given to custom event definition, allowing developers to gain insight into specific system actions.
• Web Interface: ReDel features a comprehensive web interface supporting interactive sessions, event replay for analyzing task execution, and visualization of delegation graphs. These capabilities enable users to deeply explore and optimize their multi-agent systems.

Experimental Evaluation

ReDel's efficacy is validated through evaluations on benchmarks like FanOutQA, TravelPlanner, and WebArena, with results showing that recursive systems consistently outperform single-agent baselines. Particularly notable is ReDel's performance with the GPT-4o model, which demonstrated significant improvements, even surpassing published state-of-the-art results on FanOutQA and TravelPlanner. These results indicate that recursive systems, as facilitated by ReDel, can effectively tackle complex tasks by breaking them down into manageable components.

Error Analysis

The paper identifies two prominent failure modes in recursive multi-agent systems: overcommitment and undercommitment. Overcommitment occurs when an agent attempts to solve overly complex tasks without adequate delegation, leading to context window limitations. Undercommitment, conversely, involves unnecessary re-delegation and potential infinite loops of task delegation. Addressing these modes through analysis could yield potential improvements in system design and task handling efficiency.

Future Directions

The research opens new avenues for developing more sophisticated recursive multi-agent systems by providing a robust framework that seamlessly integrates with existing LLM capabilities. Going forward, enhancements could focus on better decision-making algorithms for agent delegation, fine-tuning agent collaboration, and extending the range of tools available for agents to use. Additionally, investigating alternative evaluation methodologies and integrating reinforcement learning strategies might further bolster the system’s prowess in managing complex, real-world tasks.

In conclusion, ReDel represents a significant step forward in LLM-powered multi-agent systems, offering researchers a powerful toolkit for exploring the nuances of recursive task delegation and system-building. With its expansive capabilities and open-source accessibility, ReDel is poised to play a crucial role in the evolution of intelligent multi-agent systems.