LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems (2504.01963v1)

Abstract: This survey investigates foundational technologies essential for developing effective LLM-based multi-agent systems. Aiming to answer how best to optimize these systems for collaborative, dynamic environments, we focus on four critical areas: Architecture, Memory, Planning, and Technologies/Frameworks. By analyzing recent advancements and their limitations - such as scalability, real-time response challenges, and agent coordination constraints, we provide a detailed view of the technological landscape. Frameworks like the Mixture of Agents architecture and the ReAct planning model exemplify current innovations, showcasing improvements in role assignment and decision-making. This review synthesizes key strengths and persistent challenges, offering practical recommendations to enhance system scalability, agent collaboration, and adaptability. Our findings provide a roadmap for future research, supporting the creation of robust, efficient multi-agent systems that advance both individual agent performance and collective system resilience.

• The paper presents a detailed survey of LLM-based multi-agent architectures like CMD, CoA, Agent Forest, and MoA, highlighting improved scalability and specialization.
• It analyzes planning frameworks such as AdaPlanner and ReAct, showcasing adaptive reasoning and action integration for complex, long-horizon tasks.
• Memory strategies like RAG and MemoryBank, together with frameworks like AutoGen and MetaGPT, are evaluated to enhance system functionality and collaborative performance.

LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems

Introduction

The paper "LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems" (2504.01963) examines the crucial technological components of developing robust LLM-based multi-agent systems. With LLMs exhibiting unprecedented capabilities in single-agent tasks, there is significant interest in exploiting these models for collaborative multi-agent applications. This survey investigates four pivotal areas: Architecture, Memory, Planning, and Technologies/Frameworks, aiming to understand their roles and contributions to enhancing multi-agent systems.

Figure 1: Overview of Survey Methodology

Architecture

LLM-based multi-agent architectures focus on structuring interactions among agents to tackle complex reasoning tasks more effectively than individual agents.

• Conquer-and-Merge Discussion (CMD) engages agents in discussion-driven problem-solving, yielding superior reasoning outcomes over single-agent approaches.
• Chain-of-Agents (CoA) enables processing of longer contexts by dividing tasks among worker agents, effective in contexts surpassing individual LLM token limits.
• Agent Forest improves performance through a voting mechanism among multiple agents, highlighting the benefits of expanded agent participation.
• Mixture-of-Agents (MoA) utilizes layered roles with proposers and aggregators, achieving robust collaboration and improved outputs across benchmarks.

Analyzing these architectures reveals advantages in scalability and task specialization but acknowledges the complexity in orchestration and agent management.

Planning

Planning frameworks in LLM-based multi-agent systems integrate sophisticated reasoning and action strategies, pivotal for navigating dynamic environments.

• AdaPlanner introduces adaptive planning using real-time feedback, enhancing flexibility for complex, long-horizon tasks.
• ChatCoT leverages tool-augmented CoT reasoning for chat-based interactions, improving multi-step task handling.
• KnowAgent employs an action knowledge base, refining planning paths and reducing hallucinations.
• ReAct synergizes reasoning and action sequences, enhancing task-specific actions with external data interactions.

These frameworks reinforce the necessity of adaptive strategies and interleaved reasoning-action cycles for effective task execution.

Memory

Memory mechanisms are critical for the functional scaling and adaptability of LLM agents, enabling effective information retention and retrieval.

• Vector Databases support efficient data handling, complementing LLM constraints and mitigating inaccuracies via external knowledge access.
• Retrieval Augmented Generation (RAG) pioneers dynamic memory leveraging external data sources during language generation, improving specificity and factual accuracy.
• ChatDB integrates symbolic memory (SQL databases) to enhance multi-hop reasoning and minimize error propagation.
• MemoryBank offers long-term memory retention, simulating human forgetfulness to maintain contextual relevance over time.

Tailored memory strategies facilitate diverse intra-agent interactions, from rapid response demands to complex databased memory tasks.

Technologies / Frameworks

Frameworks for LLM-based multi-agent systems forge environments for enhanced collaboration and efficient task execution.

• AutoGen supports multifaceted agent interactions through chat-optimized frameworks, combining LLMs, human inputs, and tools for diverse applications.
• CAMEL leverages role-playing to optimize task-focused agent cooperation, minimizing human intervention.
• MetaGPT utilizes standardized procedures encoded in prompts, refining multi-agent collaboration and problem-solving processes.
• LangGraph capitalizes on graph technology to refine information retrieval and integration for knowledge-based tasks.

These frameworks highlight the imperatives of dynamic interaction management and seamless integration with modern multi-agent applications.

Conclusion

This survey refines the understanding of multi-agent systems leveraging LLMs, emphasizing innovative approaches in architecture, planning, and memory, supported by cutting-edge frameworks. While the Mixture of Agents (MoA) architecture and ReAct planning framework demonstrate promising outcomes in enhancing collaboration and reasoning, memory strategies and frameworks remain contingent on specific system needs. Continual research will address computational constraints and optimize the orchestration of agent roles, forging a path toward adaptive, resilient multi-agent systems enabled by evolving LLM capabilities.