SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning (2502.04780v1)

Abstract: Multi-agent AI systems powered by LLMs are increasingly applied to solve complex tasks. However, these systems often rely on fragile, manually designed prompts and heuristics, making optimization difficult. A key challenge in optimizing multi-agent systems is acquiring suitable training data for specialized agents. We introduce SiriuS, a self-improving, reasoning-driven optimization framework for multi-agent systems. Central to our approach is the construction of an experience library: a repository of high-quality reasoning trajectories. The library is built by retaining reasoning steps that lead to successful outcomes, providing a robust training set for optimizing multi-agent system. Additionally, we introduce a library augmentation procedure that refines unsuccessful trajectories, further enriching the library. SiriuS boosts performance by 2.86\% to 21.88\% on reasoning and biomedical QA and enhances agent negotiation in competitive settings. Our results show that SiriuS enhances multi-agent performance while generating reusable data for self-correction and self-play enhancement in the future.

• The paper introduces RoyalBlue, a methodology that improves LLM-based multi-agent systems using an experience library of successful reasoning trajectories and bootstrapped reasoning for optimization.
• RoyalBlue demonstrates improved performance across reasoning tasks and biomedical question-answering, showing accuracy boosts ranging from 2.86% to 21.88% in empirical evaluations.
• The approach promises practical efficiency gains by reducing reliance on human supervision and theoretically opens up exploration into complex multi-agent dynamics and reinforcement learning paradigms.

RoyalBlue: Self-Improving Multi-Agent Systems via Bootstrapped Reasoning

The paper introduces "RoyalBlue," a methodology aimed at enhancing the performance of multi-agent AI systems that leverage LLMs. The motivation behind this work resides in the challenges associated with optimizing these systems due to their reliance on fragile and manually designed prompts, which often result in optimization difficulties. RoyalBlue proposes a solution by introducing an experience library, serving as a repository of successful reasoning trajectories, and employing bootstrapped reasoning as an optimization framework.

RoyalBlue is designed to capitalize on successful collaborative interactions among agents by storing complete interaction trajectories that led to positive outcomes. These trajectories are then used to fine-tune each agent within the multi-agent system, enabling them to iteratively identify and adopt effective collaboration strategies. Furthermore, RoyalBlue incorporates a library augmentation procedure, aiming to refine trajectories from unsuccessful attempts through resampling with feedback, thereby enriching the dataset used for optimization.

The efficacy of RoyalBlue is demonstrated through empirical results in various domains. Specifically, it improves performance across reasoning tasks and biomedical question-answering (QA), with performance boosts in accuracy ranging from 2.86% to 21.88%. These improvements underline RoyalBlue’s potential in not just enhancing multi-agent coordination and problem-solving capabilities but also in generating scalable self-correcting data for future applications of self-play and agent negotiation in competitive settings.

Methodology

The multi-agent framework proposed in RoyalBlue is framed as a tuple consisting of several interacting components: state, actions, transition functions, reward signals, agents, and a goal. Communication between agents is structured via a directed graph, which defines the flow of interactions. RoyalBlue employs a straightforward strategy to iteratively fine-tune agent policy parameters over multiple iterations, thereby refining agent behaviors through systematic feedback and trajectory evaluations.

Central to RoyalBlue's methodology is a bootstrapped reasoning approach inspired by recent research that emphasizes learning from reasoning patterns that have succeeded in practice. The framework allows agents to indirectly learn which collaboration patterns are more likely to yield successful outcomes by observing entire interaction sequences rather than isolated decisions.

Multi-Agent Setting Exploration

A notable aspect of the paper is the exploration of various multi-agent settings to highlight the potential of RoyalBlue. These settings include problem-solving tasks, actor-critic frameworks, and competitive negotiation scenarios. In each configuration, the interactions among agents are tailored to exploit diverse disciplinary expertise, foster iterative reasoning enhancement via critique and feedback, or simulate competitive agent behavior under opposing goals.

For instance, in problem-solving contexts, agents assume roles such as physicists and mathematicians to collaboratively address domain-specific challenges. In competitive settings, agents simulate negotiation strategies under constrained environments, showcasing improved adaptability and strategic decision-making afforded by RoyalBlue's optimization process.

Implications and Future Directions

The introduction of RoyalBlue points toward significant practical and theoretical implications for the development of future AI. Practically, this approach promises substantial efficiency gains in training multi-agent systems by reducing dependence on human supervision and accelerating the learning of collaborative strategies. Theoretically, it opens up avenues for further exploration into the dynamics of multi-agent interactions, potentially incorporating reinforcement learning paradigms beyond traditional settings.

For future work, the foundation laid by RoyalBlue could be extended to explore more complex and nuanced agent interactions, possibly incorporating differential reward structures or expanding applicability across different problem domains. Additionally, the framework could benefit from advancements in LLM architectures to enhance processing efficiency and scalability of multi-agent systems.