RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation (2505.24442v1)

Abstract: Although multi-agent systems based on LLMs show strong capabilities on multiple tasks, they are still limited by high computational overhead, information loss, and robustness. Inspired by ResNet's residual learning, we propose Residual Mixture-of-Agents (RMoA), integrating residual connections to optimize efficiency and reliability. To maximize information utilization from model responses while minimizing computational costs, we innovatively design an embedding-based diversity selection mechanism that greedily selects responses via vector similarity. Furthermore, to mitigate iterative information degradation, we introduce a Residual Extraction Agent to preserve cross-layer incremental information by capturing inter-layer response differences, coupled with a Residual Aggregation Agent for hierarchical information integration. Additionally, we propose an adaptive termination mechanism that dynamically halts processing based on residual convergence, further improving inference efficiency. RMoA achieves state-of-the-art performance on the benchmarks of across alignment, mathematical reasoning, code generation, and multitasking understanding, while significantly reducing computational overhead. Code is available at https://github.com/mindhunter01/RMoA.

Enhancing Multi-Agent Systems with Residual Mixture-of-Agents

The paper "RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation" introduces an innovative approach to addressing key challenges in the efficiency, reliability, and robustness of Multi-Agent Systems (MAS) integrated with LLMs. By drawing inspiration from residual learning techniques found in ResNet architecture, the authors propose the Residual Mixture-of-Agents (RMoA), a framework designed to optimize the MAS architecture through incorporation of residual connections.

One of the primary objectives of the RMoA framework is to mitigate the computational overhead and information loss that often hinders traditional MAS frameworks such as Mixture-of-Agents (MoA) and Sparse Mixture-of-Agents (SMoA). This is achieved through several novel mechanisms:

1. Greedy Diversity Embedding Selection: This mechanism maximizes the use of information while minimizing redundancy in computational costs. It employs vector similarity to select the most diverse model responses, which helps in improving information heterogeneity and preventing information loss across layers.
2. Residual Extraction and Aggregation Agents: These agents enhance the hierarchical information integration by preserving incremental changes across layers. The extraction agent identifies the differences between successive layers' responses, while the aggregation agent integrates these differences to produce a refined output. This method effectively counters the iterative information degradation that can occur in deep-layer processing.
3. Adaptive Termination Mechanism: By dynamically determining the optimal point for halting further processing based on residual convergence, this mechanism further enhances inference efficiency and reduces unnecessary computations.

The RMoA framework, as demonstrated in extensive experimental evaluations across various benchmarks including alignment tasks, mathematical reasoning, code generation, and multitasking understanding, consistently achieved state-of-the-art performance, improving upon both reliability and efficiency while reducing overall computational overhead. Notably, on mathematical reasoning and code understanding benchmarks, RMoA outperforms existing MAS architectures by a significant margin.

The theoretical implications of RMoA suggest advancements in the scalability and adaptability of MAS architectures for complex reasoning tasks. Practically, the framework paves the way for more efficient deployment of LLM-based systems in environments where computational resources are constrained and rigorous reasoning tasks require robust performance. The paper anticipates future developments in AI research to further explore the integration of residual connections in MAS frameworks, potentially enabling even deeper layers of reasoning and collaboration among agents.

Overall, the RMoA represents a significant step forward in addressing critical challenges within multi-agent LLM systems, promising enhancements in terms of both processing speed and output reliability. By leveraging diverse information through optimized selection and residual processing, RMoA sets a new benchmark for MAS efficiency in tackling complex tasks across domains.