Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents (2408.07060v1)

Abstract: LLM agents have shown great potential in solving real-world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite. However, these sophisticated agent frameworks exhibit varying strengths, excelling in certain tasks while underperforming in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions as a meta-module atop existing SWE agent frameworks, managing agent collectives for enhanced problem-solving. Experimental results show that a DEI-guided committee of agents is able to surpass the best individual agent's performance by a large margin. For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI, making a 25% improvement and beating most closed-source solutions. Our best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative AI systems and their potential to solve complex software engineering challenges.

• The paper presents DEI, a framework that enhances issue resolution by aggregating the strengths of diverse SWE agents.
• It employs a re-ranking pipeline within a Contextual Markov Decision Process to intelligently select optimal candidate patches.
• Experimental results using SWE-Bench Lite demonstrate a significant performance boost, with ensemble resolve rates outperforming the best individual agent.

Diversity Empowers Intelligence: An Integrative Framework for Software Engineering Agents

The paper "Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents" presents a novel framework, Diversity Empowered Intelligence (DEI), to enhance the problem-solving efficacy of Software Engineering (SWE) agents. This framework operates as a meta-module atop existing SWE agent architectures to exploit the unique capabilities of each constituent agent. The paper presents both evaluations of individual agents and aggregated performance metrics when the DEI framework guides multiple agents. Notably, the results indicate a substantial improvement in issue resolution when employing DEI to harness agent diversity.

Introduction and Motivation

Recent advances in LLMs have significantly impacted the field of Software Engineering (SWE), facilitating tasks such as automated code generation, testing, and debugging. However, the performance of SWE agents, particularly in resolving real-world issues from repositories like GitHub, demonstrates substantial variability. These agents excel in specific tasks but may underperform in others. Recognizing this, the authors propose DEI as a method to aggregate and leverage the diverse strengths of multiple SWE agents.

The paper's core contribution is twofold:

1. Evaluation and Quantification of Agent Diversity: The authors highlight the high variability in performance across different agents and multiple runs of a single agent, suggesting untapped potential in SWE problem-solving.
2. Development of DEI Framework: DEI orchestrates a multi-agent system, employing a re-ranking pipeline to maximize the resolve rate of SWE issues by aggregating the best solutions from diverse agents.

SWE Agents and DEI Framework

SWE Agents and Their Diversity

SWE agents typically integrate LLMs with a suite of programmable tools for code navigation, editing, and testing. The paper classifies diversity into two types:

• Intra-agent diversity: Variability in output across multiple runs of the same agent due to the inherent non-determinism in LLMs.
• Inter-agent diversity: Differences in performance owing to variations in agent design, tools, workflows, and prompts.

Through quantitative analysis, the authors demonstrate that even agents with similar resolve rates tend to solve distinctly different sets of issues. This underlines the potential for performance improvement through intelligent aggregation of outputs.

DEI Framework

DEI aims to formalize the integration of diverse agent outputs within a Contextual Markov Decision Process (CMDP) framework. The meta-policy $\pi_{\text{DEI}}$ is designed to select the most suitable agent outputs based on the problem context, thus maximizing the cumulative reward — in this case, the effective resolution of software issues.

The paper describes a three-step DEI implementation:

1. Input Construction: Provision of issue descriptions, relevant code context, and pre- and post-patch code versions.
2. Explanation Generation: The LLM generates comprehensive explanations, including issue explanation, context explanation, location explanation, patch explanation, and conflict detection.
3. Patch Scoring: Based on the generated explanations, the LLM scores the candidate patches, facilitating the selection of the most promising solution.

Experimental Evaluation

The paper evaluates DEI across multiple SWE agents using SWE-Bench Lite, a standardized benchmark comprising 300 instances of real-world GitHub issues. The experiments assess both intra-agent and inter-agent diversity and the efficacy of DEI in enhancing issue resolution rates.

Main Findings

1. Agent Diversity: Significant inter-agent and intra-agent diversity was observed, with different agents and runs solving distinctly different sets of issues. The "Union@k" metric, which measures the upper limit of problem-solving potential if the best candidate is always selected, often showed doubled performance compared to the average case.
2. Impact of DEI: Across multiple experimental setups, including both diverse agent groups and multiple runs of single agents, DEI consistently improved the resolution rates over the best individual agent performance. For instance, a DEI-guided ensemble of open-source agents achieved a 34.3% resolve rate, outperforming the highest resolve rate of 27.3% by any single agent in the group.

Implications and Future Work

The results underscore the importance of diversity in multi-agent systems and the efficacy of the DEI framework in leveraging this diversity for improved performance. Practically, DEI can be adopted to enhance the resolution of software issues in various settings by combining the strengths of different agents.

Theoretically, this work adds to the body of research on collaborative AI systems, demonstrating an effective strategy for integrating diverse AI capabilities. Future developments could explore more sophisticated re-ranking mechanisms, better diversity metrics, and adaptive learning methods within the DEI framework to further optimize performance.

Conclusion

"Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents" convincingly establishes that exploiting the diversity among SWE agents can significantly enhance their problem-solving capacity. The DEI framework presents a scalable approach to combining the strengths of multiple agents, providing substantial improvements in resolving software engineering challenges. This paper paves the way for future research in multi-agent AI systems, emphasizing the value of diversity and collaboration.