HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale (2409.16299v2)

Abstract: LLMs have revolutionized software engineering (SE), showcasing remarkable proficiency in various coding tasks. Despite recent advancements that have enabled the creation of autonomous software agents utilizing LLMs for end-to-end development tasks, these systems are typically designed for specific SE functions. We introduce HyperAgent, an innovative generalist multi-agent system designed to tackle a wide range of SE tasks across different programming languages by mimicking the workflows of human developers. HyperAgent features four specialized agents-Planner, Navigator, Code Editor, and Executor-capable of handling the entire lifecycle of SE tasks, from initial planning to final verification. HyperAgent sets new benchmarks in diverse SE tasks, including GitHub issue resolution on the renowned SWE-Bench benchmark, outperforming robust baselines. Furthermore, HyperAgent demonstrates exceptional performance in repository-level code generation (RepoExec) and fault localization and program repair (Defects4J), often surpassing state-of-the-art baselines.

Overview of HyperAgent: Generalist Software Engineering Agents

The research article introduces HyperAgent, a sophisticated generalist multi-agent system designed to address a broad spectrum of software engineering (SE) tasks. Unlike existing specialized autonomous software agents focusing on specific tasks, HyperAgent aims to handle various programming languages and SE challenges by emulating human developer workflows. The system's architecture incorporates four specialized agents—Planner, Navigator, Code Editor, and Executor—each responsible for distinct phases of the software development lifecycle, from task conception to final verification.

System Architecture and Workflow

The HyperAgent architecture is rooted in typical workflows followed by software engineers, capturing the iterative phases of analysis & planning, feature localization, edition, and execution. Each phase corresponds to one of the four agents:

• Planner: This agent serves as the central unit, strategizing solutions, coordinating subtasks, and managing the flow of information.
• Navigator: Specialized in extracting relevant information from codebases, this agent uses IDE-similar tools for efficient context localization.
• Code Editor: Tasked with code modification and generation, this agent applies patches based on contextual inputs from the Planner.
• Executor: It verifies solutions by setting up test environments and executing necessary tests.

The system leverages an asynchronous communication model using a distributed Message Queue system for managing task delegations and processing results, which ensures parallelism and scalability in handling complex software challenges.

Evaluation and Results

HyperAgent was extensively evaluated across several diverse SE benchmarks, showing notable prowess in handling tasks such as GitHub issue resolution, repository-scale code generation, fault localization, and program repair. Key results include:

• GitHub Issues Resolution: Achieved high success rates on SWE-Bench datasets, with the HyperAgent-Full-2 configuration achieving a 31.40% success rate on SWE-Bench Verified tasks, outperforming various existing baselines.
• RepoExec Benchmark for Code Generation: HyperAgent-Lite-3 showed superior performance in Pass@5 metrics, indicating effective repository-scale code generation through auto-context retrieval capabilities.
• Fault Localization and Program Repair: The system demonstrated substantial improvements over traditional techniques, achieving a 59.70% accuracy in fault localization and 192 correct program repairs on the Defects4J dataset.

Implications and Future Directions

The introduction of HyperAgent marks a significant enhancement towards versatile, multi-tasking AI agents in software development. Its generalist design minimizes the configuration overhead seen in task-specific systems, promoting ease of adaptation to new challenges across various programming languages.

Future research could potentially explore integrating HyperAgent with existing development platforms and exploring specialized applications like security auditing and performance optimization. To sojourn ongoing advancements in software engineering, enhancing system explainability and refining its decision-making transparency may foster broader acceptance and assist developers in trust-building. Continuous upgrades to incorporate emerging programming paradigms and maintaining a dynamic knowledge repository could solidify HyperAgent’s applicability in ever-evolving software development environments.

Conclusion

HyperAgent represents a pivotal step forward in AI-assisted software engineering, proving its efficacy and adaptability through rigorous evaluation frameworks typical of real-world scenarios. Its multi-agent framework not only simplifies handling of intricate tasks but also paves the way for future AI solutions to integrate smoothly into all stages of the software lifecycle, potentially reshaping development practices. The research provides valuable insights for the evolution of autonomous systems capable of wider generalization while delivering state-of-the-art practical outcomes.