ALMAS: an Autonomous LLM-based Multi-Agent Software Engineering Framework (2510.03463v2)

Abstract: Multi-agent LLM systems have been leading the way in applied LLM research across a number of fields. One notable area is software development, where researchers have advanced the automation of code implementation, code testing, code maintenance, inter alia, using LLM agents. However, software development is a multifaceted environment that extends beyond just code. As such, a successful LLM system must factor in multiple stages of the software development life-cycle (SDLC). In this paper, we propose a vision for ALMAS, an Autonomous LLM-based Multi-Agent Software Engineering framework, which follows the above SDLC philosophy such that it may work within an agile software development team to perform several tasks end-to-end. ALMAS aligns its agents with agile roles, and can be used in a modular fashion to seamlessly integrate with human developers and their development environment. We showcase the progress towards ALMAS through our published works and a use case demonstrating the framework, where ALMAS is able to seamlessly generate an application and add a new feature.

• The paper presents ALMAS, a multi-agent framework that leverages LLMs to automate the software development lifecycle with agile-aligned roles.
• It employs a modular, context-aware design featuring specialized agents for tasks including code generation, requirement clarification, and peer review.
• The framework integrates a novel Meta-RAG strategy to overcome LLM context limitations, boosting code accuracy and reducing resource consumption.

ALMAS: An Autonomous LLM-based Multi-Agent Software Engineering Framework

The paper "ALMAS: an Autonomous LLM-based Multi-Agent Software Engineering Framework" presents an innovative framework named ALMAS, designed to enhance the software development lifecycle (SDLC) through a multi-agent system powered by LLMs. This framework aims to integrate advanced AI functionalities across various software development stages, aligning with agile methodologies and addressing the complexity of industrial software engineering environments.

Architecture and Design

ALMAS is designed to function as an autonomous framework that incorporates a multi-agent architecture aligned with agile team roles. It includes various agents such as Sprint Agent, Supervisor Agent, Developer Agent, Summary Agent, and Control Agent. These agents simulate real-world team hierarchies, ranging from product management to sprint planning, and handle tasks including requirement clarification, task decomposition, code generation, and peer review.

A standout feature of ALMAS is its dual operational mode. It supports both autonomous execution and interactive collaboration with human developers, emphasizing a "three Cs" approach: Context-aware, Collaborative, and Cost-effective. This ensures seamless communication between ALMAS agents and human developers, enhancing productivity while reducing cognitive load and resource consumption.

Figure 1: Overall conceptual system architecture diagram for ALMAS framework.

Key Innovations

One of the key innovations of ALMAS is its modularity and context-awareness, crucial for handling the context window length limitations of LLMs. The Summary Agent creates structured natural language representations of the codebase, allowing agents to engage using a programming-language-independent modality. This approach not only minimizes token usage but also enhances the performance of LLMs by reducing attention dilution.

The framework incorporates a novel Retrieval Augmented Generation (RAG) strategy named Meta-RAG. This strategy mitigates the context window limitations of LLMs by effectively guiding code generation processes with compact code representations. The retrieval strategy ensures that relevant code snippets are provided as context, enhancing the accuracy and efficiency of code modifications.

Application and Performance

ALMAS demonstrates its efficacy through a use case involving the generation of a Python Streamlit application. This application showcases the framework's capability to handle an end-to-end task, from initial planning and code generation to code augmentation with new features. The collaborative model allows different LLMs to be used across agents, enabling specialization and cost optimization.

Figure 2: Example of application generated with ALMAS from a single task.

The evaluation of ALMAS in this scenario highlights its ability to seamlessly integrate with popular development tools such as Atlassian Jira and Bitbucket, facilitating task management and version control. The framework's modular design allows for flexible integration with existing developer workflows, promoting a harmonious collaboration between AI agents and human developers.

Implications and Future Work

The introduction of ALMAS marks a significant advancement toward a fully integrated ecosystem for AI-assisted software engineering. By covering multiple phases of the SDLC and ensuring context-aware, modular automation, ALMAS sets the stage for more efficient and effective software development processes. Its design addresses key challenges in the use of LLMs, such as context length restrictions and attention mechanism limitations.

Looking forward, the research team plans to conduct comprehensive end-to-end evaluations on a broader range of coding tasks, utilizing benchmarking datasets like SWE-Bench. These evaluations will aim to further validate the framework's capabilities in resolving complex software engineering challenges such as bug localization and code augmentation.

Conclusion

ALMAS presents a robust multi-agent framework for software engineering, leveraging LLMs to automate and optimize various stages of the software development lifecycle. Its design aligns with agile methodologies, promoting efficient resource allocation and seamless human-AI interaction. The framework's modularity and context-aware strategies highlight significant progress in integrating AI into complex industrial software environments, with promising implications for future developments in AI and software engineering.