Agents in Software Engineering: Survey, Landscape, and Vision

Abstract: In recent years, LLMs have achieved remarkable success and have been widely used in various downstream tasks, especially in the tasks of the software engineering (SE) field. We find that many studies combining LLMs with SE have employed the concept of agents either explicitly or implicitly. However, there is a lack of an in-depth survey to sort out the development context of existing works, analyze how existing works combine the LLM-based agent technologies to optimize various tasks, and clarify the framework of LLM-based agents in SE. In this paper, we conduct the first survey of the studies on combining LLM-based agents with SE and present a framework of LLM-based agents in SE which includes three key modules: perception, memory, and action. We also summarize the current challenges in combining the two fields and propose future opportunities in response to existing challenges. We maintain a GitHub repository of the related papers at: https://github.com/DeepSoftwareAnalytics/Awesome-Agent4SE.

• The paper presents a comprehensive survey of LLM-based agents in software engineering, organizing them into perception, memory, and action modules.
• It employs systematic categorization and analysis techniques to highlight challenges such as perception diversification and hallucination mitigation.
• The study proposes actionable future research directions to enhance multi-agent collaboration and integration with software engineering methodologies.

Agents in Software Engineering: Survey, Landscape, and Vision

Introduction

The integration of LLMs within the field of software engineering (SE) represents a burgeoning area of research, characterized by substantial contributions, yet also marked by challenges and opportunities for further exploration. Recent efforts have illuminated various applications of LLM-based agents across SE, leveraging the intelligence and adaptability of agents for diverse tasks such as code summarization, generation, and translation. This paper conducts a rigorous survey of the literature, systematically categorizing the functionality of LLM-based agents into three principal modules: perception, memory, and action. Additionally, the paper identifies key challenges and proposes a roadmap for future advancements.

LLM-based Agents Framework

Perception Module

The perception module acts as the sensory interface, enabling LLM-based agents to interpret various forms of input. SE applications necessitate the adaptation of input processing to accommodate both natural language and code, which can be token-based, tree/graph-based, or hybrid-formatted. Existing research predominantly focuses on token-based inputs, demanding more profound exploration into tree/graph-based and other multimodal perceptions, including visual and auditory types, which remain underexplored in current research literature.

Figure 1: An overview of agent framework in SE.

Memory Module

Memory modules in LLM-based agents comprise semantic, episodic, and procedural memory types, facilitating complex reasoning and decision-making processes:

• Semantic Memory: Encompasses an external knowledge base that enriches the agent's cognitive repository with essential world knowledge, often drawn from documents, libraries, and APIs. Its continuous expansion through carefully curated knowledge bases remains pivotal for enhancing agent performance.
• Episodic Memory: Extends the contextual awareness of agents through historical data and decision timelines, aiding in dynamic adaptation and contextual reasoning.
• Procedural Memory: Distinguishes between explicit knowledge hard-coded into the agent and implicit knowledge embedded within LLM parameters. Techniques for fine-tuning LLM weights to refine implicit knowledge present ongoing computational challenges.

Action Module

The action module's dual-faceted internal and external pathways depict agent interaction and adaptation. Internal actions like reasoning and retrieval are refined by CoT methods and advanced retrieval techniques, while external actions involve interactions with humans or digital environments to provide feedback and enhance agent learning.

Figure 2: Different CoTs from different methods, illustrating various strategies for improving reasoning via structured formats.

Challenges and Opportunities

The paper highlights several challenges impeding the advancement of LLM-based agents in SE:

• Perception Diversification: Current research has not sufficiently explored diverse perception modalities. A more comprehensive understanding of code's intrinsic characteristics, beyond natural language processing, presents a rich avenue for innovation.
• Role Versatility: SE agents require heightened adaptability to perform multi-role tasks efficiently, especially in complex SE scenarios demanding diverse capabilities.
• Knowledge Base Development: The establishment of a recognized, authoritative knowledge base encompassing diverse programming knowledge is essential for advancing semantic memory capabilities.
• Hallucination Mitigation: Addressing inaccurate outputs from LLMs, known as hallucinations, remains critical for improving agent reliability and is an area ripe for future investigation.

  Figure 3: The pipeline of different retrieval methods, comparing dense-based and sparse-based retrieval strategies.

• Efficiency in Multi-agent Collaboration: Developing strategies to optimize resource use in multi-agent systems can enhance the computational efficiency and efficacy of collaborative SE tasks.
• Integration with SE Techniques: Bridging advances in SE techniques with LLM-based agent functionalities promises reciprocal enhancements in both domains, with significant research potential in leveraging software tools and methodologies for agent optimization.

Conclusion

This survey meticulously outlines the framework and operational dynamics of LLM-based agents in software engineering, identifying key structural components and inherent challenges. By proposing a comprehensive model and suggesting avenues for future research, the paper lays the groundwork for the ongoing evolution of intelligent agents in SE, aimed at addressing computational challenges and further enhancing the landscape of software development and engineering.