From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future

Abstract: With the rise of LLMs, researchers are increasingly exploring their applications in var ious vertical domains, such as software engineering. LLMs have achieved remarkable success in areas including code generation and vulnerability detection. However, they also exhibit numerous limitations and shortcomings. LLM-based agents, a novel tech nology with the potential for AGI, combine LLMs as the core for decision-making and action-taking, addressing some of the inherent limitations of LLMs such as lack of autonomy and self-improvement. Despite numerous studies and surveys exploring the possibility of using LLMs in software engineering, it lacks a clear distinction between LLMs and LLM based agents. It is still in its early stage for a unified standard and benchmarking to qualify an LLM solution as an LLM-based agent in its domain. In this survey, we broadly investigate the current practice and solutions for LLMs and LLM-based agents for software engineering. In particular we summarise six key topics: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. We review and differentiate the work of LLMs and LLM-based agents from these six topics, examining their differences and similarities in tasks, benchmarks, and evaluation metrics. Finally, we discuss the models and benchmarks used, providing a comprehensive analysis of their applications and effectiveness in software engineering. We anticipate this work will shed some lights on pushing the boundaries of LLM-based agents in software engineering for future research.

• The paper surveys the evolution from standalone LLMs to autonomous agents that enhance tasks like code generation, debugging, and requirement engineering.
• LLM-based agents leverage frameworks like Retrieval-Augmented Generation to integrate decision-making and tool utilization, fostering efficient software development.
• The study identifies challenges such as limited context comprehension and high computational demands, urging the need for standardized benchmarks and enhanced agent autonomy.

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future

Introduction

The paper explores the transition from LLMs to LLM-based agents in the domain of Software Engineering (SE), addressing key areas such as requirement engineering, code generation, autonomous learning, software design, test generation, and maintenance. The survey highlights the potential of LLMs and LLM-based agents in automating and enhancing SE tasks, while also acknowledging the inherent limitations and challenges.

LLMs in Software Engineering

LLMs, such as GPT and Codex, have shown promising results in SE tasks like code generation and debugging. Despite their efficacy, they face challenges such as limited context comprehension, inability to use external tools, and issues with hallucinations. These limitations impede their application in complex, dynamic SE environments, necessitating critical refinement and validation by experts.

LLM-based Agents

Emerging LLM-based agents aim to overcome LLM limitations by integrating decision-making and tool utilization capabilities, akin to AGI. These agents, using frameworks like Retrieval-Augmented Generation (RAG), demonstrate autonomy in tasks like autonomous debugging and code refactoring, enhancing efficiency and adaptability in SE processes.

Key Findings and Applications

1. Requirement Engineering and Documentation:
   • LLMs automate requirement generation and classification, improving efficiency but still require validation for accuracy.
   • LLM-based agents enhance autonomy in generating and refining user stories and safety requirements, using multi-agent systems for iterative improvement.
2. Code Generation and Software Development:
   • LLMs aid in code synthesis and refactoring, with models like CodeT5+ optimizing these tasks.
   • LLM-based agents excel in collaborative code generation and testing, distributing tasks among specialized agents for enhanced output quality.
3. Autonomous Learning and Decision Making:
   • LLMs exhibit potential in automated debugging and creative problem-solving, though limited by fixed task scopes.
   • LLM-based agents leverage multi-agent collaboration for dynamic decision-making in complex scenarios, showing superior performance in multitasking environments.

Challenges and Future Directions

The transition to LLM-based agents offers broader capabilities and improved efficiency in SE, but challenges remain in standardizing evaluation benchmarks and addressing computational demands. Future research should focus on refining agent autonomy, incorporating diverse datasets, and enhancing tool integration to further bridge the gap between current agent capabilities and AGI.

Conclusion

The survey underscores the transformative impact of LLMs and LLM-based agents on SE, highlighting their potential to automate and optimize various tasks. By addressing current limitations and exploring innovative applications, LLM-based agents can significantly advance the field of Software Engineering toward more intelligent and autonomous systems.