Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

When LLM-based Code Generation Meets the Software Development Process (2403.15852v1)

Published 23 Mar 2024 in cs.SE and cs.AI
When LLM-based Code Generation Meets the Software Development Process

Abstract: Software process models play a pivotal role in fostering collaboration and communication within software teams, enabling them to tackle intricate development tasks effectively. This paper introduces LCG, a code generation framework inspired by established software engineering practices. LCG leverages multiple LLM agents to emulate various software process models, namely LCGWaterfall, LCGTDD, and LCGScrum. Each model assigns LLM agents specific roles such as requirement engineer, architect, developer, tester, and scrum master, mirroring typical development activities and communication patterns. Through collaborative efforts utilizing chain-of-thought and prompt composition techniques, the agents continuously refine themselves to enhance code quality. Utilizing GPT3.5 as the underlying LLM and baseline (GPT), we evaluate LCG across four code generation benchmarks: HumanEval, HumanEval-ET, MBPP, and MBPP-ET. Results indicate LCGScrum outperforms other models, achieving Pass@1 scores of 75.2, 65.5, 82.5, and 56.7 in HumanEval, HumanEval-ET, MBPP, and MBPP-ET, respectively - an average 15% improvement over GPT. Analysis reveals distinct impacts of development activities on generated code, with design and code reviews contributing to enhanced exception handling, while design, testing, and code reviews mitigate code smells. Furthermore, temperature values exhibit negligible influence on Pass@1 across all models. However, variations in Pass@1 are notable for different GPT3.5 model versions, ranging from 5 to over 60 in HumanEval, highlighting the stability of LCG across model versions. This stability underscores the importance of adopting software process models to bolster the quality and consistency of LLM-generated code.

PDF HTML Abstract

Analyzing the Impact of Software Process Models on LLM-Based Code Generation

The integration of LLMs in code generation tasks represents a progression in automating software development activities. The paper "When LLM-based Code Generation Meets the Software Development Process" proposes a diversified approach in harnessing LLMs, specifically through the development of the LCG framework. This framework utilizes multiphasic agent-driven models to simulate traditional software development processes, namely Waterfall, Test-Driven Development (TDD), and Scrum. Each of these models enlists LLM agents in roles akin to real-world software engineering professions: requirement engineers, architects, developers, testers, and scrum masters, facilitating a simulated collaborative environment to improve code generation outputs.

Framework and Methodology Overview

The LCG framework extends beyond conventional prompting methods, assigning LLM agents distinct roles and tasks that accord with chosen development methodologies. In their paper, Lin et al. implemented a systematic role-implementation architecture, ensuring agents distinctly operate within their domains. Moreover, the framework employs advanced techniques such as chain-of-thought reasoning, prompt composition, and self-refinement to adaptively enhance code outputs iteratively. Notably, the work emphasizes zero-shot learning to eschew biases inherent in few-shot sample selections.

Evaluation benchmarks such as HumanEval and MBPP, including stronger evaluation tests (HumanEval-ET and MBPP-ET), were utilized to detect the robustness of these LLM configurations. Results demonstrated substantial improvements in Pass@1 scores over a baseline (GPT-3.5), with metrics showing up to 31.5% improvement in some tests. This underlines the efficacy of implementing structured software development paradigms within LLM code generation frameworks.

Numerical and Empirical Findings

Among the three process models tested, Scrum consistently outperformed others concerning not only Pass@1 scores but also favorable code smell and exception-handling metrics. The ability of these models to maintain stability amid fluctuating model versions establishes the pragmatic advantages of employing process models in LLM-driven code generation. Comparatively, the traditional usage of GPT without structured processes exhibits significant variability, particularly when different versions of the model are assessed.

Design and code review activities uniquely facilitated a decrease in code smells and incremented exception handling, signifying enhancements in the reliability of the generated code. Test execution, unsurprisingly, emerged as the most influential activity for improving code correctness, as its removal resulted in marked drops in Pass@1 scores. These insights suggest an intertwined effect of process-structure emulation and systematic testing, contributing synergistically to elevate code quality.

Implications and Future Directions

The implications of these findings extend toward new methodologies in automated code development, where the agent-based adaptation of traditional process models aligns with agile and iterative software practices. The theoretical underpinning implies a necessity to expand upon multi-agent collaborations within AI frameworks, potentially integrating similar methodologies in broader development tasks and life cycles.

In future studies, extending these frameworks to support diverse programming languages and task complexities could help streamline LLM applicability across varying software domains. Additionally, investigating the participatory role of LLMs in more intricate development settings, such as end-to-end system design and integration, holds merit. Thus, an evolved exploration into agents-as-collaborators can redefine existing paradigms in AI software engineering research.

Considering the paper's contribution to code generation techniques, it is evident that the authors provide a strong case for using process models to support stable and quality-driven implementations of LLMs in software development tasks. This research offers a promising foundation for continuous advancements in the domain of AI-driven software engineering.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations. arXiv preprint arXiv:2402.06013, 2024.
  2. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  3. Youssef Bassil. A simulation model for the waterfall software development life cycle. arXiv preprint arXiv:1205.6904, 2012.
  4. Benchmarking large language models in retrieval-augmented generation. arXiv preprint arXiv:2309.01431, 2023a.
  5. How is chatgpt’s behavior changing over time? ArXiv, abs/2307.09009, 2023b.
  6. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  7. Code quality: Examining the efficacy of automated tools. 2017.
  8. Codescore: Evaluating code generation by learning code execution. arXiv preprint arXiv:2301.09043, 2023a.
  9. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.07590, 2023b.
  10. Automatic detection and masking of nonatomic exception handling. IEEE Transactions on Software Engineering, 30(8):547–560, 2004.
  11. Martin Fowler. The new methodology, 2005. URL https://www.martinfowler.com/articles/newMethodology.html.
  12. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023.
  13. Codecot and beyond: Learning to program and test like a developer. arXiv preprint arXiv:2308.08784, 2023a.
  14. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023b.
  15. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2312–2323. IEEE, 2023.
  16. Reordering examples helps during priming-based few-shot learning. arXiv preprint arXiv:2106.01751, 2021.
  17. Log parsing with prompt-based few-shot learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2438–2449. IEEE, 2023.
  18. Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2305.06599, 2023.
  19. Is your code generated by chatgpt really correct. Rigorous evaluation of large language models for code generation. CoRR, abs/2305.01210, 2023a.
  20. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9), jan 2023b.
  21. Knowlog: Knowledge enhanced pre-trained language model for log understanding. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–13, 2024a.
  22. Llmparser: An exploratory study on using large language models for log parsing. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 883–883. IEEE Computer Society, 2024b.
  23. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
  24. Assessing test-driven development at ibm. In 25th International Conference on Software Engineering, 2003. Proceedings., pages 564–569. IEEE, 2003.
  25. Retrieval-based prompt selection for code-related few-shot learning. In IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2450–2462, 2023.
  26. OpenAI. Chatgpt. https://chat.openai.com/, 2023.
  27. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023.
  28. Python. Built-in exceptions, 2005. URL https://docs.python.org/3/library/exceptions.html.
  29. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023.
  30. A novel approach for automatic program repair using round-trip translation with large language models. arXiv preprint arXiv:2401.07994, 2024.
  31. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 2023.
  32. Character-llm: A trainable agent for role-playing. arXiv preprint arXiv:2310.10158, 2023.
  33. Git merge conflict resolution leveraging strategy classification and llm. In 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), pages 228–239. IEEE, 2023.
  34. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
  35. Pylint Development Team. Pylint. https://pypi.org/project/pylint/. Last accessed March 2024.
  36. Testsigma. What is devtestops? https://testsigma.com/devtestops. Last accessed March 2024.
  37. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  38. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  39. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2303.07839, 2023.
  40. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023.
  41. Chatunitest: a chatgpt-based automated unit test generation tool. arXiv preprint arXiv:2305.04764, 2023.
  42. Alleviating the sample selection bias in few-shot learning by removing projection to the centroid. Advances in Neural Information Processing Systems, 35:21073–21086, 2022.
  43. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658, 2023.
  44. Do code smells reflect important maintainability aspects? In 2012 28th IEEE International Conference on software maintenance (ICSM), pages 306–315, 2012.
  45. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023.
  46. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint arXiv:2304.10778, 2023.
  47. Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240, 2023a.
  48. No more manual tests? evaluating and improving ChatGPT for unit test generation. arXiv preprint arXiv:2305.04207, 2023b.
  49. Automatic commit message generation: A critical review and directions for future work. IEEE Transactions on Software Engineering, 2024.
  50. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906, 2024.
  51. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Feng Lin (89 papers)
  2. Dong Jae Kim (8 papers)
  3. Tse-Husn (2 papers)
  4. Chen (63 papers)
Citations (10)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube