Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Transformative Influence of Large Language Models on Software Development (2311.16429v1)

Published 28 Nov 2023 in cs.SE and cs.HC

Abstract: The increasing adoption and commercialization of generalized LLMs have profoundly impacted various aspects of our daily lives. Initially embraced by the computer science community, the versatility of LLMs has found its way into diverse domains. In particular, the software engineering realm has witnessed the most transformative changes. With LLMs increasingly serving as AI Pair Programming Assistants spurred the development of specialized models aimed at aiding software engineers. Although this new paradigm offers numerous advantages, it also presents critical challenges and open problems. To identify the potential and prevailing obstacles, we systematically reviewed contemporary scholarly publications, emphasizing the perspectives of software developers and usability concerns. Preliminary findings underscore pressing concerns about data privacy, bias, and misinformation. Additionally, we identified several usability challenges, including prompt engineering, increased cognitive demands, and mistrust. Finally, we introduce 12 open problems that we have identified through our survey, covering these various domains.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. B. W. Sorte, P. P. Joshi, and V. Jagtap, “Use of Artificial Intelligence in Software Development Life Cycle—A State of the Art Review,” International Journal of Advanced Engineering and Global Technology, 2015.
  2. “Stack Overflow.” \urlhttps://stackoverflow.com/, 2023.
  3. J. Liu, X. Tang, L. Li, P. Chen, and Y. Liu, “Which is a better programming assistant? A comparative study between ChatGPT and Stack Overflow,” arXiv preprint arXiv:2308.13851, 2023.
  4. A. Al Jobair, S. Mohammad, Z. R. Maisha, M. J. I. Mostafa, and M. N. Haque, “An Empirical Study on Neophytes of Stack Overflow: How Welcoming the Community is towards Them.,” in ENASE, 2022.
  5. F. Calefato, F. Lanubile, M. C. Marasciulo, and N. Novielli, “Mining Successful Answers in Stack Overflow,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, 2015.
  6. G. C. Murphy, M. Kersten, and L. Findlater, “How are Java software developers using the Eclipse IDE?,” IEEE software, 2006.
  7. T. Leich, S. Apel, L. Marnitz, and G. Saake, “Tool Support for Feature-Oriented Software Development: FeatureIDE: An Eclipse-Based Approach,” in Proceedings of the 2005 OOPSLA workshop on Eclipse technology eXchange, 2005.
  8. M. Beller, N. Spruit, D. Spinellis, and A. Zaidman, “On the Dichotomy of Debugging Behavior Among Programmers,” in Proceedings of the 40th International Conference on Software Engineering, 2018.
  9. J. Cito, P. Leitner, C. Bosshard, M. Knecht, G. Mazlami, and H. C. Gall, “Performancehat: Augmenting Source Code With Runtime Performance Traces in the IDE,” in Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, 2018.
  10. C. Mamede, E. Pinconschi, and R. Abreu, “A Transformer-Based Ide Plugin for Vulnerability Detection,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022.
  11. U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “Code2vec: Learning Distributed Representations of Code,” Proceedings of the ACM on Programming Languages, 2019.
  12. “GitHub.” \urlhttps://github.com/, 2023.
  13. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155, 2020.
  14. U. Alon, S. Brody, O. Levy, and E. Yahav, “code2seq: Generating sequences from structured representations of code,” arXiv preprint arXiv:1808.01400, 2018.
  15. “ChatGPT.” \urlhttps://chat.openai.com, 2023.
  16. “Stack Overflow Developer Survey.” \urlhttps://survey.stackoverflow.co/2023/, 2023.
  17. “GitHub Copilot.” \urlhttps://github.com/features/copilot, 2023.
  18. “Ghostwriter.” \urlhttps://replit.com/ai, 2023.
  19. “Code Llama.” \urlhttps://github.com/facebookresearch/codellama, 2023.
  20. “Tabnine.” \urlhttps://www.tabnine.com, 2023.
  21. A. Ahmad, M. Waseem, P. Liang, M. Fahmideh, M. S. Aktar, and T. Mikkonen, “Towards Human-Bot Collaborative Software Architecting With ChatGPT,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, 2023.
  22. “OpenAI Human Eval Dataset.” \urlhttps://huggingface.co/datasets/openai_humaneval, 2021.
  23. A. Sarkar, A. D. Gordon, C. Negreanu, C. Poelitz, S. S. Ragavan, and B. Zorn, “What Is It Like to Program With Artificial Intelligence?,” arXiv preprint arXiv:2208.06213, 2022.
  24. N. Nguyen and S. Nadi, “An Empirical Evaluation of Github Copilot’s Code Suggestions,” in Proceedings of the 19th International Conference on Mining Software Repositories, 2022.
  25. S. Imai, “Is Github Copilot a Substitute for Human Pair-Programming? An Empirical Study,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022.
  26. B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the Quality of Github Copilot’s Code Generation,” in Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, 2022.
  27. C. Ebert and P. Louridas, “Generative AI for Software Practitioners,” IEEE Software, 2023.
  28. N. G. Canbek and M. E. Mutlu, “On the Track of Artificial Intelligence: Learning With Intelligent Personal Assistants,” Journal of Human Sciences, 2016.
  29. A. Y. Wang, D. Wang, J. Drozdal, M. Muller, S. Park, J. D. Weisz, X. Liu, L. Wu, and C. Dugan, “Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks,” ACM Transactions on Computer-Human Interaction, 2022.
  30. A. M. McNutt, C. Wang, R. A. Deline, and S. M. Drucker, “On the Design of AI-Powered Code Assistants for Notebooks,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023.
  31. J. T. Liang, C. Yang, and B. A. Myers, “Understanding the Usability of AI Programming Assistants,” International Conference on Software Engineering, 2024.
  32. Q. Zheng, Y. Tang, Y. Liu, W. Liu, and Y. Huang, “UX Research on Conversational Human-Ai Interaction: A Literature Review of the ACM Digital Library,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022.
  33. P. Vaithilingam, E. L. Glassman, P. Groenwegen, S. Gulwani, A. Z. Henley, R. Malpani, D. Pugh, A. Radhakrishna, G. Soares, J. Wang, et al., “Towards More Effective AI-Assisted Programming: A Systematic Design Exploration To Improve Visual Studio Intelli-Code’s User Experience,” in Proceedings of the IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP’23). Association for Computing Machinery, New York, NY, USA, 2023.
  34. “Stack Overflow Excel Tagged Questions.” \urlhttps://stackoverflow.com/questions/tagged/excel-formula, 2023.
  35. M. X. Liu, A. Sarkar, C. Negreanu, B. Zorn, J. Williams, N. Toronto, and A. D. Gordon, ““What It Wants Me To Say”: Bridging The Abstraction Gap Between End-user Programmers And Code-generating Large Language Models,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023.
  36. C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Lowdermilk, and I. Gazit, “Taking Flight with Copilot,” Communications of the ACM, 2023.
  37. Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Peng, “No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation,” arXiv preprint arXiv:2305.04207, 2023.
  38. N. Jain, S. Vaidyanath, A. Iyer, N. Natarajan, S. Parthasarathy, S. Rajamani, and R. Sharma, “Jigsaw: Large Language Models Meet Program Synthesis,” in Proceedings of the 44th International Conference on Software Engineering, 2022.
  39. R. Wang, R. Cheng, D. Ford, and T. Zimmermann, “Investigating And Designing For Trust In AI-Powered Code Generation Tools,” arXiv preprint arXiv:2305.11248, 2023.
  40. A. Tikayat Ray, O. J. Pinon-Fischer, D. N. Mavris, R. T. White, and B. F. Cole, “Aerobert-ner: Named-entity Recognition For Aerospace Requirements Engineering Using Bert,” in AIAA SCITECH, 2023.
  41. A. Tikayat Ray, B. F. Cole, O. J. Pinon Fischer, A. P. Bhat, R. T. White, and D. N. Mavris, “Agile Methodology for the Standardization of Engineering Requirements Using Large Language Models,” Systems, 2023.
  42. S. Barke, M. B. James, and N. Polikarpova, “Grounded Copilot: How Programmers Interact With Code-generating Models,” Proceedings of the ACM on Programming Languages, 2023.
  43. “Real-Real-World Programming with ChatGPT.” \urlhttps://www.oreilly.com/radar/real-real-world-programming-with-chatgpt/, 2023.
  44. “How to build an enterprise LLM application: Lessons from GitHub Copilot.” \urlhttps://github.blog/, 2023.
  45. S. Jalil, S. Rafi, T. D. LaToza, K. Moran, and W. Lam, “ChatGPT and Software Testing Education: Promises & Perils,” in 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2023.
  46. “Copilot X.” \urlhttps://github.com/features/preview/copilot-x, 2023.
  47. “Tabnine Chat.” \urlhttps://docs.tabnine.com/enterprise/private-installation/end-user-guide/start-using-tabnine-enterprise/tabnine-chat-beta, 2023.
  48. “Ghostwriter Chat.” \urlhttps://www.semafor.com/article/02/15/2023/startup-replit-launches-a-chatgpt-like-bot-for-coders, 2023.
  49. “Visual Studio IntelliCode.” \urlhttps://visualstudio.microsoft.com/services/intellicode/, 2023.
  50. “DALL·E 3.” \urlhttps://openai.com/dall-e-3, 2023.
  51. “Galilleo AI.” \urlhttps://www.usegalileo.ai/, 2023.
  52. “Amazon CodeWhisperer.” \urlhttps://aws.amazon.com/codewhisperer/, 2023.
  53. M. A. Serrano, J. A. Cruz-Lemus, R. Perez-Castillo, and M. Piattini, “Quantum software components and platforms: Overview and quality assessment,” ACM Computing Surveys, 2022.
  54. “Khan Academy.” \urlhttps://www.khanacademy.org//, 2023.
  55. “Khanmigo.” \urlhttps://blog.khanacademy.org/learner-khanmigo/, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now