Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations (2402.06013v1)

Published 8 Feb 2024 in cs.SE

Abstract: LLMs, like ChatGPT, have gained widespread popularity and usage in various software engineering tasks, including refactoring, testing, code review, and program comprehension. Despite recent studies delving into refactoring documentation in commit messages, issues, and code review, little is known about how developers articulate their refactoring needs when interacting with ChatGPT. In this paper, our goal is to explore conversations between developers and ChatGPT related to refactoring to better understand how developers identify areas for improvement in code and how ChatGPT addresses developers' needs. Our approach relies on text mining refactoring-related conversations from 17,913 ChatGPT prompts and responses, and investigating developers' explicit refactoring intention. Our results reveal that (1) developer-ChatGPT conversations commonly involve generic and specific terms/phrases; (2) developers often make generic refactoring requests, while ChatGPT typically includes the refactoring intention; and (3) various learning settings when prompting ChatGPT in the context of refactoring. We envision that our findings contribute to a broader understanding of the collaboration between developers and AI models, in the context of code refactoring, with implications for model improvement, tool development, and best practices in software engineering.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. [n. d.]. https://github.com/UIUC-Chatbot/ai-ta-backend/issues/57.
  2. [n. d.]. https://futurism.com/the-byte/github-92-percent-programmers-using-ai.
  3. Towards human-bot collaborative software architecting with chatgpt. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. 279–285.
  4. Ali Al-Kaswan and Maliheh Izadi. 2023. The (ab) use of Open Source Code to Train Large Language Models. arXiv preprint arXiv:2302.13681 (2023).
  5. GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBench. arXiv preprint arXiv:2308.13963 (2023).
  6. Refactoring Practices in the Context of Modern Code Review: An Industrial Case Study at Xerox. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 348–357.
  7. Can refactoring be self-affirmed? an exploratory study on how developers document their refactoring activities in commit messages. In International Workshop on Refactoring-accepted. IEEE.
  8. Toward the automatic classification of self-affirmed refactoring. Journal of Systems and Software 171 (2021), 110821.
  9. How we refactor and how we document it? On the use of supervised machine learning algorithms to classify refactoring documentation. Expert Systems with Applications 167 (2021), 114176.
  10. A Lot of Talk and a Badge: An Empirical Analysis of Personal Achievements in GitHub. arXiv preprint arXiv:2303.14702 (2023).
  11. Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. IEEE, 275–284.
  12. Assessing the Readability of ChatGPT Code Snippet Recommendations: A Comparative Study. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. 283–292.
  13. A preliminary investigation of self-admitted refactorings in open source software (S). In International Conferences on Software Engineering and Knowledge Engineering, Vol. 2018. KSI Research Inc. and Knowledge Systems Institute Graduate School, 165–168.
  14. Self-collaboration Code Generation via ChatGPT. arXiv preprint arXiv:2304.07590 (2023).
  15. Large Language Models for Software Engineering: Survey and Open Problems. arXiv preprint arXiv:2310.03533 (2023).
  16. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence 28, 4 (2006), 594–611.
  17. Investigating Code Generation Performance of Chat-GPT with Crowdsourcing Social Data. In Proceedings of the 47th IEEE Computer Software and Applications Conference. 1–10.
  18. Refactoring: Improving the Design of Existing Code. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. http://dl.acm.org/citation.cfm?id=311424
  19. Md Asraful Haque and Shuai Li. 2023. The Potential Use of ChatGPT for Debugging and Bug Fixing. EAI Endorsed Transactions on AI and Robotics 2, 1 (2023), e4–e4.
  20. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620 (2023).
  21. Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745 (2023).
  22. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences 103 (2023), 102274.
  23. Is Model Attention Aligned with Human Attention? An Empirical Study on Large Language Models for Code Generation. arXiv preprint arXiv:2306.01220 (2023).
  24. The Scope of ChatGPT in Software Engineering: A Thorough Investigation. arXiv preprint arXiv:2305.12138 (2023).
  25. Gathering refactoring data: a comparison of four methods. In Proceedings of the 2nd Workshop on Refactoring Tools. 1–5.
  26. Artificial Intelligence vs. Software Engineers: An Empirical Study on Performance and Efficiency using ChatGPT. In Proceedings of the 33rd Annual International Conference on Computer Science and Software Engineering. 24–33.
  27. Evaluating and Explaining Large Language Models for Code Using Syntactic Structures. arXiv preprint arXiv:2308.03873 (2023).
  28. Rohith Pudari and Neil A Ernst. 2023. From Copilot to Pilot: Towards AI Supported Software Development. arXiv preprint arXiv:2303.04142 (2023).
  29. On the Relation of Refactorings and Software Defect Prediction. In Proceedings of the 2008 International Working Conference on Mining Software Repositories (Leipzig, Germany) (MSR ’08). ACM, New York, NY, USA, 35–38. https://doi.org/10.1145/1370750.1370759
  30. Unveiling the potential of large language models in generating semantic and cross-language clones. arXiv preprint arXiv:2309.06424 (2023).
  31. Daniel Russo. 2023. Navigating the complexity of generative ai adoption in software engineering. arXiv preprint arXiv:2307.06081 (2023).
  32. Why We Refactor? Confessions of GitHub Contributors. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). ACM, New York, NY, USA, 858–870. https://doi.org/10.1145/2950290.2950305
  33. An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint arXiv:2301.08653 (2023).
  34. Christoph Treude and Hideaki Hata. 2023. She Elicits Requirements and He Tests: Software Engineering Gender Bias in Large Language Models. arXiv preprint arXiv:2303.10131 (2023).
  35. Beyond accuracy: Assessing software documentation quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1509–1512.
  36. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) 53, 3 (2020), 1–34.
  37. You Augment Me: Exploring ChatGPT-based Data Augmentation for Semantic Code Search. ([n. d.]).
  38. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023).
  39. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2303.07839 (2023).
  40. Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023).
  41. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4582–4591.
  42. DevGPT: Studying Developer-ChatGPT Conversations. (2024).
  43. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712 (2023).
  44. A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair. arXiv preprint arXiv:2310.08879 (2023).
  45. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Eman Abdullah AlOmar (32 papers)
  2. Anushkrishna Venkatakrishnan (1 paper)
  3. Mohamed Wiem Mkaouer (42 papers)
  4. Christian D. Newman (19 papers)
  5. Ali Ouni (36 papers)
Citations (12)
View on Semantic Scholar