Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study (2402.16480v1)

Published 26 Feb 2024 in cs.SE

Abstract: LLMs have gained significant attention in the software engineering community. Nowadays developers have the possibility to exploit these models through industrial-grade tools providing a handy interface toward LLMs, such as OpenAI's ChatGPT. While the potential of LLMs in assisting developers across several tasks has been documented in the literature, there is a lack of empirical evidence mapping the actual usage of LLMs in software projects. In this work, we aim at filling such a gap. First, we mine 1,501 commits, pull requests (PRs), and issues from open-source projects by matching regular expressions likely to indicate the usage of ChatGPT to accomplish the task. Then, we manually analyze these instances, discarding false positives (i.e., instances in which ChatGPT was mentioned but not actually used) and categorizing the task automated in the 467 true positive instances (165 commits, 159 PRs, 143 issues). This resulted in a taxonomy of 45 tasks which developers automate via ChatGPT. The taxonomy, accompanied with representative examples, provides (i) developers with valuable insights on how to exploit LLMs in their workflow and (ii) researchers with a clear overview of tasks that, according to developers, could benefit from automated solutions.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (118)
  1. [n. d.]. ChatGPT. https://openai.com/blog/chatgpt. Accessed: 2023-03-27.
  2. [n. d.]. Copilot Website. https://copilot.github.com. Accessed: 2022-11-10.
  3. 2023a. https://github.com/woocommerce/woocommerce/pull/37233.
  4. 2023b. https://github.com/fufexan/nix-gaming/pull/73.
  5. 2023a. https://github.com/danielgross/whatsapp-gpt/issues/68.
  6. 2023c. https://github.com/typescript-eslint/typescript-eslint/pull/6915.
  7. 2023b. https://github.com/dogsheep/apple-notes-to-sqlite/issues/1.
  8. 2023c. https://github.com/pfusik/cito/issues/80.
  9. 2023d. https://github.com/greenshot/greenshot/pull/484.
  10. 2023e. https://github.com/vcmi/vcmi/pull/1659.
  11. 2023f. https://github.com/garden-io/garden/pull/4553.
  12. 2023a. https://github.com/1j01/textual-paint/commit/e9494ddf.
  13. 2023d. https://github.com/go-go-golems/glazed/issues/50.
  14. 2023b. https://github.com/fluxninja/aperture/commit/70a68635.
  15. 2023e. https://github.com/talent-connect/connect/issues/658.
  16. 2023f. https://github.com/gofiber/fiber/issues/2301.
  17. 2023g. https://github.com/libp2p/js-libp2p/issues/1648.
  18. 2023g. https://github.com/spacedriveapp/spacedrive/pull/925.
  19. 2023h. https://github.com/pizzaboxer/bloxstrap/issues/224.
  20. 2023i. https://github.com/spring-cloud/spring-cloud-stream/issues/2643.
  21. 2023j. https://github.com/shaka-project/shaka-player/issues/5015.
  22. 2023k. https://github.com/module-federation/module-federation-examples/issues/2942.
  23. 2023l. https://github.com/prosyslab-classroom/cs348-information-security/issues/365.
  24. 2023c. https://github.com/pbui/bobbit/commit/089fc145.
  25. 2023d. https://github.com/dodona-edu/dodona/commit/9efb97f8.
  26. 2023e. https://github.com/reorx/jsoncv/commit/1d5f8f1d.
  27. 2023m. https://github.com/sirupsen/napkin-math/issues/26.
  28. 2023h. https://github.com/hubtype/botonic/pull/2491.
  29. 2023i. https://github.com/kvas-it/pytest-console-scripts/pull/76.
  30. 2023j. https://github.com/kkdai/chatgpt/pull/4.
  31. 2023f. https://github.com/spotlightpa/almanack/commit/955fc76b.
  32. 2023n. https://github.com/kohya-ss/sd-webui-additional-networks/issues/43.
  33. 2023o. https://github.com/rootzoll/raspiblitz/issues/3640.
  34. 2023p. https://github.com/puppeteer/puppeteer/issues/9959.
  35. 2023k. https://github.com/kiali/kiali/pull/5973.
  36. 2023l. https://github.com/shilomagen/passport-extension/pull/16.
  37. 2023m. https://github.com/kyverno/kyverno/pull/5834.
  38. 2023n. https://github.com/failfa-st/hyv/pull/1.
  39. 2023o. https://github.com/robherley/snips.sh/pull/17.
  40. 2023q. https://github.com/pwncollege/dojo/issues/132.
  41. 2023p. https://github.com/az-digital/az_quickstart/pull/2226.
  42. 2023r. n. https://github.com/igrigorik/videospeed/issues/1035.
  43. Automated Documentation of Android Apps. IEEE Transactions on Software Engineering, TSE 47, 1 (2021), 204–220.
  44. Suggesting Accurate Method and Class Names. In 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE. 38–49.
  45. Apache Software Foundation. [n. d.]. Guide for new project contributors https://community.apache.org/contributors/. Accessed: 2023-07-08.
  46. Is github’s copilot as bad as humans at introducing vulnerabilities in code? arXiv preprint arXiv:2204.04741 (2022).
  47. Getafix: learning to fix bugs automatically. ACM Program. Lang., Object-oriented Programming, Systems, Languages, and Applications OOPSLA (2019), 159:1–159:27.
  48. Studying test-driven development and its retainment over a six-month time span. J. Syst. Softw. 176 (2021), 110937.
  49. Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios. IEEE Trans. Software Eng. 49, 4 (2023), 1782–1803.
  50. A study on prompt design, advantages and limitations of chatgpt for deep learning program repair. arXiv preprint arXiv:2304.08191 (2023).
  51. Bias in machine learning software: why? how? what to do?. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 429–440.
  52. Fairway: a way to build fair ML software. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 654–665.
  53. Neural Transfer Learning for Repairing Security Vulnerabilities in C Code. IEEE Transactions on Software Engineering 49, 1 (2023), 147–165.
  54. WATER: Web Application TEst Repair. In Proceedings of the First International Workshop on End-to-End Test Script Engineering (ETSE ’11). Association for Computing Machinery, 24–29.
  55. An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering, TSE 48, 12 (2021), 4818–4837.
  56. Analyzing reviews and code of mobile apps for better release planning. In IEEE 24th International Conference on Software Analysis, Evolution and Reengineering, SANER. IEEE Computer Society, 91–102.
  57. J Cohen. 1960. A coefficient of agreement for nominal scales. Educ Psychol Meas. (1960).
  58. On Test Repair Using Symbolic Execution. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA ’10). Association for Computing Machinery, 207–218.
  59. FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, 970–981.
  60. Self-collaboration Code Generation via ChatGPT. arXiv preprint arXiv:2304.07590 (2023).
  61. Eclipse Foundation. [n. d.]. Platform/How to contribute https://wiki.eclipse.org/Platform/How_to_Contribute. Accessed: 2023-07-08.
  62. An Empirical Study of i18n Collateral Changes and Bugs in GUIs of Android apps. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 581–592.
  63. Automated Repair of Programs from Large Language Models. arXiv preprint arXiv:2205.10583 (2022).
  64. PRHAN: Automated Pull Request Description Generation Based on Hybrid Attention Network. Journal of Systems and Software 185 (2022), 111160.
  65. Enhancing Mobile App Bug Reporting via Real-Time Understanding of Reproduction Steps. IEEE Trans. Software Eng. 49, 3 (2023), 1246–1272.
  66. A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last? IEEE Trans. Software Eng. 43, 7 (2017), 597–614.
  67. Beyond Tests: Program Vulnerability Repair via Crash Constraint Extraction. ACM Trans. Softw. Eng. Methodol. 30, 2 (2021).
  68. Inc. Google. 2023. Try Bard, an AI expertiment by Google https://bard.google.com.
  69. Learning to Predict Severity of Software Vulnerability Using Only Vulnerability Description. In 33th IEEE International Conference on Software Maintenance and Evolution ICSME. 125–136.
  70. Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 319–321.
  71. AutoPRTitle: A Tool for Automatic Pull Request Title Generation. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). 454–458.
  72. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. (2023).
  73. Automatically generating commit messages from diffs using neural machine translation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 135–146.
  74. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems.
  75. Automating code review activities by large-scale pre-training. In 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE. 1035–1047.
  76. Neural-Machine-Translation-Based Commit Message Generation: How Far Are We?. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE ’18). Association for Computing Machinery, 373–384.
  77. Neural-machine-translation-based commit message generation: how far are we?. In 33rd IEEE/ACM International Conference on Automated Software Engineering, ASE. 373–384.
  78. Automatic Generation of Pull Request Descriptions. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 176–188.
  79. On ML-Based Program Translation: Perils and Promises. In 45th International Conference on Software Engineering, ICSE ’23, Companion Proceedings, 2023.
  80. Ehsan Mashhadi and Hadi Hemmati. 2021. Applying CodeBERT for Automated Program Repair of Java Simple Bugs. In 18th IEEE/ACM International Conference on Mining Software Repositories, MSR. 505–509.
  81. On the robustness of code generation techniques: An empirical study on github copilot. arXiv preprint arXiv:2302.00438 (2023).
  82. How Can I Use This Method?. In 37th IEEE/ACM International Conference on Software Engineering, ICSE. 880–890.
  83. Comparing Software Developers with ChatGPT: An Empirical Investigation. arXiv preprint arXiv:2305.11837 (2023).
  84. Statistical Learning Approach for Mining API Usage Mappings for Code Migration. In 29th IEEE/ACM International Conference on Automated Software Engineering, ASE. 457–468.
  85. Graph-based pattern-oriented, context-sensitive source code completion. In 34th IEEE/ACM International Conference on Software Engineering, ICSE. 69–79.
  86. Migrating Code with Statistical Machine Translation. In 36th IEEE/ACM International Conference on Software Engineering, ICSE. 544–547.
  87. Nhan Nguyen and Sarah Nadi. 2022. An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR). IEEE, 1–5.
  88. Adversarial Attacks to API Recommender Systems: Time to Wake Up and Smell the Coffeef𝑓fitalic_f. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021. 253–265.
  89. GUI-Guided Test Script Repair for Mobile Apps. IEEE Transactions on Software Engineering 48, 3 (2022), 910–929.
  90. How can i improve my app? Classifying user reviews for software maintenance and evolution. In IEEE International Conference on Software Maintenance and Evolution, ICSME. IEEE Computer Society, 281–290.
  91. An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions. arXiv preprint arXiv:2108.09293 (2021).
  92. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023).
  93. Recommendation Systems for Software Engineering. IEEE Softw. 27, 4 (2010), 80–86.
  94. Listening to the Crowd for the Release Planning of Mobile Apps. IEEE Trans. Software Eng. 45, 1 (2019), 68–86.
  95. An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint arXiv:2301.08653 (2023).
  96. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 (2021).
  97. Donna Spencer. 2009. Card sorting: Designing usable categories. Rosenfeld Media.
  98. ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks. arXiv preprint arXiv:2305.16837 (2023).
  99. Automatically detecting and describing high level actions within methods. In 33rd IEEE/ACM International Conference on Software Engineering, ICSE. 101–110.
  100. Overcoming open source project entry barriers with a portal for newcomers. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016. ACM, 273–284.
  101. Visual Web Test Repair. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). Association for Computing Machinery, 503–514.
  102. IntelliCode compose: code generation using transformer. In 28th ACM Joint European Software Engineering Conference and the ACM/SIGSOFT International Symposium on the Foundations of Software Engineering ESEC-FSE. 1433–1443.
  103. AutoTransform: Automated Code Transformation to Support Modern Code Review Process. In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE). 237–248.
  104. Is ChatGPT the Ultimate Programming Assistant–How far is it? arXiv preprint arXiv:2304.11938 (2023).
  105. An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation. ACM Trans. Softw. Eng. Methodol. 28, 4 (2019), 19:1–19:29.
  106. Using Pre-Trained Models to Boost Code Review Automation. In 44th IEEE/ACM International Conference on Software Engineering, ICSE. 2291–2302.
  107. Replication Package https://github.com/unveilingchatgptsusage/unveilingchatgptsusage.
  108. Towards Automating Code Review Activities. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE. 163–174.
  109. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7.
  110. Context-Aware Retrieval-Based Deep Commit Message Generation. ACM Trans. Softw. Eng. Methodol. (2021), 30 pages.
  111. Siri, Write the Next Method. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE. 138–149.
  112. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2303.07839 (2023).
  113. Exploring the Verifiability of Code Generated by GitHub Copilot. arXiv preprint arXiv:2209.01766 (2022).
  114. Improving Automated Bug Triaging with Specialized Topic Model. IEEE Trans. Software Eng. 43, 3 (2017), 272–297.
  115. Assessing the quality of GitHub copilot’s code generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering. 62–71.
  116. ReCDroid+: Automated End-to-End Crash Reproduction from Bug Reports for Android Apps. ACM Trans. Softw. Eng. Methodol. 31, 3 (2022), 36:1–36:33.
  117. Minghui Zhou and Audris Mockus. 2010. Growth of newcomer competence: challenges of globalization. In Proceedings of the Workshop on Future of Software Engineering Research, FoSER 2010, at the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 443–448.
  118. Productivity assessment of neural code completion. In International Symposium on Machine Programming. 21–29.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Rosalia Tufano (15 papers)
  2. Antonio Mastropaolo (25 papers)
  3. Federica Pepe (2 papers)
  4. Ozren Dabić (5 papers)
  5. Massimiliano Di Penta (31 papers)
  6. Gabriele Bavota (60 papers)
Citations (7)
View on Semantic Scholar