Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The role of library versions in Developer-ChatGPT conversations (2401.16340v1)

Published 29 Jan 2024 in cs.SE

Abstract: The latest breakthroughs in LLMs (LLM) have empowered software development tools, such as ChatGPT, to aid developers in complex tasks. Developers use ChatGPT to write code, review code changes, and even debug their programs. In these interactions, ChatGPT often recommends code snippets that depend on external libraries. However, code from libraries changes over time, invalidating a once-correct code snippet and making it difficult to reuse recommended code. In this study, we analyze DevGPT, a dataset of more than 4,000 Developer-ChatGPT interactions, to understand the role of library versions in code-related conversations. We quantify how often library version constraints are mentioned in code-related conversations and when ChatGPT recommends the installation of specific libraries. Our findings show that, albeit to constantly recommend and analyze code with external dependencies, library version constraints only appear in 9% of the conversations. In the majority of conversations, the version constraints are prompted by users (as opposed to being specified by ChatGPT) as a method for receiving better quality responses. Moreover, we study how library version constraints are used in the conversation through qualitative methods, identifying several potential problems that warrant further research.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (23)
  1. 2023. Incorporate local storage to put some state in the game. https://chat.openai.com/share/b57df6eb-4eea-4466-8fdc-e4e7431313d8. (Accessed on 13/01/2024).
  2. 2023. Refactoring Legacy React App. https://chat.openai.com/share/d18913ea-f872-45d8-b74c-600addff6b54. (Accessed on 12/10/2023).
  3. Empirical analysis of security vulnerabilities in Python packages. Empirical Software Engineering 28, 3 (March 2023), 59. https://doi.org/10.1007/s10664-022-10278-4
  4. Alexandre Decan and Tom Mens. 2021. What Do Package Dependencies Tell Us About Semantic Versioning? IEEE Transactions on Software Engineering 47, 6 (2021), 1226–1240. https://doi.org/10.1109/TSE.2019.2918315
  5. An empirical comparison of dependency network evolution in seven software packaging ecosystems. Empirical Software Engineering 24, 1 (Feb. 2019), 381–416. https://doi.org/10.1007/s10664-017-9589-y
  6. Natural Language Commanding via Program Synthesis. arXiv:2306.03460 [cs.LG]
  7. Hoshotakamoto. 2023. Unit Test Setup: Jest. https://chat.openai.com/share/eb107b81-4e26-48fc-a4fb-d2c1e315bae1. (Accessed on 12/10/2023).
  8. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE]
  9. Dependency Smells in JavaScript Projects. IEEE Trans. Softw. Eng. 48, 10 (oct 2022), 3790–3807. https://doi.org/10.1109/TSE.2021.3106247
  10. Towards JavaScript program repair with Generative Pre-trained Transformer (GPT-2). In 2022 IEEE/ACM International Workshop on Automated Program Repair (APR). 61–68. https://doi.org/10.1145/3524459.3527350
  11. Which is a better programming assistant? A comparative study between chatgpt and stack overflow. arXiv:2308.13851 [cs.SE]
  12. Where to Go Now? Finding Alternatives for Declining Packages in the npm Ecosystem. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, Los Alamitos, CA, USA, 1628–1639. https://doi.org/10.1109/ASE56229.2023.00119
  13. Stack Overflow. 2023. Stack Overflow Developer Survey 2023. https://survey.stackoverflow.co/2023/##technology-most-popular-technologies. (Accessed on 12/12/2023).
  14. Tom Preston-Werner. 2023. Semantic Versioning 2.0.0 — Semantic Versioning. https://semver.org/. (Accessed on 12/10/2023).
  15. Johnny Saldaña. 2021. The coding manual for qualitative researchers. The coding manual for qualitative researchers (2021), 1–440.
  16. Kayvan Shah. 2023. Restructured OpenAI Email Generation. https://chat.openai.com/share/c9e107c8-e424-4260-a85e-0bffcf334e2b. (Accessed on 12/12/2023).
  17. Benchmarking Language Models for Code Syntax Understanding. arXiv:2210.14473 [cs.CL]
  18. Tidelift. 2023. Libraries.io - The Open Source Discovery Service. https://libraries.io/. (Accessed on 12/12/2023).
  19. I Depended on You and You Broke Me: An Empirical Study of Manifesting Breaking Changes in Client Packages. ACM Trans. Softw. Eng. Methodol. 32, 4, Article 94 (may 2023), 26 pages. https://doi.org/10.1145/3576037
  20. Improving Automatic Source Code Summarization via Deep Reinforcement Learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE ’18). Association for Computing Machinery, New York, NY, USA, 397–407. https://doi.org/10.1145/3238147.3238206
  21. DevGPT: Studying Developer-ChatGPT Conversations. https://doi.org/10.5281/zenodo.8304091. https://doi.org/10.5281/zenodo.8304091
  22. DevGPT: Studying Developer-ChatGPT Conversations. In Proceedings of the International Conference on Mining Software Repositories (MSR 2024).
  23. Open Source Software: An Approach to Controlling Usage and Risk in Application Ecosystems. In Proceedings of the 26th ACM International Systems and Software Product Line Conference - Volume A (Graz, Austria) (SPLC ’22). Association for Computing Machinery, New York, NY, USA, 154–163. https://doi.org/10.1145/3546932.3547000
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Rachna Raj (1 paper)
  2. Diego Elias Costa (28 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets