Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
95 tokens/sec
Gemini 2.5 Pro Premium
55 tokens/sec
GPT-5 Medium
22 tokens/sec
GPT-5 High Premium
29 tokens/sec
GPT-4o
100 tokens/sec
DeepSeek R1 via Azure Premium
82 tokens/sec
GPT OSS 120B via Groq Premium
469 tokens/sec
Kimi K2 via Groq Premium
210 tokens/sec
2000 character limit reached

Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace (2410.18334v1)

Published 24 Oct 2024 in cs.SE

Abstract: Generative AI coding tools are relatively new, and their impact on developers extends beyond traditional coding metrics, influencing beliefs about work and developers' roles in the workplace. This study aims to illuminate developers' preexisting beliefs about generative AI tools, their self perceptions, and how regular use of these tools may alter these beliefs. Using a mixed methods approach, including surveys, a randomized controlled trial, and a three week diary study, we explored the real world application of generative AI tools within a large multinational software company. Our findings reveal that the introduction and sustained use of generative AI coding tools significantly increases developers' perceptions of these tools as both useful and enjoyable. However, developers' views on the trustworthiness of AI generated code remained unchanged. We also discovered unexpected uses of these tools, such as replacing web searches and fostering creative ideation. Additionally, 84 percent of participants reported positive changes in their daily work practices, and 66 percent noted shifts in their feelings about their work, ranging from increased enthusiasm to heightened awareness of the need to stay current with technological advances. This research provides both qualitative and quantitative insights into the evolving role of generative AI in software development and offers practical recommendations for maximizing the benefits of this emerging technology, particularly in balancing the productivity gains from AI-generated code with the need for increased scrutiny and critical evaluation of its outputs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. 2024. AI Data Drop: The 11-by-11 Tipping Point. https://www.microsoft.com/en-us/worklab/ai-data-drop-the-11-by-11-tipping-point
  2. Ritu Agarwal and Jayesh Prasad. 2000. A field study of the adoption of software process innovations by information systems professionals. IEEE Transactions on Engineering Management 47, 3 (2000), 295–308. https://doi.org/10.1109/17.865899
  3. Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25, 4 (1999), 456–473.
  4. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue 20, 6 (Jan. 2023), 35–57. https://doi.org/10.1145/3582083
  5. Supplemental Material for Dear Diary: A randomized controlled trial of Generative AI coding tools in the workplace. https://doi.org/10.5281/zenodo.8409170
  6. The Impact of AI Tool on Engineering at ANZ Bank An Emperical Study on GitHub Copilot within Coporate Environment. ArXiv abs/2402.05636 (2024). https://api.semanticscholar.org/CorpusID:267547947
  7. James M Conway and Charles E Lance. 2010. What reviewers should expect from authors regarding common method bias in organizational research. Journal of business and psychology 25 (2010), 325–334.
  8. Bent Flyvbjerg. 2006. Five misunderstandings about case-study research. Qualitative inquiry 12, 2 (2006), 219–245.
  9. Stephen L. France. 2024. Navigating software development in the ChatGPT and GitHub Copilot era. Business Horizons 67, 5 (2024), 649–661. https://doi.org/10.1016/j.bushor.2024.05.009 SPECIAL ISSUE: WRITTEN BY CHATGPT.
  10. Haotong Ge and Yuemeng Wu. 2023. An Empirical Study of Adoption of ChatGPT for Bug Fixing among Professional Developers. Innovation and Technology Advances 1, 1 (Jun. 2023), 21–29. https://doi.org/10.61187/ita.v1i1.19
  11. Eirini Kalliamvakou GitHub. 2021. The Good Day Project: Understanding Developer Productivity. GitHub Research (2021). https://github.com/research/good-day-project
  12. Saki Imai. 2022. Is GitHub copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 319–321. https://doi.org/10.1145/3510454.3522684
  13. Generative AI in Real-World Workplaces. Technical Report.
  14. Mateusz Jaworski and Dariusz Piotrkowski. 2023. Study of software developers’ experience using the Github Copilot Tool in the software development process. ArXiv abs/2301.04991 (2023). https://api.semanticscholar.org/CorpusID:255749277
  15. Heard it through the Gitvine: an empirical study of tool diffusion across the npm ecosystem (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 505–517. https://doi.org/10.1145/3368089.3409705
  16. N. H. Mackworth. 1950. Researchers on the measurement of human performance. Medical Research Council Special Report (1950).
  17. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE Transactions on Software Engineering 43, 12 (2017), 1178–1193. https://doi.org/10.1109/TSE.2017.2656886
  18. Unravelling Belgian Blue cattle farmers’ adoption intention towards diagnostic tools: Integrating insights from behavioural economics and socio-cognitive theories. Preventive Veterinary Medicine 188 (2021), 105238. https://doi.org/10.1016/j.prevetmed.2020.105238
  19. Assessing Primary Care Physicians’ Attitudes towards Adoption of an Electronic Tool to Support Cancer Diagnosis: An Exploratory Study. ElectronicHealthcare 11 (01 2012), e7–e16.
  20. Chatting with AI: Deciphering Developer Conversations with ChatGPT. In Proceedings of the 21st International Conference on Mining Software Repositories (Lisbon, Portugal) (MSR 24). Association for Computing Machinery, New York, NY, USA, 187–191. https://doi.org/10.1145/3643991.3645078
  21. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Systems and Software 203 (2023), 111734. https://doi.org/10.1016/j.jss.2023.111734
  22. Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects. https://doi.org/10.48550/arXiv.2406.17910
  23. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP). 754–768. https://doi.org/10.1109/SP46214.2022.9833571
  24. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv preprint arXiv:2302.06590 (2023).
  25. Early Results from a Study of GenAI Adoption in a Large Brazilian Company: The Case of Globo. Springer Nature Switzerland, Cham, 275–293. https://doi.org/10.1007/978-3-031-55642-5_13
  26. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of applied psychology 88, 5 (2003), 879.
  27. Explaining software developer acceptance of methodologies: a comparison of five theoretical models. IEEE Transactions on Software Engineering 28, 12 (2002), 1135–1145. https://doi.org/10.1109/TSE.2002.1158287
  28. Invited review: Determinants of farmers’ adoption of management-based strategies for infectious disease prevention and control. Journal of dairy science 100 5 (2017), 3329–3347. https://api.semanticscholar.org/CorpusID:8685115
  29. Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward.
  30. Jacky Swan and Sue Newell. 1994. Managers Beliefs about Factors Affecting the Adoption of Technological Innovation. Journal of Managerial Psychology 9 (1994), 3–11. https://api.semanticscholar.org/CorpusID:142842946
  31. Arun Vishwanath. 2009. From Belief-Importance to Intention: The Impact of Framing on Technology Adoption. Communication Monographs 76, 2 (2009), 177–206. https://doi.org/10.1080/03637750902828438 arXiv:https://doi.org/10.1080/03637750902828438
  32. Social influences on secure development tool adoption: why security tools spread. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (Baltimore, Maryland, USA) (CSCW ’14). Association for Computing Machinery, New York, NY, USA, 1095–1106. https://doi.org/10.1145/2531602.2531722
  33. Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions. Proc. ACM Softw. Eng. 1, FSE, Article 47 (July 2024), 23 pages. https://doi.org/10.1145/3643773
  34. Practices and Challenges of Using GitHub Copilot: An Empirical Study. In International Conference on Software Engineering and Knowledge Engineering. https://api.semanticscholar.org/CorpusID:257532814

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets