Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Developers' Perceptions on the Impact of ChatGPT in Software Development: A Survey (2405.12195v1)

Published 20 May 2024 in cs.SE

Abstract: As LLMs, including ChatGPT and analogous systems, continue to advance, their robust natural language processing capabilities and diverse applications have garnered considerable attention. Nonetheless, despite the increasing acknowledgment of the convergence of AI and Software Engineering (SE), there is a lack of studies involving the impact of this convergence on the practices and perceptions of software developers. Understanding how software developers perceive and engage with AI tools, such as ChatGPT, is essential for elucidating the impact and potential challenges of incorporating AI-driven tools in the software development process. In this paper, we conducted a survey with 207 software developers to understand the impact of ChatGPT on software quality, productivity, and job satisfaction. Furthermore, the study delves into developers' expectations regarding future adaptations of ChatGPT, concerns about potential job displacement, and perspectives on regulatory interventions.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. ChatGPT: Fundamentals, Applications and Social Impacts. In 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS). 1–8. https://doi.org/10.1109/SNAMS58071.2022.10062688
  2. Software Engineering for Machine Learning: A Case Study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 291–300. https://doi.org/10.1109/ICSE-SEIP.2019.00042
  3. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). https://doi.org/10.48550/arXiv.2302.04023
  4. Arthur P. Brief and Loriann Roberson. 1989. Job Attitude Organization: An Exploratory Study1. Journal of Applied Social Psychology 19, 9 (1989), 717–727. https://doi.org/10.1111/j.1559-1816.1989.tb01254.x
  5. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 159, 25 pages. https://dl.acm.org/doi/abs/10.5555/3495724.3495883
  6. The AI Effect: Working at the Intersection of AI and SE. IEEE Software 37, 4 (2020), 26–35. https://doi.org/10.1109/MS.2020.2987666
  7. Bias in machine learning software: why? how? what to do?. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece) (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA, 429–440. https://doi.org/10.1145/3468264.3468537
  8. How is ChatGPT’s behavior changing over time? arXiv preprint arXiv:2307.09009 (2023). https://doi.org/10.48550/arXiv.2307.09009
  9. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence 6 (2023). https://doi.org/10.3389/frai.2023.1169595
  10. Jianyang Deng and Yijia Lin. 2023. The Benefits and Challenges of ChatGPT: An Overview. Frontiers in Computing and Intelligent Systems 2, 2 (Jan. 2023), 81–83. https://doi.org/10.54097/fcis.v2i2.4465
  11. Ways of Applying Artificial Intelligence in Software Engineering. In 2018 IEEE/ACM 6th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). 35–41. https://doi.org/10.1145/3194104.3194109
  12. R. A. Fisher. 1922. On the Interpretation of χ𝜒\chiitalic_χ2 from Contingency Tables, and the Calculation of P. Journal of the Royal Statistical Society 85, 1 (1922), 87–94. http://www.jstor.org/stable/2340521
  13. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds Mach. 30, 4 (dec 2020), 681–694. https://doi.org/10.1007/s11023-020-09548-1
  14. The State of the ML-universe: 10 Years of Artificial Intelligence & Machine Learning Software Development on GitHub. In Proceedings of the 17th International Conference on Mining Software Repositories (Seoul, Republic of Korea) (MSR ’20). Association for Computing Machinery, New York, NY, USA, 431–442. https://doi.org/10.1145/3379597.3387473
  15. Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 34, 13 pages. https://doi.org/10.1145/3597503.3623306
  16. Jigsaw: Large Language Models Meet Program Synthesis. (2022), 1219–1231. https://doi.org/10.1145/3510003.3510203
  17. Mateusz Jaworski and Dariusz Piotrkowski. 2023. Study of software developers’ experience using the Github Copilot Tool in the software development process. arXiv preprint arXiv:2301.04991 (2023). https://doi.org/10.48550/arXiv.2301.04991
  18. Capers Jones. 1991. Applied software measurement: assuring productivity and quality. McGraw-Hill, Inc.
  19. Advances in natural language processing–a survey of current research trends, development tools and industry applications. International Journal of Recent Technology and Engineering 7, 5C (2019), 199–202. https://doi.org/10.1126/science.aaa8685
  20. Natural language processing: State of the art, current trends and challenges. Multimedia tools and applications 82, 3 (2023), 3713–3744. https://doi.org/10.1007/s11042-022-13428-4
  21. Barbara A Kitchenham and Shari L Pfleeger. 2008. Personal opinion surveys. In Guide to advanced empirical software engineering. Springer, 63–92. https://doi.org/10.1007/978-1-84800-044-5_3
  22. Rajesh H. Kulkarni and Palacholla Padmanabham. 2017. Integration of artificial intelligence activities in software development processes and measuring effectiveness of integration. IET Software 11, 1 (2017), 18–26. https://doi.org/10.1049/iet-sen.2016.0095
  23. J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174. http://www.jstor.org/stable/2529310
  24. Kinley Larntz. 1978. Small-Sample Comparisons of Exact Levels for Chi-Squared Goodness-of-Fit Statistics. J. Amer. Statist. Assoc. 73, 362 (1978), 253–263. https://doi.org/10.1080/01621459.1978.10481567
  25. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (, Lisbon, Portugal,) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 52, 13 pages. https://doi.org/10.1145/3597503.3608128
  26. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2307.09009 36 (2023). https://doi.org/10.48550/arXiv.2305.01210
  27. The Strucplot Framework: Visualizing Multi-way Contingency Tables with vcd. Journal of Statistical Software 17, 3 (2006), 1–48. https://doi.org/10.18637/jss.v017.i03
  28. H.D. Mills. 1983. Software Productivity. Little, Brown. https://books.google.com.br/books?id=-JdQAAAAMAAJ
  29. Recent advances in natural language processing via large pre-trained language models: A survey. Comput. Surveys 56, 2 (2023), 1–40. https://doi.org/10.1145/3605943
  30. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/.
  31. OpenAI. 2023. ChatGPT Models Documentation. https://platform.openai.com/docs/models/gpt-3-5-turbo.
  32. I. Ozkaya. 2023. Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications. IEEE Software 40, 03 (may 2023), 4–8. https://doi.org/10.1109/MS.2023.3248401
  33. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023).
  34. Improving language understanding by generative pre-training. OpenAI blog (2018).
  35. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  36. Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes. In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE Computer Society, Los Alamitos, CA, USA, 354–361. https://doi.org/10.1109/SEAA60479.2023.00061
  37. Extending the Frontier of ChatGPT: Code Generation and Debugging. (2023). https://doi.org/10.48550/arXiv.2307.08260 arXiv:2307.08260
  38. Johnny Saldaña. 2021. The Coding Manual for Qualitative Researchers. SAGE Publications Ltd, London.
  39. A Literature Review of Using Machine Learning in Software Development Life Cycle Stages. IEEE Access 9 (2021), 140896–140920. https://doi.org/10.1109/ACCESS.2021.3119746
  40. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR). IEEE Computer Society, Los Alamitos, CA, USA, 23–30. https://doi.org/10.1109/APR59189.2023.00012
  41. StackOverflowLab. 2023. Developer sentiment around AI/ML. Technical Report. https://stackoverflow.co/labs/developer-sentiment-ai-ml/
  42. Towards a Theory of Software Developer Job Satisfaction and Perceived Productivity. IEEE Transactions on Software Engineering 47, 10 (2021), 2125–2142. https://doi.org/10.1109/TSE.2019.2944354
  43. Is ChatGPT the ultimate programming assistant–how far is it? arXiv preprint arXiv:2304.11938 (2023). https://doi.org/10.48550/arXiv.2304.11938
  44. Machine Learning for Technical Debt Identification. IEEE Transactions on Software Engineering 48, 12 (dec 2022), 4892–4906. https://doi.org/10.1109/TSE.2021.3129355
  45. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages. https://doi.org/10.1145/3491101.3519665
  46. Attention is all you need. Advances in neural information processing systems 30 (2017).
  47. On the Validity of Pre-Trained Transformers for Natural Language Processing in the Software Engineering Domain. IEEE Transactions on Software Engineering 49, 04 (apr 2023), 1487–1507. https://doi.org/10.1109/TSE.2022.3178469
  48. How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering 47, 9 (2021), 1857–1871. https://doi.org/10.1109/TSE.2019.2937083
  49. Software-Engineering Design Patterns for Machine Learning Applications. Computer 55, 3 (2022), 30–39. https://doi.org/10.1109/MC.2021.3137227
  50. Experimentation in Software Engineering. Springer Publishing Company, Incorporated.
  51. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. Association for Computational Linguistics, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6
  52. Exploring the verifiability of code generated by github copilot. arXiv preprint arXiv:2209.01766 (2022). https://doi.org/10.48550/arXiv.2209.01766
  53. Assessing the quality of GitHub copilot’s code generation. In Proceedings of the 18th international conference on predictive models and data analytics in software engineering. 62–71. https://doi.org/10.1145/3558489.3559072
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Thiago S. Vaillant (1 paper)
  2. Felipe Deveza de Almeida (1 paper)
  3. Paulo Anselmo M. S. Neto (2 papers)
  4. Cuiyun Gao (97 papers)
  5. Jan Bosch (20 papers)
  6. Eduardo Santana de Almeida (14 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets