Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Navigating the Complexity of Generative AI Adoption in Software Engineering (2307.06081v2)

Published 12 Jul 2023 in cs.SE

Abstract: In this paper, the adoption patterns of Generative AI tools within software engineering are investigated. Influencing factors at the individual, technological, and societal levels are analyzed using a mixed-methods approach for an extensive comprehension of AI adoption. An initial structured interview was conducted with 100 software engineers, employing the Technology Acceptance Model (TAM), the Diffusion of Innovations theory (DOI), and the Social Cognitive Theory (SCT) as guiding theories. A theoretical model named the Human-AI Collaboration and Adaptation Framework (HACAF) was deduced using the Gioia Methodology, characterizing AI adoption in software engineering. This model's validity was subsequently tested through Partial Least Squares - Structural Equation Modeling (PLS-SEM), using data collected from 183 software professionals. The results indicate that the adoption of AI tools in these early integration stages is primarily driven by their compatibility with existing development workflows. This finding counters the traditional theories of technology acceptance. Contrary to expectations, the influence of perceived usefulness, social aspects, and personal innovativeness on adoption appeared to be less significant. This paper yields significant insights for the design of future AI tools and supplies a structure for devising effective strategies for organizational implementation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (121)
  1. Ritu Agarwal and Jayesh Prasad. 1998. A conceptual and operational definition of personal innovativeness in the domain of information technology. Information Systems Research 9, 2 (1998), 204–215.
  2. Ritu Agarwal and Jayesh Prasad. 1999. Are individual differences germane to the acceptance of new information technologies? Decision Sciences 30, 2 (1999), 361–391.
  3. AI, B. 2023. BLACKBOX AI — useblackbox.io. https://www.useblackbox.io/ Accessed: October 17, 2023.
  4. Icek Ajzen. 1991. The theory of planned behavior. Organizational Behavior and Human Decision Processes 50, 2 (1991), 179–211.
  5. Icek Ajzen. 2020. The theory of planned behavior: Frequently asked questions. Human Behavior and Emerging Technologies 2, 4 (2020), 314–324.
  6. Amazon Web Services. 2023. Code-Whisperer - Amazon Web Services. https://aws.amazon.com/es/codewhisperer/ Accessed: October 17, 2023.
  7. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82–115.
  8. S. Baltes and P. Ralph. 2020. Sampling in Software Engineering Research: A Critical Review and Guidelines. arXiv:2002.07764 (2020).
  9. Albert Bandura. 2014. Social cognitive theory of moral thought and action. In Handbook of moral behavior and development. Psychology press, 69–128.
  10. Self-Efficacy: The Exercise of Control. Journal of Cognitive Psychotherapy 13, 2 (1999), 158–166.
  11. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages 7 (2023), 85–111.
  12. James Bessen. 2019. Automation and jobs: When technology boosts employment. Economic Policy 34, 100 (2019), 589–626.
  13. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue 20, 6 (2022), 35–57.
  14. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901.
  15. On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML. Software and Systems Modeling (2023), 1–13.
  16. ”It would work for me too”: How Online Communities Shape Software Developers’ Trust in AI-Powered Code Generation Tools. arXiv preprint arXiv:2212.03491 (2022).
  17. W. W. Chin. 1998. The partial least squares approach to structural equation modeling. Modern Methods for Business Research 295, 2 (1998), 295–336.
  18. Clayton M. Christensen. 1997. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Harvard Business Review Press.
  19. Michael Chui et al. 2023. The economic potential of generative AI: The next productivity frontier. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier. [Accessed 03-Jul-2023].
  20. Thematic analysis. Qualitative psychology: A practical guide to research methods 3 (2015), 222–248.
  21. CodeComplete. 2023. CodeComplete: AI Coding Assistant for Enterprise. https://codecomplete.ai/ Accessed: October 17, 2023.
  22. Codeium. 2023. Codeium · Free AI Code Completion and Chat. https://codeium.com/ Accessed: October 17, 2023.
  23. J. Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Routledge.
  24. Deborah R Compeau and Christopher A Higgins. 1995. Computer self-efficacy: Development of a measure and initial test. MIS Quarterly (1995), 189–211.
  25. J. Corbin and A. Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology 13, 1 (1990), 3–21.
  26. J. Creswell. 2013. Research design: Qualitative, quantitative, and mixed methods approaches. Sage.
  27. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 203 (2023), 111734.
  28. Do you really code? Designing and Evaluating Screening Questions for Online Surveys with Programmers. In International Conference on Software Engineering. IEEE, 537–548.
  29. F. Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13, 3 (1989), 319–340.
  30. User acceptance of computer technology: A comparison of two theoretical models. Management science 35, 8 (1989), 982–1003.
  31. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2019).
  32. Paul Dourish. 2003. The appropriation of interactive technologies: Some lessons from placeless documents. Computer Supported Cooperative Work 12 (2003), 465–490.
  33. Christof Ebert and Panos Louridas. 2023. Generative AI for software practitioners. IEEE Software 40, 4 (2023), 30–38.
  34. Neil A Ernst and Gabriele Bavota. 2022. Ai-driven development is here: Should you worry? IEEE Software 39, 2 (2022), 106–110.
  35. Carl Benedikt Frey and Michael A Osborne. 2017. The future of employment: How susceptible are jobs to computerisation? Technological Forecasting and Social Change 114 (2017), 254–280.
  36. An Empirical Study on Quality Issues of Deep Learning Platform. In International Conference on Software Engineering: Software Engineering in Practice. IEEE, 455–466.
  37. Structural equation modeling and regression: Guidelines for research practice. Communications of the Association for Information Systems 4, 1 (2000), 7.
  38. Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational Research Methods 16, 1 (2013), 15–31.
  39. Symbolism and strategic change in academia: The dynamics of sensemaking and influence. Organization Science 5, 3 (1994), 363–383.
  40. B. Glaser and A. Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge.
  41. Dale L Goodhue and Ronald L Thompson. 1995. Task-technology fit and individual performance. MIS quarterly (1995), 213–236.
  42. Roberto Gozalo-Brizuela and Eduardo C Garrido-Merchán. 2023. A survey of Generative AI Applications. arXiv preprint arXiv:2306.02781 (2023).
  43. F. J. Gravetter and L.-A. B. Forzano. 2018. Research methods for the behavioral sciences. Cengage Learning.
  44. Achieving rigor in qualitative analysis: The role of active categorization in theory building. Academy of Management Review 46, 3 (2021), 591–612.
  45. E. Guba. 1981. Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Communication and Technology Journal 29, 2 (1981), 75–91.
  46. A primer on partial least squares structural equation modeling (PLS-SEM). Sage.
  47. Douglas M Hawkins. 2004. The problem of overfitting. Journal of chemical information and computer sciences 44, 1 (2004), 1–12.
  48. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019).
  49. Saki Imai. 2022. Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE International Conference on Software Engineering: Companion Proceedings. 319–321.
  50. L. Isabella. 1990. Evolving interpretations as a change unfolds: How managers construe key organizational events. Academy of Management Journal 33, 1 (1990), 7–41.
  51. Mateusz Jaworski and Dariusz Piotrkowski. 2023. Study of software developers’ experience using the Github Copilot Tool in the software development process.
  52. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020).
  53. Ron Kohavi. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, Vol. 14. 1137–1145.
  54. Rajiv Kohli and Nigel P Melville. 2019. Digital innovation: A review and synthesis. Information Systems Journal 29, 1 (2019), 200–223.
  55. A knowledge management success model: Theoretical development and empirical validation. Journal of Management Information Systems 23, 3 (2006), 309–347.
  56. Sources of influence on beliefs about information technology use: An empirical study of knowledge workers. MIS Quarterly (2003), 657–678.
  57. AlphaCode — alphacode.deepmind.com. https://alphacode.deepmind.com/ Accessed: October 17, 2023.
  58. Understanding the Usability of AI Programming Assistants. arXiv preprint arXiv:2303.17125 (2023).
  59. Y. Lincoln and E. Guba. 1985. Naturalistic inquiry. Sage.
  60. Mai Skjott Linneberg and Steffen Korsgaard. 2019. Coding qualitative data: A synthesis guiding the novice. Qualitative Research Journal 19, 3 (2019), 259–270.
  61. K. Locke. 1996. Rewriting the discovery of grounded theory after 25 years? Journal of Management Inquiry 5, 3 (1996), 239–245.
  62. On the robustness of code generation techniques: An empirical study on github copilot. arXiv preprint arXiv:2302.00438 (2023).
  63. J. Miles. 2014. Tolerance and Variance Inflation Factor. American Cancer Society.
  64. The ethics of algorithms: Mapping the debate. Big Data & Society 3, 2 (2016).
  65. Geoffrey A Moore and Regis McKenna. 1999. Crossing the chasm. (1999).
  66. Reading between the lines: Modeling user behavior and costs in AI-assisted programming. arXiv preprint arXiv:2210.14306 (2022).
  67. mutable.ai. 2023. AI Accelerated Software Development. — mutable.ai. https://mutable.ai/ Accessed: October 17, 2023.
  68. Nhan Nguyen and Sarah Nadi. 2022. An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the International Conference on Mining Software Repositories. 1–5.
  69. Oded Nov and Chen Ye. 2008. Users’ personality and perceived ease of use of digital libraries: The case for resistance to change. Journal of the American Society for Information Science and Technology 59, 5 (2008), 845–851.
  70. J. Nunnally. 1978. Psychometric methods. McGraw-Hill.
  71. General Assembly of the World Medical Association et al. 2014. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. The Journal of the American College of Dentists 81, 3 (2014), 14.
  72. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  73. Ipek Ozkaya. 2023. The next frontier in software development: AI-augmented software development processes. IEEE Software 40, 4 (2023), 4–9.
  74. Stefan Palan and Christian Schitter. 2018. Prolific.ac—-A subject pool for online experiments. Journal of Behavioral and Experimental Finance 17 (2018), 22–27.
  75. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023).
  76. How do i refactor this? An empirical study on refactoring trends and topics in Stack Overflow. Empirical Software Engineering 27, 1 (2022), 11.
  77. ” It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. arXiv preprint arXiv:2304.02491 (2023).
  78. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  79. Paul Ralph et al. 2020. Empirical standards for software engineering research. arXiv preprint arXiv:2010.03525 (2020).
  80. Q-TAM: A quality technology acceptance model for predicting organizational buyers’ continuance intentions for e-procurement services. International Journal of Production Economics 216 (2019), 333–348.
  81. A large scale study of programming languages and code quality in github. In International Symposium on Foundations of Software Engineering. ACM, 155–165.
  82. replit. 2023. Ghostwriter - Code faster with AI — replit.com. https://replit.com/site/ghostwriter Accessed: October 17, 2023.
  83. C. M. Ringle and M. Sarstedt. 2016. Gain more insight from your PLS-SEM results: The importance-performance map analysis. Industrial Management & Data Systems 116, 9 (2016), 1865–1886.
  84. SmartPLS 3.
  85. Everett M. Rogers. 2010. Diffusion of Innovations. Simon and Schuster.
  86. Daniel Russo. 2021. The agile success model: a mixed-methods study of a large-scale agile transformation. ACM Transactions on Software Engineering and Methodology 30, 4 (2021), 1–46.
  87. A Meta Model for Information Systems Quality: a Mixed-Study of the Financial Sector. ACM Transactions on Management Information Systems 9, 3 (2018).
  88. The daily life of software engineers during the covid-19 pandemic. In International Conference on Software Engineering. IEEE, 364–373.
  89. Predictors of Well-being and Productivity among Software Professionals during the COVID-19 Pandemic–A Longitudinal Study. Empirical Software Engineering 26, 62 (2021), 1–64. https://doi.org/10.1007/s10664-021-09945-9
  90. Daniel Russo and Klaas-Jan Stol. 2020. Gender Differences in Personality Traits of Software Engineers. IEEE Transactions on Software Engineering 48, 3 (2020), 16.
  91. Daniel Russo and Klaas-Jan Stol. 2021. PLS-SEM for software engineering research: An introduction and survey. Comput. Surveys 54, 4 (2021), 1–38.
  92. Mahadev Satyanarayanan. 2001. Pervasive computing: Vision and challenges. IEEE Personal Communications 8, 4 (2001), 10–17.
  93. Albrecht Schmidt. 2023. Speeding Up the Engineering of Interactive Systems with Generative AI. In ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 7–8.
  94. AH Segars and V Grover. 1993. Re-examining perceived ease of use and usefulness: A confirmatory factor analysis. MIS Quarterly 17, 4 (1993), 517–525.
  95. The elephant in the room: Predictive performance of PLS models. Journal of Business Research 69, 10 (2016), 4552–4564.
  96. Predictive model assessment in PLS-SEM: guidelines for using PLSpredict. European Journal of Marketing 53, 11 (2019).
  97. Choose your programming copilot: A comparison of the program synthesis performance of github copilot and genetic programming. In Proceedings of the Genetic and Evolutionary Computation Conference. 1019–1027.
  98. Klaas-Jan Stol and Brian Fitzgerald. 2018. The ABC of software engineering research. ACM Transactions on Software Engineering and Methodology 27, 3 (2018), 11.
  99. A. Strauss and J. Corbin. 1990. Basics of qualitative research: Grounded theory procedures and techniques. Sage.
  100. Tabnine. 2023. AI Assistant for software developers — Tabnine. https://www.tabnine.com/ Accessed: October 17, 2023.
  101. Emmanuel Tenakwah. 2021. What do employees want?: Halting record-setting turnovers globally. Strategic HR Review 20, 6 (2021), 206–210.
  102. Personal Computing: Toward a Conceptual Model of Utilization. MIS Quarterly 15, 1 (1991), 125–143.
  103. Is ChatGPT the Ultimate Programming Assistant–How far is it? arXiv preprint arXiv:2304.11938 (2023).
  104. A contingency model of computer and Internet self-efficacy. Information & Management 43, 4 (2006), 541–550.
  105. Louis G. Tornatzky and Katherine J. Klein. 1982. Innovation characteristics and innovation adoption-implementation: A meta-analysis of findings. IEEE Transactions on Engineering Management 29, 1 (1982), 28–45.
  106. The Cybersecurity Crisis of Artificial Intelligence: Unrestrained Adoption and Natural Language-Based Attacks. Available at SSRN 4578165 (2023).
  107. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In International Conference on Human Factors in Computing Systems. 1–7.
  108. J. Van Maanen. 1979. The fact of fiction in organizational ethnography. Administrative Science Quarterly 24, 4 (1979), 539–550.
  109. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).
  110. Viswanath Venkatesh and Fred D Davis. 2000. A theoretical extension of the technology acceptance model: four longitudinal field studies. Management science 46, 2 (2000), 186–204.
  111. User acceptance of information technology: toward a unified view. MIS quarterly (2003), 425–478.
  112. Unified theory of acceptance and use of technology: A synthesis and the road ahead. Journal of the Association for Information Systems 17, 5 (2016), 328–376.
  113. Vv.AA. 2023. The widespread adoption of AI by companies will take a while. https://www.economist.com/leaders/2023/06/29/the-widespread-adoption-of-ai-by-companies-will-take-a-while
  114. Investigating and Designing for Trust in AI-powered Code Generation Tools. arXiv preprint arXiv:2305.11248 (2023).
  115. Code generation as a dual task of code summarization. In International Conference on Neural Information Processing Systems. 6563–6573.
  116. Michel Wermelinger. 2023. Using GitHub Copilot to solve simple programming problems. In Proceedings of the ACM Technical Symposium on Computer Science Education. 172–178.
  117. Experimentation in Software Engineering. Springer Science & Business Media.
  118. Assessing the quality of GitHub copilot’s code generation. In Proceedings of the International Conference on Predictive Models and Data Analytics in Software Engineering. 62–71.
  119. Practices and Challenges of Using GitHub Copilot: An Empirical Study. arXiv preprint arXiv:2303.08733 (2023).
  120. The AI index 2021 annual report. arXiv preprint arXiv:2103.06312 (2021).
  121. GitHub - THUDM/CodeGeeX: CodeGeeX: An Open Multilingual Code Generation Model. https://github.com/THUDM/CodeGeeX Accessed: October 17, 2023.
Citations (40)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets