Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers (2405.17739v1)

Published 28 May 2024 in cs.AI and cs.HC

Abstract: Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. From Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). 172–183. https://doi.org/10.1109/ICPC52881.2021.00025
  2. Many Small Programs in CS1: Usage Analysis from Multiple Universities. In 2019 ASEE Annual Conference & Exposition ”. ASEE Conferences, Tampa, Florida, 1–13. https://peer.asee.org/33084.
  3. An Analysis of Using Many Small Programs in CS1. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 585–591. https://doi.org/10.1145/3287324.3287466
  4. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang. 7, OOPSLA1, Article 78 (apr 2023), 27 pages. https://doi.org/10.1145/3586030
  5. Generative AI in Introductory Programming. https://csed.acm.org/large-language-models-in-introductory-programming
  6. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 500–506. https://doi.org/10.1145/3545945.3569759
  7. iTrace-Toolkit: A Pipeline for Analyzing Eye-Tracking Data of Software Engineering Studies. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 46–50. https://doi.org/10.1109/ICSE-Companion58688.2023.00022
  8. Eye Tracking in Computing Education. ICER 2014 - Proceedings of the 10th Annual International Conference on International Computing Education Research. https://doi.org/10.1145/2632320.2632344
  9. Doga Cambaz and Xiaoling Zhang. 2024. Use of AI-driven Code Generation Models in Teaching and Learning Programming: a Systematic Literature Review. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 172–178. https://doi.org/10.1145/3626252.3630958
  10. Iris: a tool for designing contextually relevant gaze visualizations. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications (Denver, Colorado) (ETRA ’19). Association for Computing Machinery, New York, NY, USA, Article 79, 5 pages. https://doi.org/10.1145/3317958.3318228
  11. Sarah D’Angelo and Darren Gergle. 2018. An Eye For Design: Gaze Visualizations for Remote Collaborative Work. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (, Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173923
  12. Impact of AI assistance on student agency. Computers & Education 210 (2024), 104967.
  13. Prompt Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). ACM, NY, USA, 7 pages.
  14. Desirable Characteristics for AI Teaching Assistants in Programming Education. arXiv preprint arXiv:2405.14178 (2024).
  15. Computing Education in the Era of Generative AI. Commun. ACM 67, 2 (jan 2024), 56–67. https://doi.org/10.1145/3624720
  16. Obstacles Women and Historically Marginalized Racial and Ethnic Groups (HMREG) Face in the Computer Science Field. Journal of Research Initiatives 6, 1 (2022), 9.
  17. Karl Anders Ericsson and Herbert Alexander Simon. 1993. Protocol Analysis (1st ed.). MIT Press, Cambridge, MA.
  18. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863
  19. My AI Wants to Know If This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE ’23). Association for Computing Machinery, New York, NY, USA, 97–104. https://doi.org/10.1145/3576123.3576134
  20. J. H. Flavell. 1976. Metacognitive Aspects of Problem Solving. The Nature of Intelligence (1976), 231–235. https://ci.nii.ac.jp/naid/10021876052/en/
  21. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 93–105. https://doi.org/10.1145/3568813.3600139
  22. More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems. In Proceedings of the 26th Australasian Computing Education Conference. 29–38.
  23. The Effects of Generative AI on Computing Students’ Help-Seeking Preferences. In Proceedings of the 26th Australasian Computing Education Conference (ACE ’24). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/3636243.3636248
  24. The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Applied Sciences 14, 10 (2024), 4115.
  25. CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 650, 20 pages. https://doi.org/10.1145/3613904.3642773
  26. Amy Ko. 2024. More than calculators: Why large language models threaten learning, teaching, and education. https://medium.com/bits-and-behavior/more-than-calculators-why-large-language-models-threaten-public-education-480dd5300939. Accessed: 2024-03-28.
  27. Sam Lau and Philip J Guo. 2023. From” Ban It Till We Understand It” to” Resistance is Futile”: How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. The 19th ACM Conference on International Computing Education Research (ICER) (2023).
  28. Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 124–130. https://doi.org/10.1145/3587102.3588785
  29. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 563–569. https://doi.org/10.1145/3545945.3569770
  30. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’23). Association for Computing Machinery, New York, NY, USA, Article 8, 11 pages. https://doi.org/10.1145/3631802.3631830
  31. Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 750–756. https://doi.org/10.1145/3626252.3630938
  32. Dastyni Loksa and Amy J. Ko. 2016. The Role of Self-Regulation in Programming Problem Solving Process and Success. In Proceedings of the 2016 ACM Conference on International Computing Education Research (Melbourne, VIC, Australia) (ICER ’16). Association for Computing Machinery, New York, NY, USA, 83–91. https://doi.org/10.1145/2960310.2960334
  33. Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1449–1461. https://doi.org/10.1145/2858036.2858252
  34. Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use. ACM Trans. Comput. Educ. 22, 4, Article 39 (sep 2022), 31 pages. https://doi.org/10.1145/3487050
  35. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 931–937. https://doi.org/10.1145/3545945.3569785
  36. Self-Regulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming Problems. In Proceedings of the 2024 Conference on Innovation and Technology in Computer Science Education V. 1 (Milan, Italy) (ITiCSE 2024). ACM, NY, USA.
  37. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282.
  38. Marvin Lee Minsky. 1994. Negative Expertise. International Journal of Expert Systems Research and Applications 7 (1994), 13–18. Issue 1.
  39. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 142, 16 pages. https://doi.org/10.1145/3613904.3641936
  40. A Survey on the Usage of Eye-Tracking in Computer Programming. Comput. Surveys 51 (01 2018), 1–58. https://doi.org/10.1145/3145904
  41. A Decade of Demographics in Computing Education Research: A Critical Review of Trends in Collection, Reporting, and Use. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER ’22). Association for Computing Machinery, New York, NY, USA, 323–343. https://doi.org/10.1145/3501385.3543967
  42. Metacodenition: Scaffolding the Problem-Solving Process for Novice Programmers. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE ’23). Association for Computing Machinery, New York, NY, USA, 59–68. https://doi.org/10.1145/3576123.3576130
  43. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590 [cs.SE]
  44. Bruno Pereira Cipriano and Pedro Alves. 2024. ” ChatGPT Is Here to Help, Not to Replace Anybody”–An Evaluation of Students’ Opinions On Integrating ChatGPT In CS Courses. arXiv e-prints (2024), arXiv–2404.
  45. Paul R Pintrich et al. 1991. A Manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). (1991).
  46. Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming with GitHub Copilot and ChatGPT. Manning, Shelter Island, NY, USA.
  47. What Do We Think We Think We Are Doing? Metacognition and Self-Regulation in Programming. In Proceedings of the 2020 ACM Conference on International Computing Education Research (Virtual Event, New Zealand) (ICER ’20). Association for Computing Machinery, New York, NY, USA, 2–13. https://doi.org/10.1145/3372782.3406263
  48. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (Turku, Finland) (ITiCSE-WGR ’23). Association for Computing Machinery, New York, NY, USA, 108–159. https://doi.org/10.1145/3623762.3633499
  49. First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 531–537. https://doi.org/10.1145/3287324.3287374
  50. Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER ’18). Association for Computing Machinery, New York, NY, USA, 41–50. https://doi.org/10.1145/3230977.3230981
  51. “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. ACM Trans. Comput.-Hum. Interact. 31, 1, Article 4 (nov 2024), 31 pages. https://doi.org/10.1145/3617367
  52. Self-efficacy and mental models in learning to program. In Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (Leeds, United Kingdom) (ITiCSE ’04). Association for Computing Machinery, New York, NY, USA, 171–175. https://doi.org/10.1145/1007996.1008042
  53. Jeffrey Rubin and Dana Chisnell. 2008. Handbook of usability testing: How to plan, design, and conduct effective tests. John Wiley & Sons.
  54. Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement. In Proceedings of the ACM Conference on Global Computing Education Vol 1 (Hyderabad, India) (CompEd 2023). Association for Computing Machinery, New York, NY, USA, 147–153. https://doi.org/10.1145/3576882.3617909
  55. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER ’22). Association for Computing Machinery, New York, NY, USA, 27–43. https://doi.org/10.1145/3501385.3543957
  56. Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. The 19th ACM Conference on International Computing Education Research (ICER) (2023).
  57. Marlene Schommer. 1990. Effects of Beliefs About the Nature of Knowledge on Comprehension. Journal of Educational Psychology 82, 3 (1990), 498–504. https://doi.org/10.1037/0022-0663.82.3.498
  58. Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Porland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 1223–1229. https://doi.org/10.1145/3626252.3630880
  59. The Metacognitive Demands and Opportunities of Generative AI. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 680, 24 pages. https://doi.org/10.1145/3613904.3642902
  60. A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers. In Proceedings of the 15th Australasian Computing Education Conference [Conferences in Research and Practice in Information Technology, Volume 136]. Australian Computer Society, 87–95.
  61. CS1-LLM: Integrating LLMs into CS1 Instruction. In Proceedings of the 29th Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’24) (Milan, Italy). ACM.
  62. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. Association for Computing Machinery, New York, NY, USA, 1–7.
  63. A Large Scale RCT on Effective Error Messages in CS1. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Porland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 1395–1401. https://doi.org/10.1145/3626252.3630764
  64. Jacqueline Whalley and Nadia Kasto. 2014. A qualitative think-aloud study of novice programmers’ code writing strategies. In Proceedings of the 2014 conference on Innovation & technology in computer science education. 279–284.
  65. Developing Novice Programmers’ Self-Regulation Skills with Code Replays. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 298–313. https://doi.org/10.1145/3568813.3600127
  66. A Theory of Instruction for Introductory Programming Skills. Computer Science Education 29, 2-3 (2019), 205–253. https://doi.org/10.1080/08993408.2019.1565235
  67. Does ChatGPT Help With Introductory Programming? An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th ACM/IEEE International Conference on Software Engineering (Lisbon, Portugal) (ICSE-SEET 2024). Association for Computing Machinery, New York, NY, USA.
  68. Ivie: Lightweight Anchored Explanations of Just-Generated Code. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 140, 15 pages. https://doi.org/10.1145/3613904.3642239
  69. Generative AI in Computing Education: Perspectives of Students and Instructors. In 2023 IEEE Frontiers in Education Conference (FIE). 1–9. https://doi.org/10.1109/FIE58773.2023.10343467
  70. Measuring GitHub Copilot’s Impact on Productivity. Commun. ACM 67, 3 (feb 2024), 54–63. https://doi.org/10.1145/3633453
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. James Prather (21 papers)
  2. Brent Reeves (2 papers)
  3. Juho Leinonen (41 papers)
  4. Stephen MacNeil (37 papers)
  5. Arisoa S. Randrianasolo (1 paper)
  6. Brett Becker (2 papers)
  7. Bailey Kimmel (4 papers)
  8. Jared Wright (2 papers)
  9. Ben Briggs (1 paper)
Citations (17)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets