Beyond the Hype: A Cautionary Tale of ChatGPT in the Programming Classroom (2406.11104v1)

Abstract: Due to the proliferation of LLMs research and the use of various AI tools, the field of information systems (IS) and computer science (CS) has evolved. The use of tools such as ChatGPT to complete various student programming exercises (e.g., in Python) and assignments has gained prominence amongst various academic institutions. However, recent literature has suggested that the use of ChatGPT in academia is problematic and the impact on teaching and learning should be further scrutinized. More specifically, little is known about how ChatGPT can be practically used with code (programming) writing to complete programming exercises amongst IS and CS undergraduate university students. Furthermore, the paper provides insights for academics who teach programming to create more challenging exercises and how to engage responsibly in the use of ChatGPT to promote classroom integrity. In this paper, we used Complex Adaptive Systems (CAS) theory as a theoretical guide to understand the various dynamics through classroom code demonstrations. Using ChatGPT 3.5, we analyzed the various practical programming examples from past IS exercises and compared those with memos created by tutors and lecturers in a university setting. This paper highlights common ways of assessment, programming errors created by ChatGPT and the potential consideration for IS academics to ensure the development of critical programming skills among students.