ChatGPT in the Classroom: Boon or Bane for Physics Students' Academic Performance?

Abstract: This study investigates the influence of ChatGPT, an AI-based LLM, on student performance in a physics course. We conducted an experimental analysis with two cohorts of students in a second-semester engineering physics course. The control group (Physics 2 2022B) used traditional teaching methods, while the experimental group (Physics 2 2023A) integrated ChatGPT as a learning tool. Our results indicate that the use of ChatGPT led to a significant decrease in student performance, as evidenced by lower grades and negative Hake factors compared to the control group. In addition, a survey of students revealed conflicting perceptions of the usefulness of ChatGPT in teaching physics. While most recognized its usefulness in understanding concepts and providing information, concerns were raised about its potential to reduce critical thinking and independent learning. These findings suggest that while ChatGPT can be a useful tool, it should be used with caution and as a supplement to traditional teaching methods, rather than as a stand-alone solution. The study underlines the importance of critical and reflective use of AI tools in educational settings and highlights the irreplaceable role of teachers in providing comprehensive educational support.