AI Meets the Classroom: When Do Large Language Models Harm Learning? (2409.09047v2)

Abstract: The effect of LLMs in education is debated: Previous research shows that LLMs can help as well as hurt learning. In two pre-registered and incentivized laboratory experiments, we find no effect of LLMs on overall learning outcomes. In exploratory analyses and a field study, we provide evidence that the effect of LLMs on learning outcomes depends on usage behavior. Students who substitute some of their learning activities with LLMs (e.g., by generating solutions to exercises) increase the volume of topics they can learn about but decrease their understanding of each topic. Students who complement their learning activities with LLMs (e.g., by asking for explanations) do not increase topic volume but do increase their understanding. We also observe that LLMs widen the gap between students with low and high prior knowledge. While LLMs show great potential to improve learning, their use must be tailored to the educational context and students' needs.

Analyzing the Use of ChatGPT in Coding Education: Impacts on Learning

The paper titled “Training to Code with Generative AI: How Students' Use of ChatGPT Affects their Learning” investigates the nuanced role of LLMs, specifically ChatGPT, in facilitating coding education. Conducted by Matthias Lehmann, Philipp B. Cornelius, and Fabian J. Sting, the paper offers empirical insights through a combination of observational data from university courses and controlled experimental studies.

Methodology and Findings

The research comprises three distinct studies. Initially, field data from programming courses highlight that LLM usage has dual effects. Positive learning outcomes were observed when students used ChatGPT as a supplemental tool for gaining explanations. Conversely, negative consequences arose when students excessively relied on the AI to directly solve practice problems, bypassing the problem-solving process.

Two additional experiments were conducted to establish causality and investigate these effects further. These experiments confirmed that students benefitting from ChatGPT used it relationally as a tutor. The contrast in outcomes emphasizes the role of methodological engagement in educational success with AI tools.

Key Results

1. Causal Influence: Through controlled experimentation, it was demonstrated that ChatGPT, when used to seek explanations, positively influences learning outcomes. This establishes LLMs as potential support tools in educational settings, paralleling the interaction with knowledgeable tutors.
2. Harmful Dependence: The research confirms that over-reliance on ChatGPT for straightforward solutions impairs learning. Students might resolve problems presented in exercises without engaging deeply, which is detrimental to long-term understanding.
3. Beginner Benefits: Students with limited prior exposure to coding derived more benefits from LLMs, suggesting that these tools serve as effective aids in bridging knowledge gaps. However, initial users are prone to misusing such tools, indicating a critical area for future guidance and educational policy development.
4. Perceived vs. Actual Learning: There was a noted discrepancy between perceived learning and actual performance improvements, indicating the potential overestimation of capabilities fostered by AI assistance.

Implications and Future Directions

The research contributes substantially to the discourse surrounding AI in education by delineating when and how LLMs can either support or hinder learning processes. Practically, this suggests that educational institutions must provide structured guidelines on the optimal use of AI tools. This includes fostering critical engagement with LLMs to prevent over-reliance.

On a theoretical level, the paper raises questions about integrating AI seamlessly into pedagogical strategies, indicating potential avenues for curriculum development that incorporate AI while retaining essential cognitive engagement.

Speculations on Future Developments

Looking ahead, further refinement of AI functionalities could tailor educational experiences more precisely to student needs, providing context-aware scaffolding without undermining learning through practice. Further research might explore adaptive AI models that dynamically adjust the extent of support based on user interaction patterns, providing more personalized learning experiences.

Overall, while LLMs like ChatGPT hold promise as educational allies, strategic implementation and continuous evaluation are crucial in harnessing their full potential while mitigating risks associated with their misuse.