An Empirical Study to Understand How Students Use ChatGPT for Writing Essays (2501.10551v1)

Abstract: As LLMs advance and become widespread, students increasingly turn to systems like ChatGPT for assistance with writing tasks. Educators are concerned with students' usage of ChatGPT beyond cheating; using ChatGPT may reduce their critical engagement with writing, hindering students' learning processes. The negative or positive impact of using LLM-powered tools for writing will depend on how students use them; however, how students use ChatGPT remains largely unknown, resulting in a limited understanding of its impact on learning. To better understand how students use these tools, we conducted an online study $(n=70)$ where students were given an essay-writing task using a custom platform we developed to capture the queries they made to ChatGPT. To characterize their ChatGPT usage, we categorized each of the queries students made to ChatGPT. We then analyzed the relationship between ChatGPT usage and a variety of other metrics, including students' self-perception, attitudes towards AI, and the resulting essay itself. We found that factors such as gender, race, and perceived self-efficacy can help predict different AI usage patterns. Additionally, we found that different usage patterns were associated with varying levels of enjoyment and perceived ownership over the essay. The results of this study contribute to discussions about how writing education should incorporate generative AI-powered tools in the classroom.

An Empirical Study on Student Utilization of ChatGPT in Essay Writing

The paper "An Empirical Study to Understand How Students Use ChatGPT for Writing Essays" explores the increasingly prevalent use of LLMs like ChatGPT in student writing. With LLMs impacting various domains, including education, the paper seeks to empirically understand how students use these tools in writing assignments. This research provides an essential examination of LLM usage and reflects the attitudes toward and implications of such technologies in an educational setting.

The authors conducted an online paper involving 70 college students who completed an essay-writing task using a customized platform designed to capture their interactions with ChatGPT. By tracking queries sent to ChatGPT and subsequent writing behaviors, the paper aims to answer four primary research questions: the nature of student usage of ChatGPT in writing, predictive factors for ChatGPT usage patterns, how usage manifests in essay writing, and perceptions of the writing experience facilitated by ChatGPT.

Key Findings

1. Patterns and Categories of Usage: The paper identified four main categories of student interactions with ChatGPT, informed by Flower and Hayes' cognitive writing model: Planning, Translating, Reviewing, and delegating full Writing tasks (All). The paper's results indicate significant variance in ChatGPT usage across students, suggesting diverse motivations and strategies for leveraging AI in essay writing.
2. Demographic Predictors: The research highlighted that individual differences, such as gender, race, and self-efficacy in writing, could predict how frequently students engage ChatGPT. For example, students with lower self-efficacy in writing were more likely to rely on ChatGPT across all categories except for full writing tasks. Contrarily, there were no significant predictors found for the All category, indicating that other factors might be at play when students choose to delegate their writing entirely to ChatGPT.
3. Impact on Writing Output: The use of ChatGPT was associated with increased word counts in essays, particularly for students who used it for full writing tasks. However, the authors reported no significant difference in user-contributed text across user types. These findings suggest that while ChatGPT aids in generating more content, the interactive depth of student involvement in writing varies by individual usage strategy.
4. Perception and Experience: Students' perceptions of the essays they wrote with ChatGPT's assistance differed across user types. The paper found that those who used ChatGPT primarily for planning reported higher ownership of their written work compared to those who let ChatGPT write full parts of their essays. Notably, a mixed usage approach fostered greater enjoyment in the writing process, emphasizing the importance of user engagement in improving the subjective writing experience.

Practical and Theoretical Implications

The empirical insights from this paper have significant implications for pedagogical strategies and the integration of AI tools in writing education:

• Integration in Pedagogy: The research suggests the potential for integrating LLMs like ChatGPT in educational settings as assistants for planning, ideation, and review, rather than solely for content generation. Educators could leverage these tools to enhance learning objectives by encouraging critical engagement with writing tasks.
• Policy and Guidance Development: Findings reinforce the need for developing policies and guidelines regarding AI usage in academia. A deeper understanding of AI interaction patterns can assist educators in framing nuanced rules on ethical AI usage while fostering exploratory learning environments.
• Broader AI Adoption Discussions: By highlighting demographic predictors and variances in LLM usage, the paper enriches discussions on AI acceptance and its diverse impact on educational methodologies. Future research can extend beyond student perceptions and assess AI's broader impact on academic integrity and skill development.

Future Research Directions

The paper opens avenues for future work by raising questions about how ChatGPT's integration into academia can balance skill development with technological reliance. It suggests investigating strategies that foster authentic learning experiences and academic honesty amidst widespread AI adoption. Additionally, exploring gender and cultural nuances in AI interactions can yield insights into optimizing AI tools for inclusive educational practices.

In summary, this paper offers a comprehensive empirical investigation into how students use ChatGPT for essay writing. Its findings elucidate the complex interplay between LLMs and writing processes, providing a foundation for discussions on the evolving educational landscape influenced by advanced AI technologies.