Untangling Critical Interaction with AI in Students Written Assessment (2404.06955v1)

Abstract: AI has become a ubiquitous part of society, but a key challenge exists in ensuring that humans are equipped with the required critical thinking and AI literacy skills to interact with machines effectively by understanding their capabilities and limitations. These skills are particularly important for learners to develop in the age of generative AI where AI tools can demonstrate complex knowledge and ability previously thought to be uniquely human. To activate effective human-AI partnerships in writing, this paper provides a first step toward conceptualizing the notion of critical learner interaction with AI. Using both theoretical models and empirical data, our preliminary findings suggest a general lack of Deep interaction with AI during the writing process. We believe that the outcomes can lead to better task and tool design in the future for learners to develop deep, critical thinking when interacting with AI.

Critical Interaction with AI in Students' Written Assessment

The paper "Untangling Critical Interaction with AI in Students' Written Assessment" examines the essential challenge posed by integrating AI into educational practices, specifically focusing on developing critical interaction skills among students. The authors highlight the necessity for learners, especially in the age of generative AI, to be equipped with critical thinking and AI literacy skills to interact effectively with AI technologies in educational assessments.

Conceptual Framework and Methodology

The paper introduces a novel Critical Interaction with AI for Writing (CIAW) framework, developed by synthesizing three established theoretical models. These models include the Cognitive Process Model of Writing, the IPS-I model for information problem-solving using the Internet, and the Student Approaches to Learning framework. The CIAW framework categorizes student interactions with AI into five dimensions: Critical Interaction for Planning and Ideation, Critical Interaction for Information Seeking and Evaluation, Critical Interaction for Writing and Presentation, Personal Reflection on AI-assisted Learning, and Conversational Engagement. Each dimension is further classified into deep, shallow, or absent levels of engagement.

The paper employs a qualitative content analysis of 49 graduate student assignments from a data science course to evaluate the framework. Students were encouraged to use ChatGPT as a writing support tool, and the analysis focused on their self-reflections and ChatGPT interactions.

Findings

The analysis reveals a predominant trend of shallow engagement with AI tools across most dimensions. Notably, 83.7% of interactions in Planning and Ideation were shallow, while 89.8% of interactions in Information Seeking and Evaluation fell into the same category. Writing and Presentation processes also predominantly involved shallow interactions (18.4%), indicating a tendency among students to limit their use of AI to basic tasks such as rephrasing and formatting.

However, a significant portion of students demonstrated deep engagement in Personal Reflection (49%), suggesting an awareness of the limitations and ethical considerations of using AI tools. This insight points to students' potential to utilize AI responsibly, provided they are guided to think critically about their writing processes.

Implications

The paper has several implications for educational practice and future research. The shallow engagement observed in most dimensions points to an urgent need for rethinking educational curricula and assessment designs. Educators must emphasize AI literacy and critical interaction skills, empowering students to harness the full potential of AI technologies without compromising their learning experience.

Moreover, the findings underscore the necessity of integrating guided reflection into educational settings. Such reflections could help students critically analyze their interactions with AI, fostering deeper learning and understanding.

Future Directions

Future research could leverage learning analytics to automate the evaluation of critical interactions, analyzing metrics such as similarity scores between student submissions and AI outputs, prompting efficiency indicators, and diversity in source referencing. Expanding the dataset across varying disciplines and educational contexts would provide broader insights into AI's role in education.

Overall, the paper initiates an important discussion on the implications of AI integration in education, laying the groundwork for further investigations into optimizing human-AI partnerships in learning environments.