Validate student-reported LLM role categorizations

Determine whether students’ self-reported classifications of an LLM as an Assistant, Peer, Expert, or None reflect genuine differences in how the LLM was used, such as providing a starting point for reasoning versus supplying a complete solution, rather than differences in students’ interpretations of the role labels.

Background

The study categorizes student interactions with LLMs using four self-reported roles: Assistant, Peer, Expert, and None. However, students selected these labels without accompanying definitions, and the survey instrument was not formally piloted to establish that students interpreted the categories consistently.

Consequently, it remains unresolved whether the reported role differences represent meaningful variation in students’ interactions with LLMs or merely variation in how students understood the labels. The paper suggests that future versions should provide explicit role definitions and compare self-reported categorizations with independent evidence, such as prompt logs.