From Self-Crafted to Engineered Prompts: Student Evaluations of AI-Generated Feedback in Introductory Physics

Abstract: The abilities of Generative-AI to produce real-time, sophisticated responses across diverse contexts has promised a huge potential in physics education, particularly in providing customized feedback. In this study, we investigate around 1200 introductory students' preferences about AI-feedback generated from three distinct prompt types: (a) self-crafted, (b) entailing foundational prompt-engineering techniques, and (c) entailing foundational prompt-engineering techniques along with principles of effective-feedback. The results highlight an overwhelming fraction of students preferring feedback generated using structured prompts, with those entailing combined features of prompt engineering and effective feedback to be favored most. However, the popular choice also elicited stronger preferences with students either liking or disliking the feedback. Students also ranked the feedback generated using their self-crafted prompts as the least preferred choice. Students' second preferences given their first choice and implications of the results such as the need to incorporate prompt engineering in introductory courses are discussed.