Policy for handling incorrect extraneous information in AI-assisted rubric grading

Determine a principled and consistent grading policy for GPT-4-based automated short-answer grading of handwritten university-level mathematics when students include additional statements that are not referenced by any grading rubric items and those statements are incorrect, specifying whether such extraneous content should be ignored or penalized and how to encode this policy in grading prompts and rubrics to align with human grading practice.

Background

Within the grading workflow, the authors included an instruction to ignore irrelevant information in student answers to prevent the model from being misled by scribbles, annotations, or material pertaining to other rubric items. However, they encountered edge cases where students provided additional statements not covered by any rubric items, raising the question of whether such content—especially if incorrect—should affect grading.

This ambiguity is not only a challenge for the AI prompt design but also for human graders, as existing human grading rubrics do not specify a policy for penalizing or ignoring incorrect extraneous statements. Establishing a clear policy is necessary to ensure fairness and consistency between AI and human grading.