Moderating effects of pre-trained and zero-shot ML on automatic scoring performance

Determine how pre-trained language models (e.g., BERT, fine-tuned GPT-3.5) and zero-shot learning approaches (e.g., Matching Exemplar as Next Sentence Prediction, MeNSP) moderate the performance of machine learning-based automatic scoring systems for science assessments, specifically in terms of machine-human score agreements and scoring accuracy across tasks and contexts.

Background

Within the proposed framework for factors moderating machine-human agreements in ML-based automatic scoring, technical features such as algorithm type and attribute abstraction are identified as crucial contributors. Earlier studies have examined traditional supervised algorithms and feature engineering, while more recent work introduces pre-trained models (e.g., BERT, fine-tuned ChatGPT) and zero-shot methods (e.g., MeNSP).

Despite these advances, the chapter notes that the latest ML approaches have not been thoroughly investigated regarding their influence on assessment performance, leaving unclear to what extent and under what conditions these algorithms improve or alter machine-human agreement and accuracy in scoring complex, open-ended science responses.