Empirical Comparison of Encoder-Based Language Models and Feature-Based Supervised Machine Learning Approaches to Automated Scoring of Long Essays

Abstract: Long context may impose challenges for encoder-only LLMs in text processing, specifically for automated scoring of essays. This study trained several commonly used encoder-based LLMs for automated scoring of long essays. The performance of these trained models was evaluated and compared with the ensemble models built upon the base LLMs with a token limit of 512?. The experimented models include BERT-based models (BERT, RoBERTa, DistilBERT, and DeBERTa), ensemble models integrating embeddings from multiple encoder models, and ensemble models of feature-based supervised machine learning models, including Gradient-Boosted Decision Trees, eXtreme Gradient Boosting, and Light Gradient Boosting Machine. We trained, validated, and tested each model on a dataset of 17,307 essays, with an 80%/10%/10% split, and evaluated model performance using Quadratic Weighted Kappa. This study revealed that an ensemble-of-embeddings model that combines multiple pre-trained LLM representations with gradient-boosting classifier as the ensemble model significantly outperforms individual LLMs at scoring long essays.