Automated Short Answer Grading

• Automated Short Answer Grading (ASAG) is a method for computationally evaluating free-text student responses by comparing them against reference answers and rubrics.
• It employs diverse techniques including supervised regression, discrete classification, and neural architectures—ranging from feature-engineered models to LLM pipelines—to assess answer quality.
• Current research focuses on enhancing cross-domain generalization, explanation quality, calibration, and robustness against adversarial attacks.

Automated Short Answer Grading (ASAG) is the computational assessment of a student’s short, free-text response to a question by comparing it with a reference answer, a rubric, or both. In its classical form, the task takes a question, a desired answer, and a student answer, and predicts a grade; in contemporary formulations it may also predict discrete correctness labels, analytic sub-scores for multiple sub-questions, justification spans, and feedback text. The field spans regression and classification settings, domain-specific and multi-domain evaluation, feature-engineered and end-to-end neural architectures, and, increasingly, large-language-model (LLM) pipelines that combine grading with explanation, retrieval, uncertainty estimation, and human oversight (Gaddipati et al., 2020, Aggarwal et al., 2024, Yoon, 2023).

1. Task structure and problem formulations

ASAG has been formalized in several closely related ways. A common formulation treats grading as supervised regression: given a question, a reference answer, and a student answer, predict a score on a numeric scale such as $0$–$5$ or a normalized variant of that scale. Other work frames it as discrete classification, including binary correct/incorrect prediction, three-way schemes such as incorrect/partially correct/correct, and five-way schemes such as correct, contradictory, partially_correct_incomplete, irrelevant, and non_domain (Gaddipati et al., 2020, Liu et al., 2019, Aggarwal et al., 2024, Cong et al., 30 Apr 2026).

The input representation has also broadened. Some systems consume only the student answer and a reference answer; others incorporate the question text explicitly, and recent rubric-based systems condition on a textual grading guideline or a set of rubric items. Feedback-oriented ASAG further expands the output space from a single score to a pair $(y,F)$, where $y\in\{0,1,2\}$ is a discrete label and $F$ is content-focused feedback. In multi-part items, analytic sub-scores $s_i\in\{0,1\}$ can be assigned to sub-questions and then summed to a holistic score $S=\sum_i s_i$ (Aggarwal et al., 2024, Yoon, 2023).

A persistent technical issue is that semantic similarity in ASAG is not universal across domains. Joint multi-domain work argues that some similarity patterns are generic, while others are domain-specific: words or phrases that are not strict synonyms may still be equivalent within a subject area. This observation motivated architectures that learn both generic and domain-specific components rather than relying on a single universal notion of answer similarity (Saha et al., 2019).

2. Datasets, benchmarks, and evaluation protocols

Public ASAG research is distributed across heterogeneous corpora that vary in subject matter, scoring scale, answer length, and label granularity. Widely used resources include the Mohler dataset for introductory computer-science questions, SciEntsBank and Beetle from SemEval-2013, ASAP-SAS subsets in science and biology, the bilingual Short-Answer Feedback corpus, and the engineering-focused EngSAF dataset with feedback annotations. Larger proprietary or institutional corpora also appear, including a 120,000-pair K–12 dataset and cross-disciplinary university exam collections (Gaddipati et al., 2020, Yoon, 2023, Künnecke et al., 2024, Aggarwal et al., 2024, Liu et al., 2019, Gobrecht et al., 2024).

Corpus	Domain / scale	Noted properties
Mohler (Gaddipati et al., 2020)	Introductory computer science, $0$–$5$	80 questions, 31 students, 2,273 answers
SciEntsBank / Beetle (Kortemeyer, 2023)	Science / electronics, 2-way and 3-way; also 5-way in later work	504 questions/items, official SemEval test splits
ASAP-SAS subset (Yoon, 2023)	Science and biology	6,542 answers, 3 sub-questions per item
EngSAF (Aggarwal et al., 2024)	Engineering, $y\in\{0,1,2\}$	119 questions, 5,814 responses, feedback text
Bilingual SAF (Künnecke et al., 2024)	German micro-job training and English communication networks, $5$0 in 0.125 increments	6 German and 26 English questions
K–12 corpus (Liu et al., 2019)	Multiple school subjects, binary	120,000 answer/reference pairs
ASAG2024 (Meyer et al., 2024)	Seven combined English-language datasets, normalized $5$1	Roughly 19,000 triplets

Evaluation practice is correspondingly diverse. Regression-oriented work reports RMSE and Pearson correlation; classification settings use accuracy, macro-F1, weighted F1, Cohen’s kappa, and AUC; feedback-oriented work adds BLEU, METEOR, ROUGE-2 recall, and BERTScore; confidence-aware grading uses Brier score, ECE, MCE, AUROC, and AUARC. ASAG2024 standardized seven datasets to a common $5$2 scale and emphasized RMSE and weighted RMSE, explicitly to expose cross-dataset generalization rather than single-corpus optimization (Gaddipati et al., 2020, Aggarwal et al., 2024, Liu et al., 2019, Meyer et al., 2024, Cong et al., 30 Apr 2026).

Benchmark composition strongly affects conclusions. The Mohler dataset is skewed toward correct answers, with mean score approximately $5$3 and median approximately $5$4, while the K–12 dataset used by the multiway attention model is balanced between correct and incorrect answers. EngSAF separates unseen-answer and unseen-question splits, and ASAG2024 highlights cross-subject and cross-scale variability by combining Beetle II, CU-NLP, DigiKlausur, Mohler, SAF, SciEntsBank, and Stita under a single schema (Gaddipati et al., 2020, Liu et al., 2019, Aggarwal et al., 2024, Meyer et al., 2024).

3. Methodological development before the LLM era

Early ASAG systems relied on ontology construction, semantic similarity, and statistical aggregation. The 2010 ASAGS system combined BLEU-style lexical overlap, WordNet path and information-content measures, tf–idf cosine similarity, and ontology-based concept mapping, and reported an average correlation of $5$5, compared with $5$6 for a vector-space baseline and $5$7 for an existing n-gram overlap method. In that regime, semantic augmentation rather than raw lexical matching was the central design principle (Selvi et al., 2010).

A major later step was transfer learning without labeled target-question answers. The iterative ensemble of Roy et al. combined a text classifier over student answers with a numeric-feature classifier based on similarity to model answers, then used canonical correlation analysis to align source and target questions. On SciEntsBank it reported weighted-F1 scores of $5$8 on unseen answers, $5$9 on unseen questions, and $(y,F)$0 on unseen domains, and it outperformed all winning supervised SemEval entries while also approaching skyline supervised performance on several other datasets (Roy et al., 2016).

Deep-learning work then diversified into generic embeddings, sequence models, and attention architectures. The 2022 survey organized the literature into word-embedding methods, sequential models such as BiLSTMs, and attention-based methods such as transformers, and argued that learned representations alone were usually not sufficient for the best ASAG performance; hybrid systems that fused hand-engineered lexical, syntactic, or symbolic features with neural representations often remained strongest. This conclusion is consistent with several empirical lines in the literature rather than with a single dominant architecture (Haller et al., 2022).

Two 2019 architectures illustrate this transition. JMD-ASAG learned a generic scorer and a domain-specific scorer jointly and showed that their combination outperformed either component alone, reaching $(y,F)$1 accuracy and $(y,F)$2 macro-F1 on a large five-domain industry dataset and macro-F1 $(y,F)$3 on the 3-way SciEntsBank setting. In parallel, the multiway attention network used transformer encoding together with self-attention and cross-attention between student and reference answers, obtaining $(y,F)$4 accuracy and $(y,F)$5 AUC on a 120,000-pair K–12 corpus (Saha et al., 2019, Liu et al., 2019).

Pretrained contextual encoders further complicated the picture. On the Mohler dataset, a deliberately simple pipeline that summed contextual token embeddings and used cosine similarity as its sole feature found that ELMo outperformed BERT, GPT, and GPT-2, with isotonic regression giving ELMo RMSE $(y,F)$6 and Pearson $(y,F)$7, compared with BERT RMSE $(y,F)$8, GPT RMSE $(y,F)$9, and GPT-2 RMSE $y\in\{0,1,2\}$0. The analysis attributed poorer transformer results partly to domain mismatch, very small similarity magnitudes, the simplicity of a single-feature setup, and the suitability of isotonic regression for human grading behavior (Gaddipati et al., 2020).

4. LLM-based grading, prompting, and retrieval grounding

The LLM phase of ASAG did not eliminate the distinction between general-purpose and task-adapted systems. Zero-shot GPT-4 on SciEntsBank achieved weighted F1 $y\in\{0,1,2\}$1 in the 2-way setting and $y\in\{0,1,2\}$2 in the 3-way setting, broadly comparable to early hand-engineered systems, but on the more technical Beetle corpus it dropped to $y\in\{0,1,2\}$3 and $y\in\{0,1,2\}$4, respectively, and lagged behind fine-tuned transformer baselines such as roberta-large. Withholding the reference answer had only minor impact on SciEntsBank and improved GPT-4’s Beetle performance in the reported setup, indicating that reference-answer dependence is not uniform across domains (Kortemeyer, 2023).

Prompt-based adaptation with graded examples substantially improved LLM grading. In the few-shot grading pipeline of Kumar et al., the model prompt included general instructions, the question, five human-graded examples chosen by no-example, random, or retrieval-augmented strategies, and optionally a rubric. On the Texas dataset, GPT-4o improved from Pearson $y\in\{0,1,2\}$5, RMSE $y\in\{0,1,2\}$6 with no examples to $y\in\{0,1,2\}$7, RMSE $y\in\{0,1,2\}$8 with random examples and $y\in\{0,1,2\}$9, RMSE $F$0 with RAG-based example selection. GPT-4o was reported as the best balance between accuracy and cost-effectiveness, while o1-preview had higher single-run accuracy but larger error variance, greater latency, and higher cost (Zhao et al., 18 Feb 2025).

Retrieval itself has become more structural. GraphRAG replaces flat vector retrieval with explicit knowledge graphs built from rubric texts, exemplar answers, and conceptual definitions, then retrieves connected subgraphs rather than isolated chunks. On an NGSS benchmark, HippoRAG achieved average accuracies of $F$1 on DCI, $F$2 on SEP, and $F$3 on CCC, compared with $F$4, $F$5, and $F$6 for flat RAG. The largest jump occurred on SEP, the dimension most directly associated with multi-hop reasoning, which rose from $F$7 to $F$8 (Chu et al., 28 Feb 2026).

A plausible implication is that LLM-based ASAG increasingly depends less on generic semantic matching alone and more on the quality of conditioning: graded examples, rubric items, graph-grounded evidence, and prompt structures that expose the decision boundary explicitly. This interpretation is reinforced by reports that including the question in the input improves both label accuracy and feedback quality, and that rubric-augmented prompts improve structured tasks such as proof grading (Aggarwal et al., 2024, Zhao et al., 18 Feb 2025).

5. Explainability, feedback, and rubric engineering

One prominent branch of recent ASAG research extends grading into explanation. An example is the one-shot prompting plus SBERT pipeline for short-answer items with multiple sub-questions. It first extracts “justification keys” as exact text spans using one-shot prompting with GPT-3.5, then scores each extracted span against reference answers with a domain-adapted SBERT bi-encoder, yielding binary analytic scores that sum to a holistic score. On a subset of ASAP-SAS, the full pipeline reached accuracy $F$9, QWK $s_i\in\{0,1\}$0, and Pearson $s_i\in\{0,1\}$1, while justification-key extraction matched gold spans by overlap in $s_i\in\{0,1\}$2 of cases. The authors emphasized interpretability and actionable feedback rather than only holistic prediction (Yoon, 2023).

EngSAF generalizes this objective by pairing three-class grading with content-focused feedback. The dataset contains 119 questions and 5,814 student responses from 25 engineering courses, and feedback was synthesized through Label-Aware Synthetic Feedback Generation, in which an LLM receives $s_i\in\{0,1\}$3 and produces feedback conditioned on the gold label. In the reported baselines, Mistral_w_quest achieved $s_i\in\{0,1\}$4 accuracy and $s_i\in\{0,1\}$5 F1 on unseen answers, but only $s_i\in\{0,1\}$6 accuracy and $s_i\in\{0,1\}$7 F1 on unseen questions. A real-world deployment in an IIT Bombay end-semester exam reported SME-evaluated label accuracy $s_i\in\{0,1\}$8, feedback quality/correctness $s_i\in\{0,1\}$9, and emotional impact $S=\sum_i s_i$0 (Aggarwal et al., 2024).

Explainability has also been pursued through neuro-symbolic design. The bilingual multi-domain pipeline of Camus et al. weakly supervised justification cues with programmatic labeling functions and HMM aggregation, detected cues with multilingual transformers, then graded via symbolic similarity to rubric elements followed by a decision tree or summation head. In the unseen-answer test split, the best span-prediction plus decision-tree configuration reduced RMSE by $S=\sum_i s_i$1 on German and $S=\sum_i s_i$2 on English relative to an mBART baseline, while also returning the text spans used to justify each rubric-item score (Künnecke et al., 2024).

Rubric engineering itself has become a target of optimization. GradeOpt introduced a multi-agent framework in which a grader, reflector, and refiner iteratively improve grading guidelines; on a five-question pedagogical-content-knowledge and content-knowledge task it reported average accuracy $S=\sum_i s_i$3 and Cohen’s $S=\sum_i s_i$4, outperforming SBERT+LR, RoBERTa fine-tuning, GPT-4o zero-shot, and APO. GradeHITL extended this logic by allowing the LLM to ask clarifying questions to human experts, validate the resulting Q&A pairs, and use RL-based retrieval of those Q&As during rubric refinement; across six questions it outperformed non-LLM and LLM baselines, with reported gains of approximately $S=\sum_i s_i$5–$S=\sum_i s_i$6 absolute over GradeOpt and lower variance (Chu et al., 2024, Li et al., 7 Apr 2025).

6. Reliability, auditing, uncertainty, and unresolved problems

A recurring misconception in ASAG is that high agreement with human labels is sufficient evidence of reliability. Condor and Pardos directly challenged this assumption by auditing a BERT-base five-class grader with deep reinforcement learning. The grader had held-out Quadratic Weighted Kappa $S=\sum_i s_i$7 and ROC-AUC $S=\sum_i s_i$8, yet a PPO agent learned short edit sequences that pushed low-quality responses above a high-score threshold, often by repeating “helpful” or even “unhelpful” phrases. The study exposed keyword-triggering and repetition exploits, showing that surface agreement metrics do not guarantee robustness (Condor et al., 2024).

Adversarial vulnerability persists in LLM-based ASAG. GradingAttack formalized token-level suffix attacks and prompt-level role-play attacks, and proposed the Camouflage Attack Score to balance attack success against preservation of native grading accuracy. Averaged across five ASAG benchmarks and seven open-source LLMs, token-level GradingAttack$S=\sum_i s_i$9 achieved ASR approximately $0$0 with CAS approximately $0$1, while prompt-level GradingAttack$0$2 achieved ASR approximately $0$3 with CAS approximately $0$4. The prompt-level method was more effective, whereas token-level attacks were more covert (Li et al., 1 Feb 2026).

Another limitation of standard evaluation is that aggregate metrics hide difficulty structure. The 2026 item-response-theoretic analysis modeled each LLM as a grader with latent ability and each student response as an item with latent difficulty, using a 1PL testlet model. Across 17 open-weight LLMs on SciEntsBank and Beetle, accuracy declined monotonically as difficulty increased, but the slopes varied sharply by model; for example, on the hardest SciEntsBank bin Gemma-3-12B retained $0$5 accuracy while Qwen3.5-9B fell to $0$6 and Llama-3.1-8B to $0$7. Errors on difficult items disproportionately collapsed into the \texttt{partially_correct_incomplete} label, indicating intermediate-label defaulting under ambiguity (Cong et al., 30 Apr 2026).

Confidence estimation addresses a complementary problem: deciding when not to automate. Cong et al. compared verbalized, latent, and consistency-based confidence signals for a zero-shot generative LLM grader and found that model-based signals alone were insufficient. They therefore fused them with an aleatoric uncertainty estimate derived from clustering semantically embedded responses and measuring within-cluster label entropy. On SciEntsBank Test_UD, the hybrid model with aleatoric uncertainty achieved the best selective-grading and calibration results, and at a $0$8 rejection rate the accuracy on the retained $0$9 of cases rose to approximately $5$0, compared with the baseline grading accuracy of $5$1 (Cong et al., 30 Apr 2026).

The remaining performance gap to human grading is therefore best understood as a systems problem rather than a single-model shortcoming. ASAG2024 showed that GPT-4o achieved the best mean weighted RMSE, $5$2, across seven normalized datasets, followed by GPT-3.5 at $5$3, yet on SAF the estimated human–human RMSE was approximately $5$4. Other large-scale evidence is more optimistic in narrowly defined settings: a fine-tuned encoder-only transformer trained on proprietary university exam data achieved normalized MAE $5$5 against official grades, compared with $5$6 for certified human re-graders, and its median absolute deviation was $5$7 smaller. Taken together, these results suggest that ASAG is already operationally useful in some controlled settings, but that unseen-question generalization, adversarial robustness, calibration, rubric adherence, and explanation quality remain core research problems (Meyer et al., 2024, Gobrecht et al., 2024).

Future work in the literature converges on a relatively stable agenda: better balanced and more diverse corpora, stronger handling of unseen questions and unseen domains, external knowledge integration, selective human review driven by difficulty or confidence, and evaluation protocols that go beyond single aggregate scores. This suggests that the next phase of ASAG will likely be defined less by isolated gains on a single benchmark than by robust, confidence-aware, rubric-grounded, human-in-the-loop grading pipelines that can explain and defend their decisions across domains (Aggarwal et al., 2024, Chu et al., 28 Feb 2026, Cong et al., 30 Apr 2026, Cong et al., 30 Apr 2026).