---
title: 'Mohler Dataset: ASAG Benchmark'
url: https://www.emergentmind.com/topics/mohler-dataset
type: topic
---

# Mohler Dataset: ASAG Benchmark

The Mohler dataset is a long-standing benchmark for Automatic Short Answer Grading (ASAG), centered on student–reference answer pairs and human-assigned scores intended to operationalize semantic agreement between a student response and a model answer. In contemporary work it functions both as an intrinsic semantic-similarity benchmark and as an evaluation set for grading systems, including large language model (LLM) graders and sentence-encoder transfer studies [2508.15837; 2405.19694]. Across these uses, the dataset is consistently treated as a public ASAG resource, but its reported scale, domain characterization, and evaluation protocol vary with the study in which it is embedded.

## 1. Benchmark identity and reported corpus characteristics

In "Statistical Comparative Analysis of Semantic Similarities and Model Transferability Across Datasets for Short Answer Grading" [2508.15837], the Mohler dataset is described as one of two established benchmarks selected for comparison with the recently introduced SPRAG dataset. It is said to have been originally introduced for ASAG and to consist of paired student–reference sentences in natural English, with each pair annotated by an integer similarity score $y \in \{0,1,2,3,4,5\}$, where $0$ indicates no similarity and $5$ indicates highest semantic overlap. That study reports an approximate size of 1,000–1,200 sentence pairs, notes that the exact cardinality is not stated there, and characterizes the material as drawn from classroom environments in which student responses are compared against model answers [2508.15837].

In "Grade Like a Human: Rethinking Automated Assessment with Large Language Models" [2405.19694], the same dataset is described more concretely as containing 2,273 student answers spanning 80 distinct questions, collected from 31 learners working on 10 assignments and 2 exams. That paper also states that the questions are in the domain of undergraduate computer science, provides an example question, and notes that each answer is associated with a question ID, question text, student response, and model or desired answer [2405.19694].

These descriptions are not identical. One study emphasizes paired student–reference sentences in natural English and describes the domain as general-English short answer questions; the other emphasizes undergraduate computer science assessment artifacts and a richer record structure. A plausible implication is that later work may be referring to different processed views or task-specific subsets of the same benchmark.

| Aspect | Report in [2508.15837] | Report in [2405.19694] |
|---|---|---|
| Size | Approximately 1,000–1,200 sentence pairs | 2,273 student answers, 80 questions |
| Domain | General-English short answer questions | Undergraduate computer science |
| Core unit | Student–reference sentence pair | Question plus student answer plus model answer |

## 2. Annotation scheme, score semantics, and class distribution

The score space is central to how the Mohler dataset is operationalized. In the transferability study, each student–reference pair is annotated with an integer similarity score $y \in \{0,1,2,3,4,5\}$, explicitly interpreted as an ordinal-to-interval semantic similarity scale from no similarity to highest semantic overlap [2508.15837]. In the LLM-grading study, assignment questions were scored 0–5 by two human graders, whereas exam questions were scored 0–10 and then linearly normalized to the 0–5 range; the resulting ground truth is represented as two independent human grades per answer with scores in $\{0,\dots,5\}$ [2405.19694].

A notable property reported in the comparative analysis is severe label imbalance: the dataset is heavily skewed toward high-similarity labels, with over 60% of pairs annotated as 5. That same paper lists this skew as a limitation for ASAG, because label imbalance at the high-similarity end risks overfitting in direct regression models [2508.15837]. In the LLM-grading setting, the absence of fine-grained rubric annotations is also important: no explicit subcriteria are supplied in the original Mohler release, only the standard answer [2405.19694].

Taken together, these reports position the dataset as a benchmark where score interpretation is relatively simple, but supervision granularity is limited. This suggests that Mohler is well suited to studying semantic alignment and coarse grading fidelity, while being less directly informative for rubric-level error analysis.

## 3. Text properties and preprocessing conventions

The preprocessing choices reported for Mohler in the transferability study are straightforward and oriented toward cross-dataset similarity computation. All sentences were tokenized and lowercased. Stop-word removal was performed prior to calculating non-contextual set-based metrics such as Jaccard. TF–IDF vectors were constructed over the corpus of all three datasets used in the comparative experiments, and no additional cleaning such as spelling correction or syntactic normalization is described [2508.15837].

That study also reports descriptive corpus statistics. Density plots of character lengths show a mean length of roughly 50–60 characters per sentence, and a top-20 word-frequency list is said to be dominated by terms such as "function," "array," and "value" [2508.15837]. This vocabulary profile sits somewhat awkwardly beside the same paper’s description of the domain as general-English short answer questions rather than code or specialized technical text. By contrast, the LLM-grading paper explicitly places the dataset in undergraduate computer science [2405.19694]. The tension between these descriptions is itself informative: the benchmark appears to contain short answers whose linguistic surface can be treated as natural English while still reflecting discipline-specific classroom content.

The comparative paper further states that vocabulary overlaps 20% with SPRAG but practically none with STSB [2508.15837]. Within that study, this lexical fact is used to motivate subsequent behavior of both non-contextual and contextual similarity functions.

## 4. Similarity metrics and statistical analysis

A major contemporary use of Mohler is as a substrate for measuring whether semantic encoders transfer across datasets. The comparative study applies seven semantic-similarity measures to each pair of Mohler sentences and compares their behavior across Mohler, STSB, and SPRAG [2508.15837].

The non-contextual metrics are Jaccard similarity,
$$
J(S_1,S_2)=\frac{|S_1 \cap S_2|}{|S_1 \cup S_2|},
$$
TF–IDF cosine similarity, with
$$
w_{ij}=tf_{ij}\cdot \log\!\left(\frac{N}{df_i}\right),
$$
and cosine
$$
\mathrm{Cosine}(S_1,S_2)=\frac{S_1 \cdot S_2}{\|S_1\|\cdot \|S_2\|},
$$
and Word Mover’s Distance (WMD), where the transport cost is defined as $c(i,j)=\|x_i-x_j\|_2$ between word embeddings $x_i$ and $x_j$ [2508.15837].

The contextual metrics are Universal Sentence Encoder (USE), described there as transformer-based and yielding a fixed 512-dim embedding per sentence with cosine similarity; SBERT Cross-Encoder (CE), which jointly encodes the pair $(s_1,s_2)$; SBERT Bi-Encoder (BiE), which computes cosine between independent embeddings; and supervised and unsupervised SimCSE sentence embeddings [2508.15837].

For inferential comparison, the study uses paired Student’s $t$-tests under the null hypothesis of no mean difference in similarity scores, with
$$
t=\frac{\sum d}{\sqrt{n\cdot \sum d^2-(\sum d)^2}},
$$
where
$$
d_i = \mathrm{metric}(s_{1i},s_{2i})|_{\mathrm{Dataset\ A}}-\mathrm{metric}(s_{1i},s_{2i})|_{\mathrm{Dataset\ B}},
$$
and Cohen’s $d$,
$$
d=\frac{\mu_1-\mu_2}{\sigma_{\mathrm{pooled}}},
$$
using the interpretation Small: $|d|\approx 0.2$, Medium: $|d|\approx 0.5$, Large: $|d|\approx 0.8$ [2508.15837].

This methodological framing is important because Mohler is not only treated as a grading benchmark; it is also treated as a semantic distribution whose proximity to other datasets can be quantified independent of task-specific model retraining.

## 5. Evaluation protocols: encoder transfer and LLM grading

Two distinct experimental paradigms are reported.

In the transferability study, no new predictive classifiers are trained on Mohler. Instead, the paper evaluates state-of-the-art encoders by computing sentence-pair similarities on Mohler, with all encoders kept at their pre-trained weights and no further fine-tuning on Mohler reported [2508.15837]. The focus is therefore on representational adequacy and dataset affinity rather than supervised ASAG optimization.

In the LLM-grading study, the protocol is explicitly grading-oriented rather than similarity-oriented. There is no train–test split in the conventional sense; all 2,273 answers are run through the LLM grader guided either by the original standard answer alone or by a refined rubric [2405.19694]. The paper formalizes rubric refinement as an iterative optimization process. Let $q$ be a question, $r_0$ the initial coarse rubric, and $A$ the set of all student answers. At iteration $i$, a small sample $A_m \subset A$ is human-graded to yield $\{(a_j,g_j)\}$, after which the LLM produces
$$
r_{i+1}=O\bigl(r_i,\{(a_j,g_j)\},p\bigr),
$$
where $O$ is the OptimizeRubric operator and $p$ the system prompt [2405.19694].

The coarse-grained rubric condition simply uses the original standard answer from Mohler as the sole guidance. The fine-grained rubric condition is generated by the authors’ LLM-based rubric optimization loop even though Mohler itself supplies no explicit subcriteria [2405.19694]. This distinction is methodologically significant because it turns Mohler into a test case for whether additional rubric structure improves or degrades grading behavior.

## 6. Quantitative findings and interpretation

The comparative study reports mean similarity values for each metric across dataset pairs. For Mohler–SPRAG, the reported means are $1.09 \times 10^{-1}$ for Jaccard, $6.89 \times 10^{-5}$ for TF–IDF cosine, $9.55 \times 10^{-5}$ for WMD, $5.34 \times 10^{-1}$ for USE, $4.30 \times 10^{-8}$ for SBERT CE, $4.41 \times 10^{-1}$ for SBERT BiE, $1.40 \times 10^{-8}$ for SimCSE supervised, and $9.55 \times 10^{-5}$ for SimCSE unsupervised [2508.15837]. The same paper states that non-contextual metrics yield vanishingly small mean similarities on STSB–Mohler and STSB–SPRAG, but are one order of magnitude larger for Mohler–SPRAG, while contextual metrics remain stable across Mohler and achieve mean scores 0.4–0.5 on Mohler–SPRAG pairs [2508.15837].

For effect size, the reported Cohen’s $d$ values for Mohler vs. SPRAG are $+0.040$ for Jaccard, $+0.228$ for TF–IDF, $+0.021$ for WMD, $+0.012$ for USE, $+0.283$ for SBERT CE, $+0.341$ for SBERT BiE, $+0.295$ for SimCSE supervised, and $+0.283$ for SimCSE unsupervised [2508.15837]. The paper interprets the contextual metrics as yielding small-to-medium positive effect sizes and states that paired $t$-tests with $p < 0.05$ confirm that Mohler–SPRAG similarity is significantly higher than STSB–Mohler [2508.15837].

The LLM-grading paper reports a different pattern. On the Mohler questions it evaluates, fine-grained rubrics generated by the LLM hurt rather than help grading quality. The paper gives the example that on Q1.1, MAE rose from 0.78 to 1.67 and Pearson correlation fell from 0.69 to 0.03 when moving from coarse-grained to fine-grained rubric guidance [2405.19694]. It states that the coarse rubric, relying on the standard answer and the model’s world knowledge, consistently outperforms the heavily constrained fine rubric on every Mohler question tested [2405.19694].

The coexistence of these findings is not contradictory. One concerns encoder-level semantic transfer across datasets; the other concerns rubric-mediated LLM scoring. A plausible implication is that Mohler is relatively hospitable to pre-trained semantic encoders while being less hospitable to over-specified rubric decomposition for straightforward fact-based questions.

## 7. Limitations, controversies, and continuing relevance

Two limitations are explicit in the comparative study: the dataset’s skew toward label 5 and its limited size. That paper also states that narrow domain vocabulary may limit generalization to technical or code-heavy responses, even though the reported frequent terms include "function," "array," and "value" [2508.15837]. In the LLM-grading study, the principal limitation is supervisory granularity: Mohler provides no fine-grained rubric, only the standard answer, which constrains analyses of rubric adherence and post-grading review [2405.19694].

A second issue is interpretive heterogeneity. One paper describes Mohler as general-English short answer questions rather than code or specialized technical text, whereas another describes all questions as belonging to undergraduate computer science [2508.15837; 2405.19694]. This discrepancy should not be ignored. It indicates that "the Mohler dataset" may be discussed at different levels of abstraction: as a natural-language semantic similarity resource, as a computer-science educational benchmark, or as both. For research design, this means that reported transferability or grading results should be read in light of the paper-specific representation of the corpus.

Despite these caveats, Mohler remains useful because it supports at least two lines of inquiry. First, it enables statistical comparison of semantic spaces, where contextual encoders such as SBERT, USE, and SimCSE are reported to transfer effectively to Mohler-like domains with minimal or no fine-tuning [2508.15837]. Second, it provides a public benchmark for probing how rubric design interacts with LLM grading, where coarse guidance may outperform fine-grained rubric expansion on simple questions [2405.19694]. Together, these results suggest that the dataset continues to serve as a compact but methodologically revealing testbed for ASAG research.

Source: https://www.emergentmind.com/topics/mohler-dataset