---
title: 'SPRAG: Python ASAG Dataset'
url: https://www.emergentmind.com/topics/sprag-dataset
type: topic
---

# SPRAG: Python ASAG Dataset

Searching arXiv for the cited SPRAG-related paper and closely related short answer grading work.
SPRAG is a recently introduced dataset specifically designed for **automatic short answer grading (ASAG)** in the domain of **Python programming**. It is formulated as a sentence-pair dataset with **numeric similarity labels** from **0** to **5**, where one sentence typically corresponds to a **student-authored short answer** and the other to a **reference or model answer**. In the comparative framework of "Statistical Comparative Analysis of Semantic Similarities and Model Transferability Across Datasets for Short Answer Grading," SPRAG serves as the **“unexplored domain”** against which the transferability of similarity-based models trained on established datasets is examined [2508.15837].

## 1. Purpose and position within ASAG research

SPRAG is introduced as a **novel ASAG dataset** intended to address the lack of diverse, domain-specific resources for automated grading of short answers. The comparative study that foregrounds SPRAG organizes three datasets into a single analytical framework: **STSB** as an established general-purpose semantic textual similarity benchmark, **Mohler** as a well-known short answer grading dataset in natural language, and **SPRAG** as a new ASAG dataset centered on Python programming [2508.15837].

Within that framework, SPRAG functions as the target domain for a transferability question: whether **state-of-the-art (SOTA)** similarity-based models and techniques trained on existing datasets, especially Mohler and STSB, can be applied to a new, domain-specific dataset without fully dataset-specific training. This places SPRAG at the intersection of **educational NLP**, **semantic textual similarity**, and **domain adaptation**.

A common misunderstanding is to treat SPRAG as a general semantic similarity benchmark analogous to STSB. The comparative analysis does not support that characterization. SPRAG is instead positioned as a **more programming-specific counterpart** to Mohler and as a **domain-shifted target** relative to STSB. This suggests that its primary significance lies not merely in providing additional sentence pairs, but in testing how far existing ASAG and similarity models can be extended into **code-related short answers**.

## 2. Dataset structure, labels, and annotation semantics

SPRAG belongs to a class of short answer grading datasets in which **each dataset comprises pairs of sentences along with a numerical label indicating their similarity, ranging from 0 (least similar) to 5 (most similar)** [2508.15837]. In ASAG usage, the sentence pair typically instantiates a **student answer** and a **reference answer**, and the label operates as a **semantic similarity / grading score**. The labels therefore function both as similarity judgments and as practical proxies for correctness.

In terms of content, SPRAG contains **question prompts** implicitly through its ASAG design, **reference answers**, **student answers**, and **grading labels** on the \(0\)–\(5\) scale. The paper does not reproduce SPRAG’s original annotation rubric, but it explicitly treats the labels as **gold similarity scores** used for similarity comparisons and for examining whether metric outputs track the gold scores.

The scoring scheme is especially important because it connects semantic similarity to grading behavior. Higher scores are used as stronger alignment with the reference answer, while lower scores indicate weaker alignment. The paper does not formalize this further as an error-minimization problem; it does **not** fit predictive models to SPRAG in this study, and consequently does **not** report metrics such as MSE or RMSE. Instead, the labels are used as the ground truth for **correlation-oriented and interpretive comparison** of similarity metrics.

Specific preprocessing steps are not enumerated. The methodology includes an **analysis and pre-processing** stage, and the dataset is processed in a way that supports **TF-IDF cosine similarity**, **Word Mover’s Distance (WMD)**, and contextual encoders such as **USE**, **SBERT**, and **SimCSE**. Because SPRAG includes Python keywords and symbols, it is plausible that tokenization must preserve such items, but the paper does not specify the exact preprocessing pipeline.

## 3. Domain specificity and linguistic profile

SPRAG is explicitly described as **“a recently developed dataset centered around the domain of Python programming”** [2508.15837]. Its defining property is therefore not only that it addresses short answer grading, but that it does so in a domain where answers include **Python concepts, code behavior, and programming constructs**. The paper highlights tokens such as `def`, `del`, and `elif`, as well as symbols including `*`, `&`, and `#`, to distinguish SPRAG from datasets written in ordinary natural English.

The dataset’s linguistic surface reflects this specialization. The vocabulary analysis reports that SPRAG’s top words are strongly programming-specific, with examples including `keyword`, `return`, `code`, `variable`, `argument`, `operator`, `list`, `program`, `python`, and `statement`. This contrasts with STSB, whose common words are general and non-domain, and with Mohler, which mixes CS-related and general terminology.

The sentence-length analysis likewise places SPRAG closer to Mohler than to STSB. **STSB sentences are shorter on average**, whereas **SPRAG and Mohler have longer sentences**, a pattern the paper associates with more technical wording, detailed explanations, or code-like constructs. This matters for ASAG because longer sentences with mixed natural-language and programming content create a more challenging representational regime for lexical similarity measures and for sentence encoders trained primarily on general English.

SPRAG’s label distribution also exhibits a characteristic internal structure. Label **5** covers a **substantial portion** of records, but the distribution of the remaining labels **0–4** is **more balanced than Mohler**. The paper interprets this as consistent with a dataset that contains many fully correct or near-correct programming answers while still preserving meaningful variation in partially correct and incorrect responses. This suggests a grading environment in which both binary correctness and finer semantic distinctions are relevant.

## 4. Comparative relation to STSB and Mohler

The comparative analysis in the paper situates SPRAG between a general semantic similarity benchmark and an established ASAG dataset, but clearly closer to the latter [2508.15837]. In score-distribution terms, **Mohler** is described as highly imbalanced, with a majority of examples labeled **5**, while **STSB** is characterized as **well-balanced** across similarity scores. **SPRAG** also contains many label-5 items, yet remains less skewed than Mohler because the other labels are more evenly represented.

Vocabulary overlap reinforces this positioning. The paper reports that **Mohler–SPRAG overlap** accounts for about **20%** of the top-20 words, whereas **STSB–SPRAG** shows **no overlap** among top-20 words, and **STSB–Mohler** also shows no overlap among top-20 words. This places SPRAG and Mohler in a shared technical-educational lexical region, while isolating STSB as a general-language resource with limited direct vocabulary continuity.

The same pattern appears in the statistical and semantic comparison of dataset pairs. The study computes average similarity scores across **(STSB, Mohler)**, **(STSB, SPRAG)**, and **(Mohler, SPRAG)** using multiple metrics, and visualizes the results with a heatmap and **Cohen’s \(d\)** effect sizes. The paper’s interpretation is explicit: there is **“a minor resemblance between the STSB and Mohler datasets”** and a **“more pronounced similarity”** between **Mohler and SPRAG**. The corresponding effect-size analysis confirms that **SPRAG is semantically and statistically closer to Mohler than to STSB**.

This comparative result is central to the dataset’s research function. SPRAG is not merely another ASAG corpus; it is the empirical basis for evaluating how semantic similarity models behave under **domain shift** from natural-language grading data to **program-related text**.

## 5. Similarity metrics and transferability analysis

The study evaluates SPRAG with both **non-contextual** and **contextual** similarity measures [2508.15837]. Among the non-contextual methods are **Jaccard similarity**, **TF-IDF cosine similarity**, and **Word Mover’s Distance**. Jaccard similarity is defined over token sets \(S_1\) and \(S_2\) as

$$
J(S1,S2) = \frac{|S1 \cap S2|}{|S1 \cup S2|}.
$$

The statistical comparison also uses **Cohen’s \(d\)** to quantify the magnitude of differences between datasets:

$$
cohen's\ d = \frac{Mean1 - Mean2}{Pooled\ Standard\ Deviation}.
$$

The contextual methods include **Universal Sentence Encoder (USE)**, **SBERT Cross Encoder (SBERT CE)**, **SBERT Bi-Encoder (SBERT BiE)**, and **SimCSE** in both supervised and unsupervised forms. Their outputs are compared against the true SPRAG labels \(0\)–\(5\) for interpretive analysis rather than for end-to-end predictive benchmarking.

The methodological pipeline is four-stage: **data analysis and preprocessing** for STSB, Mohler, and SPRAG; computation of semantic textual similarity using the non-contextual and contextual metrics; **paired t-test** and **Cohen’s \(d\)** analysis; and an interpretability step that compares metric-derived similarity values with the actual labels. Importantly, the metrics are applied to SPRAG **without re-training specifically for SPRAG**, so the experimental setup directly tests transferability.

The findings are mixed. The non-contextual metrics—**Jaccard**, **TF-IDF**, and **WMD**—perform **well on Mohler** but **less satisfactorily on STSB and SPRAG**. The contextual metrics—**USE**, **SBERT CE**, **SBERT BiE**, and **SimCSE**—show **consistent performance across Mohler and STSB**, but they **“encountered challenges with the SPRAG dataset due to its intricate sentence structures, which hindered the identification of similarity.”** The paper attributes this to domain shift associated with **programming tokens**, **mixed natural language and code structure**, and **longer, more complex sentences**.

Even so, the overall conclusion is not that transfer fails. Rather, the semantic and statistical analyses indicate that **transfer from Mohler to SPRAG is promising**, whereas transfer from STSB to SPRAG is less direct because of vocabulary and domain mismatch.

## 6. Research uses, limitations, and availability

SPRAG is directly usable as a short answer grading resource for **Python/programming short answers** [2508.15837]. Its \(0\)–\(5\) labels can serve as **regression targets** for models that predict numeric grades or as **ordered categorical labels** for classification-oriented grading. It is also suitable for benchmarking **semantic similarity between student answers and reference solutions** in a programming context.

For transfer learning research, SPRAG is especially valuable because it supports systematic comparison between general-text similarity modeling and domain-specific ASAG. The paper explicitly recommends leveraging **SOTA models and techniques developed on Mohler** for evaluation on SPRAG, given the demonstrated semantic and statistical proximity between the two datasets. This makes SPRAG a relevant benchmark for studies of **cross-dataset model reuse**, **contextual versus non-contextual similarity**, and **domain shift in educational NLP**.

Several limitations are also explicit. The paper does **not** provide exact counts of questions or pairs, although it establishes that SPRAG has enough instances for **score distribution analysis**, **sentence length analysis**, **vocabulary comparison**, and evaluation with models such as **SBERT**, **USE**, **SimCSE**, and **WMD**. It likewise does **not** specify exact file formats, licensing conditions, or a public repository URL. It refers to SPRAG as a recently developed dataset and cites the original introduction by Bonthu, Sree, and Prasad in *Engineering Applications of Artificial Intelligence* (2023), but operational details such as whether the released form is CSV, JSON, or another tabular format are not given in the paper itself.

A plausible implication is that SPRAG should be understood less as a fully standardized benchmark with exhaustively documented packaging and more as a **domain-specific evaluation resource** whose importance lies in what it reveals about the portability of short answer grading models. Within that role, the paper’s central result is clear: SPRAG is a **Python-programming ASAG dataset** whose semantic and statistical behavior is **closer to Mohler than to STSB**, making it a natural target for transfer-oriented ASAG research rather than for broad, domain-agnostic semantic textual similarity evaluation.

Source: https://www.emergentmind.com/topics/sprag-dataset