SPRAG: Python ASAG Dataset
- SPRAG is a dataset designed for automatic short answer grading in Python, pairing student answers with reference solutions and numeric similarity scores.
- It benchmarks the transferability of similarity models by comparing performance across domain-specific datasets like Mohler and general texts like STSB.
- The dataset’s structure, including balanced label distributions and programming-specific content, offers actionable insights for educational NLP and domain adaptation research.
Searching arXiv for the cited SPRAG-related paper and closely related short answer grading work. SPRAG is a recently introduced dataset specifically designed for automatic short answer grading (ASAG) in the domain of Python programming. It is formulated as a sentence-pair dataset with numeric similarity labels from 0 to 5, where one sentence typically corresponds to a student-authored short answer and the other to a reference or model answer. In the comparative framework of "Statistical Comparative Analysis of Semantic Similarities and Model Transferability Across Datasets for Short Answer Grading," SPRAG serves as the “unexplored domain” against which the transferability of similarity-based models trained on established datasets is examined (Bonthu et al., 19 Aug 2025).
1. Purpose and position within ASAG research
SPRAG is introduced as a novel ASAG dataset intended to address the lack of diverse, domain-specific resources for automated grading of short answers. The comparative study that foregrounds SPRAG organizes three datasets into a single analytical framework: STSB as an established general-purpose semantic textual similarity benchmark, Mohler as a well-known short answer grading dataset in natural language, and SPRAG as a new ASAG dataset centered on Python programming (Bonthu et al., 19 Aug 2025).
Within that framework, SPRAG functions as the target domain for a transferability question: whether state-of-the-art (SOTA) similarity-based models and techniques trained on existing datasets, especially Mohler and STSB, can be applied to a new, domain-specific dataset without fully dataset-specific training. This places SPRAG at the intersection of educational NLP, semantic textual similarity, and domain adaptation.
A common misunderstanding is to treat SPRAG as a general semantic similarity benchmark analogous to STSB. The comparative analysis does not support that characterization. SPRAG is instead positioned as a more programming-specific counterpart to Mohler and as a domain-shifted target relative to STSB. This suggests that its primary significance lies not merely in providing additional sentence pairs, but in testing how far existing ASAG and similarity models can be extended into code-related short answers.
2. Dataset structure, labels, and annotation semantics
SPRAG belongs to a class of short answer grading datasets in which each dataset comprises pairs of sentences along with a numerical label indicating their similarity, ranging from 0 (least similar) to 5 (most similar) (Bonthu et al., 19 Aug 2025). In ASAG usage, the sentence pair typically instantiates a student answer and a reference answer, and the label operates as a semantic similarity / grading score. The labels therefore function both as similarity judgments and as practical proxies for correctness.
In terms of content, SPRAG contains question prompts implicitly through its ASAG design, reference answers, student answers, and grading labels on the $0$–$5$ scale. The paper does not reproduce SPRAG’s original annotation rubric, but it explicitly treats the labels as gold similarity scores used for similarity comparisons and for examining whether metric outputs track the gold scores.
The scoring scheme is especially important because it connects semantic similarity to grading behavior. Higher scores are used as stronger alignment with the reference answer, while lower scores indicate weaker alignment. The paper does not formalize this further as an error-minimization problem; it does not fit predictive models to SPRAG in this study, and consequently does not report metrics such as MSE or RMSE. Instead, the labels are used as the ground truth for correlation-oriented and interpretive comparison of similarity metrics.
Specific preprocessing steps are not enumerated. The methodology includes an analysis and pre-processing stage, and the dataset is processed in a way that supports TF-IDF cosine similarity, Word Mover’s Distance (WMD), and contextual encoders such as USE, SBERT, and SimCSE. Because SPRAG includes Python keywords and symbols, it is plausible that tokenization must preserve such items, but the paper does not specify the exact preprocessing pipeline.
3. Domain specificity and linguistic profile
SPRAG is explicitly described as “a recently developed dataset centered around the domain of Python programming” (Bonthu et al., 19 Aug 2025). Its defining property is therefore not only that it addresses short answer grading, but that it does so in a domain where answers include Python concepts, code behavior, and programming constructs. The paper highlights tokens such as def, del, and elif, as well as symbols including *, &, and #, to distinguish SPRAG from datasets written in ordinary natural English.
The dataset’s linguistic surface reflects this specialization. The vocabulary analysis reports that SPRAG’s top words are strongly programming-specific, with examples including keyword, return, code, variable, argument, operator, list, program, python, and statement. This contrasts with STSB, whose common words are general and non-domain, and with Mohler, which mixes CS-related and general terminology.
The sentence-length analysis likewise places SPRAG closer to Mohler than to STSB. STSB sentences are shorter on average, whereas SPRAG and Mohler have longer sentences, a pattern the paper associates with more technical wording, detailed explanations, or code-like constructs. This matters for ASAG because longer sentences with mixed natural-language and programming content create a more challenging representational regime for lexical similarity measures and for sentence encoders trained primarily on general English.
SPRAG’s label distribution also exhibits a characteristic internal structure. Label 5 covers a substantial portion of records, but the distribution of the remaining labels 0–4 is more balanced than Mohler. The paper interprets this as consistent with a dataset that contains many fully correct or near-correct programming answers while still preserving meaningful variation in partially correct and incorrect responses. This suggests a grading environment in which both binary correctness and finer semantic distinctions are relevant.
4. Comparative relation to STSB and Mohler
The comparative analysis in the paper situates SPRAG between a general semantic similarity benchmark and an established ASAG dataset, but clearly closer to the latter (Bonthu et al., 19 Aug 2025). In score-distribution terms, Mohler is described as highly imbalanced, with a majority of examples labeled 5, while STSB is characterized as well-balanced across similarity scores. SPRAG also contains many label-5 items, yet remains less skewed than Mohler because the other labels are more evenly represented.
Vocabulary overlap reinforces this positioning. The paper reports that Mohler–SPRAG overlap accounts for about 20% of the top-20 words, whereas STSB–SPRAG shows no overlap among top-20 words, and STSB–Mohler also shows no overlap among top-20 words. This places SPRAG and Mohler in a shared technical-educational lexical region, while isolating STSB as a general-language resource with limited direct vocabulary continuity.
The same pattern appears in the statistical and semantic comparison of dataset pairs. The study computes average similarity scores across (STSB, Mohler), (STSB, SPRAG), and (Mohler, SPRAG) using multiple metrics, and visualizes the results with a heatmap and Cohen’s effect sizes. The paper’s interpretation is explicit: there is “a minor resemblance between the STSB and Mohler datasets” and a “more pronounced similarity” between Mohler and SPRAG. The corresponding effect-size analysis confirms that SPRAG is semantically and statistically closer to Mohler than to STSB.
This comparative result is central to the dataset’s research function. SPRAG is not merely another ASAG corpus; it is the empirical basis for evaluating how semantic similarity models behave under domain shift from natural-language grading data to program-related text.
5. Similarity metrics and transferability analysis
The study evaluates SPRAG with both non-contextual and contextual similarity measures (Bonthu et al., 19 Aug 2025). Among the non-contextual methods are Jaccard similarity, TF-IDF cosine similarity, and Word Mover’s Distance. Jaccard similarity is defined over token sets and as
The statistical comparison also uses Cohen’s to quantify the magnitude of differences between datasets:
The contextual methods include Universal Sentence Encoder (USE), SBERT Cross Encoder (SBERT CE), SBERT Bi-Encoder (SBERT BiE), and SimCSE in both supervised and unsupervised forms. Their outputs are compared against the true SPRAG labels $0$–$5$ for interpretive analysis rather than for end-to-end predictive benchmarking.
The methodological pipeline is four-stage: data analysis and preprocessing for STSB, Mohler, and SPRAG; computation of semantic textual similarity using the non-contextual and contextual metrics; paired t-test and Cohen’s $5$0 analysis; and an interpretability step that compares metric-derived similarity values with the actual labels. Importantly, the metrics are applied to SPRAG without re-training specifically for SPRAG, so the experimental setup directly tests transferability.
The findings are mixed. The non-contextual metrics—Jaccard, TF-IDF, and WMD—perform well on Mohler but less satisfactorily on STSB and SPRAG. The contextual metrics—USE, SBERT CE, SBERT BiE, and SimCSE—show consistent performance across Mohler and STSB, but they “encountered challenges with the SPRAG dataset due to its intricate sentence structures, which hindered the identification of similarity.” The paper attributes this to domain shift associated with programming tokens, mixed natural language and code structure, and longer, more complex sentences.
Even so, the overall conclusion is not that transfer fails. Rather, the semantic and statistical analyses indicate that transfer from Mohler to SPRAG is promising, whereas transfer from STSB to SPRAG is less direct because of vocabulary and domain mismatch.
6. Research uses, limitations, and availability
SPRAG is directly usable as a short answer grading resource for Python/programming short answers (Bonthu et al., 19 Aug 2025). Its $5$1–$5$2 labels can serve as regression targets for models that predict numeric grades or as ordered categorical labels for classification-oriented grading. It is also suitable for benchmarking semantic similarity between student answers and reference solutions in a programming context.
For transfer learning research, SPRAG is especially valuable because it supports systematic comparison between general-text similarity modeling and domain-specific ASAG. The paper explicitly recommends leveraging SOTA models and techniques developed on Mohler for evaluation on SPRAG, given the demonstrated semantic and statistical proximity between the two datasets. This makes SPRAG a relevant benchmark for studies of cross-dataset model reuse, contextual versus non-contextual similarity, and domain shift in educational NLP.
Several limitations are also explicit. The paper does not provide exact counts of questions or pairs, although it establishes that SPRAG has enough instances for score distribution analysis, sentence length analysis, vocabulary comparison, and evaluation with models such as SBERT, USE, SimCSE, and WMD. It likewise does not specify exact file formats, licensing conditions, or a public repository URL. It refers to SPRAG as a recently developed dataset and cites the original introduction by Bonthu, Sree, and Prasad in Engineering Applications of Artificial Intelligence (2023), but operational details such as whether the released form is CSV, JSON, or another tabular format are not given in the paper itself.
A plausible implication is that SPRAG should be understood less as a fully standardized benchmark with exhaustively documented packaging and more as a domain-specific evaluation resource whose importance lies in what it reveals about the portability of short answer grading models. Within that role, the paper’s central result is clear: SPRAG is a Python-programming ASAG dataset whose semantic and statistical behavior is closer to Mohler than to STSB, making it a natural target for transfer-oriented ASAG research rather than for broad, domain-agnostic semantic textual similarity evaluation.
