SCALEFeedback Synthetic Assignment Dataset

• The paper introduces SCALEFeedback by demonstrating a one-to-one LLM-based imitation method that preserves semantic meaning while fully protecting student privacy.
• SCALEFeedback couples assignment descriptions, rubrics, and synthetic student submissions, offering a structured benchmark for feedback quality and rubric-aware evaluation.
• Quantitative evaluations show that the SAM framework outperforms naïve mimicry with improved semantic similarity, length correlation, and mark precision.

Searching arXiv for papers on SCALEFeedback to ground the article in the relevant literature. SCALEFeedback denotes a large-scale open-source dataset of Synthetic Computer science Assignments for LLM-generated Educational Feedback research, introduced to support research on generalisable methodology for automatic generation of effective and responsible educational feedback when openly shareable assignment corpora are otherwise unavailable (Qian et al., 8 Aug 2025). The dataset was constructed through a one-to-one LLM-based imitation process that transforms real assignment descriptions and student submissions into synthetic counterparts while preserving semantic meaning and student data distribution and protecting student privacy and institutional copyright. Its reported scale is 10,000 synthetic student submissions spanning 155 assignments across 59 university-level computer science courses, and its evaluation centers on resemblance to the source corpus, feedback quality parity with real data, and privacy protection (Qian et al., 8 Aug 2025).

1. Dataset scope and record structure

SCALEFeedback contains 10,000 synthetic student submissions, 155 distinct assignments, and 59 university-level computer science courses (Qian et al., 8 Aug 2025). Each record has three main components: an Assignment Description, a Rubric, and a Student Submission. The assignment description is a textual prompt including title, theme, objectives, detailed rubric, reading materials, and supplementary instructions. The rubric consists of explicit marking criteria such as correctness, completeness, and style, embedded within the description. The student submission is the student’s solution text and may contain code snippets, explanations, SQL queries, and related material.

This composition is consequential for educational-feedback research because the dataset couples task specification, evaluative criteria, and response content in a single record. A plausible implication is that this structure makes the dataset suitable not only for feedback generation, but also for rubric-aware evaluation, grading-related studies, and prompt design experiments, although the paper states these uses more specifically as benchmarking, testing prompt-engineering and scoring methods, and enabling safe open-sourcing of sensitive educational data.

2. Sophisticated Assignment Mimicry (SAM)

The dataset is generated by the “Sophisticated Assignment Mimicry (SAM)” framework, which performs one-to-one synthetic data generation via LLM imitation in two phases (Qian et al., 8 Aug 2025). In Phase I, the input is a real assignment description. The in-prompt procedure has four steps: evaluate the original assignment on theme, objectives, style, length, and rubric details; generate a candidate synthetic description imitating those dimensions; evaluate the candidate on the same dimensions; and compare candidate and original, looping back if mismatch remains. The output is a synthetic assignment description.

Phase II operates on the tuple consisting of the real assignment description, the synthetic assignment description, and the real student submission. It uses the same four-step mimicry pattern, but the evaluation criteria for submissions explicitly include the extent, detail, and correctness of the answer, the mark awarded on a $0$–$100$ scale according to the original rubric, and the word count of the real submission. This phase is followed by an external privacy gate. The gate generates a synthetic submission, uses a separate LLM, “o4-mini,” to extract student name or ID from real and synthetic texts, rejects the candidate if names or IDs match, and otherwise accepts it.

The paper’s pseudo-code formalizes SAM as a repeated synthesis-and-judge loop for assignment descriptions and a repeated synthesis-and-privacy-check loop for student submissions. This architecture suggests that SCALEFeedback is not merely a batch paraphrasing pipeline: its central design principle is controlled imitation under explicit similarity and privacy constraints.

3. Corpus resemblance and quantitative evaluation

The empirical resemblance analysis compares SAM with a naïve mimicry baseline using BERTScore F1, Pearson Correlation Coefficient (PCC), and mean absolute error (MAE) on assignment descriptions and student submissions (Qian et al., 8 Aug 2025). For synthetic text $X$ and reference text $Y$, the paper defines

$$P = \frac{1}{|X|}\sum_{i=1}^{|X|}\max_j \cos(h_i^X, h_j^Y), \qquad R = \frac{1}{|Y|}\sum_{j=1}^{|Y|}\max_i \cos(h_i^X, h_j^Y),$$

and then

$\mathrm{BERTScore\ F1} = \frac{2PR}{P + R}.$

For paired variables $\{x_i\}$ and $\{y_i\}$, PCC is

$$\rho_{x,y} = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)} {\sqrt{\sum_i (x_i - \bar x)^2\,\sum_i (y_i - \bar y)^2}}.$$

PCC is applied to submission length and assignment marks, where marks are on a $0$–$100$0 scale and averaged across three graders: o3-high, o4-mini, and GPT-4.1.

Item	Metric	Naïve / SAM
Assignment Descriptions	BERTScore F1	0.875 / 0.859
Assignment Descriptions	PCC (length)	0.656 / 0.931
Assignment Descriptions	MAE (words)	1,018.21 / 586.65
Student Submissions	BERTScore F1	0.819 / 0.840
Student Submissions	PCC (length)	0.598 / 0.852
Student Submissions	MAE (words)	524.86 / 335.43
Student Submissions	PCC (marks)	0.421 / 0.624
Student Submissions	MAE (marks)	26.09 / 19.98

The paper’s discussion emphasizes three points. First, both methods achieve high semantic similarity, with $100$1. Second, SAM strongly outperforms naïve mimicry on length correlation, with PCC approximately $100$2–$100$3 rather than approximately $100$4, and reduces MAE by more than $100$5. Third, for marks, SAM yields moderate correlation, $100$6 versus $100$7, and lower MAE, although absolute errors remain substantial. In the abstract, the reported headline results for synthetic submissions are BERTScore F1 $100$8, PCC $100$9 for assignment marks, and $X$0 for length, relative to the corresponding real-world assignment dataset (Qian et al., 8 Aug 2025).

4. Feedback quality assessment

A central validation question is whether LLM-generated feedback on the synthetic dataset behaves like feedback on the original data. The study evaluates this by having 10 commercial LLMs from OpenAI, Google Gemini, and Deepseek each generate 100 feedback instances on real submissions and 100 on the corresponding synthetic submissions, yielding 2,000 feedback samples in total (Qian et al., 8 Aug 2025).

Feedback is assessed along 16 dimensions grouped into three categories. The content category uses 3-point Likert scales for Alignment with goals, Specificity, Motivational Tone, Strengths, and Weaknesses. The effectiveness category uses binary indicators for Feed forward, Feed up, Feed back, and for task, process, self-regulation, and self levels. The hallucination category uses binary indicators for context-conflicting, input-conflicting, and fact-conflicting hallucinations.

The reported result is that mean feedback quality is nearly identical for real and synthetic data, with a maximum difference of $X$1. The standard deviation is slightly larger on real data, with $X$2 for the content dimension, which the paper attributes to more varied real submissions. The stated conclusion is that LLM-generated feedback on SCALEFeedback matches the effectiveness of feedback on original data (Qian et al., 8 Aug 2025). This narrows an important concern in educational-feedback research: synthetic assignment data may differ from real submissions in surface form, but the paper reports parity at the level of generated feedback effectiveness.

5. Privacy protection and ethical framing

SCALEFeedback is explicitly framed around privacy and copyright-safe release of educational data. The privacy design is two-tiered (Qian et al., 8 Aug 2025). The first tier is in-prompt abstraction: the prompt instructions explicitly prevent copying personal identifiers. The second tier is the external LLM-based privacy gate, which checks equality of names or IDs between the real and synthetic submissions and rejects any candidate that leaks real student information.

The ablation on privacy “protection rate” reports the following values: naïve mimicry with no protection gives $X$3; SAM with prompt only gives $X$4; SAM with gate only gives $X$5; and SAM with both gives $X$6 (Qian et al., 8 Aug 2025). The paper therefore states that SAM fully protects student privacy and removes institutional copyright concerns.

The ethical significance of this result is direct. The paper identifies the absence of large-scale open-source assignment datasets as a limiting factor for research on LLM-generated educational feedback. SCALEFeedback addresses that constraint by proposing one-to-one LLM imitation as a route to preserve semantic meaning and student data distribution while ensuring perfect protection of student private information. This suggests a broader template for privacy-preserving release of sensitive educational corpora, provided the same protections and evaluations are retained.

6. Research uses, limits, and future directions

The paper lists three principal uses for SCALEFeedback (Qian et al., 8 Aug 2025). It is a benchmark for developing generalisable LLM-based educational feedback systems across varied computer science assignments; a testbed for new prompt-engineering, scoring, and feedback evaluation methods; and a template for institutions to safely open-source sensitive educational data via one-to-one LLM synthesis.

At the same time, the paper identifies several limitations. Synthetic submissions cluster around the mean and fail to reproduce the long tail of very long or very high-scoring responses. BERTScore F1 may lack sensitivity to thematic or style differences, motivating more advanced metrics such as LLM-as-Judge. In addition, although feedback quality is nearly identical between real and synthetic conditions, the resemblance results for marks remain only moderate, and absolute errors remain substantial.

The future-work agenda follows directly from these limits. The paper proposes integrating Mixture-of-Personas prompting to enrich student diversity, developing comparative LLM-based evaluation metrics for synthetic data, and extending SAM beyond computer science to mathematics, humanities, and other disciplines in order to create broader open educational data ecosystems (Qian et al., 8 Aug 2025). This suggests that SCALEFeedback is intended less as a completed endpoint than as an infrastructure layer for a wider program of synthetic educational-data research.

7. Terminological ambiguity

The term “SCALEFeedback” is not unique to the educational-feedback dataset. In “Scaling-aware rating of count forecasts” (Tichy et al., 2022), the same label is used for a scaling-aware forecast rating method for count-forecast evaluation rather than for a dataset of computer science assignments. In that work, “SCALEFeedback” refers to a procedure that accounts for irreducible Poisson noise, avoids the “naïve scaling trap,” and produces diagnostic plots and a single interpretable score for fair cross-segment benchmarking (Tichy et al., 2022).

For disambiguation, the name most commonly denotes the educational dataset only in the sense introduced by “SCALEFeedback: A Large-Scale Dataset of Synthetic Computer Science Assignments for LLM-generated Educational Feedback Research” (Qian et al., 8 Aug 2025). In contexts where both literatures may be visible, precise citation is therefore necessary: the dataset and SAM framework belong to the AI in Education setting, whereas the scaling-aware rating method belongs to count-forecast evaluation.