Sophisticated Assignment Mimicry (SAM)

• Sophisticated Assignment Mimicry (SAM) is a prompt-based, privacy-preserving framework that generates synthetic educational assignments by imitating real samples at scale.
• It employs a two-phase synthesis workflow that ensures semantic preservation, length distribution matching, and reproducible mark correlations between real and synthetic data.
• SAM integrates dual privacy controls and an evaluation loop, making its synthetic outputs highly effective for downstream automated feedback research.

Sophisticated Assignment Mimicry (SAM) is a prompt-based, multi-step LLM framework for generating synthetic educational assignment data from sensitive real-world coursework. In the SCALEFeedback study, SAM is defined as a one-to-one LLM-based imitation process that takes real assignment descriptions and student submissions and produces synthetic counterparts intended to preserve semantic meaning and student data distributions while protecting student privacy and institutional copyright. The associated open-source dataset contains 10,000 synthetic student submissions spanning 155 assignments across 59 university-level computer science courses (Qian et al., 8 Aug 2025).

1. Origin and problem setting

SAM was introduced in response to a specific data bottleneck in AI for education: although using LLMs to provide educational feedback has attracted substantial attention, there was stated to be no large-scale open-source dataset of student assignments containing detailed assignment descriptions, rubrics, and student submissions across multiple courses. The absence of such a resource was presented as a major constraint on research into generalisable methods for automatic generation of effective and responsible educational feedback. SAM addresses this gap by synthesizing open-source substitutes for sensitive educational records through controlled imitation rather than direct release of original materials (Qian et al., 8 Aug 2025).

Within this formulation, the central objective is not generic text generation but distribution-preserving substitution. The framework is designed to preserve assignment themes, structure, expected student performance, and length distributions while replacing the original text with newly generated content. This orientation distinguishes SAM from simpler paraphrastic or template-driven synthetic-data schemes: its target is correspondence at the level of assignment semantics and student-performance characteristics, rather than mere surface resemblance.

2. One-to-one synthesis workflow

The SAM pipeline operates in two main phases—assignment description synthesis and student submission synthesis—with an additional privacy protection layer. Let $D_{real}$ denote a real assignment description, $S_{real}$ a real student submission, $D_{syn}$ a synthetic assignment description, and $S_{syn}$ a synthetic student submission. The first phase takes the real assignment description, including rubrics, objectives, and materials, and prompts the LLM to analyze its theme, objectives, writing style, length, and rubric details. The model then generates a synthetic assignment description intended to imitate those evaluated characteristics. The generated description is itself evaluated, and the synthesis step is repeated if the match is not satisfactory (Qian et al., 8 Aug 2025).

The second phase conditions on three inputs simultaneously: the real student submission, the corresponding real assignment description, and the previously generated synthetic assignment description. The LLM first evaluates the real submission in terms of extent of answer, level of detail, correctness, an estimated mark per rubric, and number of words. It then synthesizes a new student submission for the synthetic assignment while matching the real submission’s level of detail, correctness, and length. A self-evaluation step follows, after which the synthetic submission is compared with the real one on the same criteria. If the match is insufficient, synthesis is repeated.

This evaluation–synthesis–self-evaluation loop is the operative core of SAM. The framework therefore uses explicit intermediate assessments to constrain generation, rather than relying on a single imitation prompt. A plausible implication is that the method treats semantic preservation and distribution matching as controllable objectives within generation, not merely as post hoc desiderata.

3. Privacy-preserving control structure

Privacy protection is implemented at two levels. The first is prompt-level protection: the generation prompts instruct the LLM to abstract from the source materials rather than copy private or identifying content. The second is an external privacy gate applied to each synthetic student submission. In this stage, a secondary LLM, specified as o4-mini with high reasoning, is asked through a 3-step prompt to detect whether any private information from the real submission—such as student names or IDs—has leaked into the synthetic output. If leakage is detected, the synthetic generation for that sample is repeated from scratch (Qian et al., 8 Aug 2025).

This dual mechanism is framed as addressing both privacy and copyright. The prompt-level abstraction encourages novel generation, while the external gate functions as post hoc verification. The design therefore separates synthesis quality control from privacy control, which is significant because a sample could be semantically faithful yet still fail the privacy criterion.

A common misunderstanding would be to treat SAM as equivalent to simple imitation prompting. The paper’s baseline for “naive mimicry” was a much simpler prompt of the form “Please try to generate a university assignment description (submission) by imitating the following...”, without the detailed evaluation, synthesis, comparison steps, or privacy checks. On that basis, SAM is better understood as a controlled imitation protocol rather than a single prompting trick.

4. Fidelity, distribution matching, and privacy evaluation

SAM was evaluated with BERTScore F1 for semantic similarity, Pearson correlation coefficient (PCC) and Mean Absolute Error (MAE) for length and mark distributions, and privacy-protection ablations. The reported metric definitions were:

$$\text{BERTScore}_{F1}(\hat{y}, y)=2\cdot\frac{P\cdot R}{P+R}$$

$$\text{PCC}=\frac{\sum_{i=1}^N (x_i-\overline{x})(y_i-\overline{y})}{\sqrt{\sum_{i=1}^N (x_i-\overline{x})^2}\sqrt{\sum_{i=1}^N (y_i-\overline{y})^2}}$$

$\text{MAE}=\frac{1}{N}\sum_{i=1}^N |x_i-y_i|$

The principal quantitative results were as follows (Qian et al., 8 Aug 2025).

Synthetic data	BERTScore F1	PCC (Length)	MAE (Length)	PCC (Marks)	MAE (Marks)
Assignment Desc. (Naive)	0.875	0.656	1018.21	—	—
Assignment Desc. (SAM)	0.859	0.931	586.65	—	—
Student Submissions (Naive)	0.819	0.598	524.86	0.421	26.09
Student Submissions (SAM)	0.840	0.852	335.43	0.624	19.98

The reported interpretation was that both methods achieved high BERTScore F1 values above $0.8$, indicating that LLMs can mimic assignment themes and style, but SAM substantially improved preservation of assignment and submission length distributions and produced markedly stronger correlation with original marks. For assignment descriptions, SAM yielded PCC $0.931$ for length versus $0.656$ for naive mimicry. For student submissions, SAM yielded PCC $S_{real}$0 for length and $S_{real}$1 for marks, compared with $S_{real}$2 and $S_{real}$3 respectively for the naive baseline.

Privacy was examined through ablations. The reported protection rates were 96.5% for naive mimicry with no protection, 99.1% for prompt-only protection, 98.9% for gate-only protection, and 100% when both prompt-level protection and the external privacy gate were used. In the study’s terms, this amounted to perfect privacy protection under the evaluated protocol.

5. Dataset utility for feedback-generation research

The practical motivation for SAM was not only synthetic-data generation in isolation but support for research on automated educational feedback. To test utility, the study used 10 LLMs from OpenAI, Google, and Deepseek and generated 1000 pairs of feedback for synthetic and real assignments. These outputs were evaluated along 16 dimensions covering content, effectiveness, and hallucinations. The reported result was that feedback quality on synthetic data was statistically indistinguishable from feedback quality on real data, with the largest mean difference across categories equal to 0.02 (Qian et al., 8 Aug 2025).

This result is important because it ties dataset resemblance to downstream research validity. The claim is not simply that synthetic assignments resemble originals according to intrinsic metrics, but that they support LLM-feedback experiments at the same level of effectiveness as the real-world assignment dataset. The paper also reported that the standard deviation in feedback for real data was slightly higher, attributing this to more naturally long-tailed variation in real submissions.

The framework was further evaluated across multiple generation models. Larger state-of-the-art LLMs, including GPT-4.1 and o3/o4 variants, were tested, and smaller models such as GPT-4.1 mini and nano were reported to perform worse on distributional resemblance, although BERTScore F1 remained strong across models. This suggests that semantic similarity is easier to preserve than fine-grained distributional properties such as marks and lengths.

6. Limitations, interpretation, and related uses of the term

The study identified two principal limitations. First, the synthetic data matched overall distributions but underrepresented the long tail of especially strong or weak student behaviors, with clustering around the mean described as more likely. Second, BERTScore F1 was noted as a potentially incomplete measure of thematic and writing-style mimicry, motivating future use of more sensitive evaluation procedures such as LLM-as-a-Judge (Qian et al., 8 Aug 2025).

These limitations shape how SAM should be interpreted. It is a high-fidelity one-to-one imitation framework, but not a guarantee of exhaustive behavioral realism. Its strongest empirical support lies in semantic preservation, length matching, moderate mark correlation, and downstream feedback utility under the tested conditions. Claims beyond those axes would require further validation.

The phrase “assignment mimicry” also appears in other technical literatures, especially in computer vision systems built around the Segment Anything Model. There it has been used to describe automatic prompt assignment, identity-assignment correction, or semantic patch assignment, as in self-prompting segmentation, selective mask propagation for multi-object tracking, and composable-prompt patch merging (Zhou et al., 2024, Holmberg, 11 Jun 2026, Chen et al., 2024). This suggests that the expression has broader methodological resonance as a label for systems that learn or simulate expert assignment decisions. In the SCALEFeedback context, however, Sophisticated Assignment Mimicry refers specifically to a privacy-preserving, one-to-one LLM imitation framework for synthetic educational assignment generation (Qian et al., 8 Aug 2025).