LDSim: LLM Distillation-based QA Simulator

• LDSim is a QA simulator that distills LLM’s knowledge and reasoning into a lightweight student model to predict correct responses using simulated student histories.
• It integrates knowledge distillation and reasoning distillation modules with a graph attention network to construct a hierarchical concept-question graph for enhanced simulation.
• Empirical results show significant performance and efficiency improvements over both LLM-free and LLM-based baselines for multi-step and single-step predictions.

LLM Distillation based Simulator (LDSim) is a question-answering (QA) simulator introduced to mimic real student learning behaviors and predict the correctness of responses to questions given a student’s QA history. In the formulation of "Constructing a Question-Answering Simulator through the Distillation of LLMs" (Liu et al., 11 Sep 2025), LDSim targets both multi-step QA simulation for Educational Recommender Systems (ERS) and single-step knowledge tracing (KT), and is designed to bridge two existing categories of solutions: LLM-free sequential models, which offer fast inference but generally yield suboptimal performance, and LLM-based methods, which leverage domain knowledge and reasoning capability of LLMs but incur slower inference speed and higher GPU memory consumption.

1. Problem setting and motivation

A QA simulator predicts whether a student will answer upcoming questions correctly. Let $u \in \mathcal{U}$ be a student, $q \in \mathcal{Q}$ a question, $c \in \mathcal{C}$ a concept, and $x_i^u = (q_i, \mathcal{C}_i, r_i)$ the QA record at step $i$, with $r_i \in \{0,1\}$. The student’s history up to time $t$ is $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$. For a sequence of upcoming questions $q_{t+1:t+n}$ with associated concept sets $\mathcal{C}_{t+1:t+n}$, the simulator predicts multi-step correctness as

$q \in \mathcal{Q}$0

During interaction with ERS, future correctness labels are unknown, so the simulator conditions on its own previous predictions, thereby simulating student behavior with partially synthetic histories. In the single-step KT formulation, LDSim effectively models

$q \in \mathcal{Q}$1

with the shorthand

$q \in \mathcal{Q}$2

where $q \in \mathcal{Q}$3 is the latent state induced by $q \in \mathcal{Q}$4.

The motivation is tied directly to ERS training. QA simulators enable ERS to collect large amounts of training data without interacting with real students, thereby preventing harmful recommendations made by an undertrained ERS from undermining actual student learning. Within this setting, prior LLM-free KT models encode sequences of question IDs, concept IDs, and correctness labels into a hidden state, but typically underuse semantics and concept relations. Prior LLM-based predictors achieve stronger accuracy by exploiting domain knowledge and reasoning, but are impractical for real-time ERS training because inference is slow and memory-heavy. LDSim addresses this trade-off by distilling domain knowledge and reasoning from a teacher LLM into a lightweight student simulator.

2. Distillation of domain knowledge and reasoning

LDSim consists of a Knowledge Distillation (KD) module, a Reasoning Distillation (RD) module, and a Simulation (SiM) module. The KD module extracts the LLM’s world knowledge on prerequisite relations among concepts in order to build a concept relation graph. The procedure is two-stage. First, for concepts $q \in \mathcal{Q}$5 and $q \in \mathcal{Q}$6, the LLM assesses whether they are related using a prompt template $q \in \mathcal{Q}$7: $q \in \mathcal{Q}$8 The method swaps positions $q \in \mathcal{Q}$9 to reduce generation randomness. If both $c \in \mathcal{C}$0 and $c \in \mathcal{C}$1, the system asks whether $c \in \mathcal{C}$2 is a prerequisite of $c \in \mathcal{C}$3: $c \in \mathcal{C}$4 It again swaps the roles of $c \in \mathcal{C}$5 to capture asymmetric prerequisites. The resulting concept relation graph is $c \in \mathcal{C}$6, where $c \in \mathcal{C}$7 and $c \in \mathcal{C}$8 if $c \in \mathcal{C}$9; the edge $x_i^u = (q_i, \mathcal{C}_i, r_i)$0 indicates that $x_i^u = (q_i, \mathcal{C}_i, r_i)$1 is a prerequisite of $x_i^u = (q_i, \mathcal{C}_i, r_i)$2 (Liu et al., 11 Sep 2025).

The RD module distills the LLM’s ability to reason about a student’s mastery on concepts at each time step. Given the full QA record sequence $x_i^u = (q_i, \mathcal{C}_i, r_i)$3, the KD graph $x_i^u = (q_i, \mathcal{C}_i, r_i)$4, and aggregate statistics such as question and concept correct rates, a prompt template $x_i^u = (q_i, \mathcal{C}_i, r_i)$5 asks the LLM to infer the student’s average mastery over the concepts in $x_i^u = (q_i, \mathcal{C}_i, r_i)$6 at step $x_i^u = (q_i, \mathcal{C}_i, r_i)$7, returning $x_i^u = (q_i, \mathcal{C}_i, r_i)$8: $x_i^u = (q_i, \mathcal{C}_i, r_i)$9 Here $i$0 is an LLM “credit score.” This produces a distilled dataset for actual QA records,

$i$1

To enrich coverage of concepts beyond those actually answered, LDSim adds pseudo QA (PQA) augmentation. For each step $i$2, it selects $i$3 random questions $i$4 and prompts the LLM to infer mastery even though correctness is unknown: $i$5 This better distills the teacher’s reasoning across a broader concept set.

The teacher LLM used for both knowledge and reasoning signals is GLM-4-Flash. A common misconception is to treat LDSim as a standard logit-matching KD system. In the paper’s formulation, the distilled signals are not teacher logits but a concept prerequisite graph $i$6, stepwise mastery scores $i$7, and associated credit scores $i$8.

3. Simulation module and latent-state construction

The Simulation module is the lightweight student network that consumes the distilled concept graph and mastery signals to predict correctness quickly and accurately. It constructs a hierarchical concept-question graph $i$9 with nodes $r_i \in \{0,1\}$0. Edges include the concept–concept prerequisites $r_i \in \{0,1\}$1 from KD and concept–question edges when a concept $r_i \in \{0,1\}$2 is covered by a question $r_i \in \{0,1\}$3. A Graph Attention Network (GAT) encodes the graph: $r_i \in \{0,1\}$4 where $r_i \in \{0,1\}$5, $r_i \in \{0,1\}$6, and $r_i \in \{0,1\}$7 is the embedding dimension.

Each concept $r_i \in \{0,1\}$8 is contextualized with historical student statistics. Let $r_i \in \{0,1\}$9 denote the student’s historical correct rate on $t$0, and $t$1 the number of attempts. LDSim computes

$t$2

where $t$3 and $t$4 are linear layers, $t$5 is concatenation, and $t$6 is multi-head attention. The contextual concept embeddings covered at step $t$7 are then aggregated as

$t$8

The QA history is encoded into a learning state $t$9. Let $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$0 be the global correct rate of question $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$1 estimated from training data, and let $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$2 be a linear layer: $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$3

Distilled reasoning enters the student through a mastery head. LDSim maps $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$4 to a discrete mastery level distribution via $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$5, defined as an MLP followed by softmax over $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$6 levels: $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$7 Correctness prediction then fuses the learning state, mastery embedding, and question embedding: $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$8 With binary labels, $H_t^u = \{(x_1^u,\ldots,x_t^u)\}$9.

This design makes the latent learning state explicitly dependent on concept semantics, concept-question coverage, and historical performance signals. The use of a hierarchical concept-question graph is central: the graph injects prerequisite relations distilled from the teacher LLM into the student model’s representation space.

4. Optimization, training schedule, and preprocessing

LDSim’s student losses do not use KL distillation or temperature scaling. Instead, the system directly regresses mastery and trains correctness prediction with binary cross-entropy plus an auxiliary mastery-level alignment term. In stage 1, the model optimizes mastery regression: $q_{t+1:t+n}$0 applied over both $q_{t+1:t+n}$1 and $q_{t+1:t+n}$2. In stage 2, the model optimizes

$q_{t+1:t+n}$3

where

$q_{t+1:t+n}$4

and

$q_{t+1:t+n}$5

The $q_{t+1:t+n}$6 term encourages consistency of the sampled mastery level when the prediction matches ground truth. Training proceeds in two stages for stability: first optimize $q_{t+1:t+n}$7, then optimize $q_{t+1:t+n}$8.

The training pipeline has three parts. In the KD pipeline, every concept pair $q_{t+1:t+n}$9 is assessed for relevance twice with swapped order, and mutually relevant pairs are further queried for prerequisite relations in both directions. In the RD pipeline, each QA sequence is prompted with $\mathcal{C}_{t+1:t+n}$0, the full sequence $\mathcal{C}_{t+1:t+n}$1, and aggregate statistics to obtain per-step mastery $\mathcal{C}_{t+1:t+n}$2 and credit $\mathcal{C}_{t+1:t+n}$3, producing $\mathcal{C}_{t+1:t+n}$4 and $\mathcal{C}_{t+1:t+n}$5. Student training then applies the two-stage optimization described above (Liu et al., 11 Sep 2025).

The reported hyperparameters are embedding dimension $\mathcal{C}_{t+1:t+n}$6, attention head count in history-attention $\mathcal{C}_{t+1:t+n}$7, Adam optimizer with learning rate $\mathcal{C}_{t+1:t+n}$8, multi-step simulation horizon $\mathcal{C}_{t+1:t+n}$9, and $q \in \mathcal{Q}$00 in stage 2. The mastery-level embedding size $q \in \mathcal{Q}$01 is not enumerated in the paper and is implemented as an embedding table.

Data preprocessing fixes the maximum QA history length per student at 200. Students with fewer than 50 records are removed for Assist09, and students with fewer than 100 records are removed for the other datasets. The train/validation/test split is $q \in \mathcal{Q}$02. Extracted features include per-student concept-level historical correct rates and attempt counts, global question correct rates, concept coverage per question, and timestamps implicit in sequence order.

5. Benchmarks, baselines, and empirical performance

The evaluation uses four datasets. Junyi contains 2,525,877 QA records, 9,879 students, 9,811 questions, and 574 concepts, with max concepts per question $q \in \mathcal{Q}$03 and positive label rate $q \in \mathcal{Q}$04. Assist09 contains 258,461 records, 2,186 students, 12,625 questions, and 139 concepts, with max concepts per question $q \in \mathcal{Q}$05 and positive label rate $q \in \mathcal{Q}$06. Assist12 contains 825,548 records, 5,183 students, 15,575 questions, and 103 concepts, with max concepts per question $q \in \mathcal{Q}$07 and positive label rate $q \in \mathcal{Q}$08. Algebra (KDD Cup 2010) contains 989,189 records, 5,047 students, 13,994 questions, and 692 concepts, with max concepts per question $q \in \mathcal{Q}$09 and positive label rate $q \in \mathcal{Q}$10.

The baselines span both LLM-free and LLM-based methods. LLM-free KT or simulator baselines are DKT, AKT, ATKT, SAKT, Deep-IRT, DisKT, and DSim. LLM-based baselines are LLM-KT, Agent4Edu, and SINKT. The predictive metrics are Accuracy (ACC) and AUC, while the computational metrics are inference time in seconds and GPU memory in MB for simulating the last 30 steps of a student with a 200-record history.

For multi-step simulation with 30-step horizon, LDSim reports the following results. On Junyi, LDSim reaches ACC $q \in \mathcal{Q}$11 and AUC $q \in \mathcal{Q}$12, versus best baseline SINKT with ACC $q \in \mathcal{Q}$13 and AUC $q \in \mathcal{Q}$14, corresponding to $q \in \mathcal{Q}$15 ACC and $q \in \mathcal{Q}$16 AUC. On Assist09, LDSim reaches ACC $q \in \mathcal{Q}$17 and AUC $q \in \mathcal{Q}$18, versus SINKT at ACC $q \in \mathcal{Q}$19 and AUC $q \in \mathcal{Q}$20, corresponding to $q \in \mathcal{Q}$21 ACC and $q \in \mathcal{Q}$22 AUC. On Assist12, LDSim reaches ACC $q \in \mathcal{Q}$23 and AUC $q \in \mathcal{Q}$24, versus SINKT at ACC $q \in \mathcal{Q}$25 and AUC $q \in \mathcal{Q}$26, corresponding to $q \in \mathcal{Q}$27 ACC and $q \in \mathcal{Q}$28 AUC. On Algebra, LDSim reaches ACC $q \in \mathcal{Q}$29 and AUC $q \in \mathcal{Q}$30, versus SINKT at ACC $q \in \mathcal{Q}$31 and AUC $q \in \mathcal{Q}$32, corresponding to $q \in \mathcal{Q}$33 ACC and $q \in \mathcal{Q}$34 AUC. All LDSim wins indicated with “$q \in \mathcal{Q}$35” have $q \in \mathcal{Q}$36-value $q \in \mathcal{Q}$37 (Liu et al., 11 Sep 2025).

For single-step KT, LDSim reports ACC $q \in \mathcal{Q}$38 and AUC $q \in \mathcal{Q}$39 on Junyi, versus SINKT with ACC $q \in \mathcal{Q}$40 and AUC $q \in \mathcal{Q}$41. On Assist09, LDSim reports ACC $q \in \mathcal{Q}$42 and AUC $q \in \mathcal{Q}$43, versus SINKT with ACC $q \in \mathcal{Q}$44 and AUC $q \in \mathcal{Q}$45. On Assist12, LDSim reports ACC $q \in \mathcal{Q}$46 and AUC $q \in \mathcal{Q}$47, versus AKT with ACC $q \in \mathcal{Q}$48 and AUC $q \in \mathcal{Q}$49. On Algebra, LDSim reports ACC $q \in \mathcal{Q}$50 and AUC $q \in \mathcal{Q}$51, versus AKT with ACC $q \in \mathcal{Q}$52 and AUC $q \in \mathcal{Q}$53. The observed trend is that single-step prediction is easier than multi-step simulation due to lack of error accumulation.

6. Ablation, efficiency, limitations, and broader LDSim usage

The ablation study isolates the effects of KD and RD. In the w/o KD condition, the LLM-derived $q \in \mathcal{Q}$54 is replaced with a fully connected concept graph, and performance drops modestly. In the w/o RD condition, the LLM-generated mastery is removed and stage-1 training is skipped, and performance drops substantially. The paper’s key observation is that both KD and RD contribute positively, with RD being the dominant source of gains. KD still helps, but some world knowledge may already be indirectly captured via mastery signals, making KD’s marginal effect smaller. A case study reports that KD produces meaningful concept relations, such as “Area Parallelogram” as a prerequisite for “Surface Area of Prism,” and that post-training embeddings reflect distilled structure because correlated concept embeddings are closer than uncorrelated ones (Liu et al., 11 Sep 2025).

The efficiency comparison is reported for predicting the last 30 steps for one student with a 200-record history. LLM-KT requires 170.24 s and 2338.01 MB. Agent4Edu requires 3117.62 s, with memory unavailable because it is API-based. SINKT requires 12.30 s and 66.43 MB. LDSim requires 0.73 s and 170.28 MB. The reported speedups are approximately $q \in \mathcal{Q}$55 versus LLM-KT, $q \in \mathcal{Q}$56 versus Agent4Edu, and $q \in \mathcal{Q}$57 versus SINKT. The discussion attributes this to lightweight graph-attention and attention encoders that yield near-real-time inference suitable for ERS training and online deployments.

The limitations are also explicit. The prerequisite graph and mastery signals inherit LLM biases and nondeterminism. The paper mitigates this by symmetric prompting but does not apply self-consistency or majority voting. Domain shift may make distilled knowledge noisy if the concept taxonomy or question phrasing is out-of-domain for the teacher LLM. LDSim does not distill chain-of-thought rationales; only mastery scalars and graph structure are used. Explicit calibration is not explored, although the paper notes that improved calibration could benefit downstream ERS policies. Ethics and privacy are discussed in terms of aggregate performance statistics, textual concept descriptions, and the need for privacy-preserving distillation and fairness across subpopulations.

Reproducibility is supported through a public code repository and a concise implementation summary: a student simulator with GAT over a hierarchical concept-question graph, attention encoders for concept context and history, a mastery-level head and classifier, MLP prediction, two-stage training with Stage 1 on MSE mastery regression and Stage 2 on BCE plus $q \in \mathcal{Q}$58, Adam with $q \in \mathcal{Q}$59, $q \in \mathcal{Q}$60, single-head history attention, GLM-4-Flash as teacher LLM, data split $q \in \mathcal{Q}$61, maximum history length $q \in \mathcal{Q}$62, short-history filtering, and simulation horizon $q \in \mathcal{Q}$63.

Beyond educational QA simulation, the label or framing of an LLM Distillation based Simulator appears in other arXiv work as a broader pattern rather than a single architecture. "LLM-Powered User Simulator for Recommender System" frames an LLM-powered recommender simulator as an LDSim in which ChatGLM-6B distills interpretable pros/cons and category predicates into logical models and a SASRec-based statistical component (Zhang et al., 2024). "Can Low-Rank Knowledge Distillation in LLMs be Useful for Microelectronic Reasoning?" presents design guidance for an LDSim built from LoRA-KD with a Llama-2-70B teacher and Llama-2-7B student for offline microelectronic reasoning (Rouf et al., 2024). "SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generation" maps a multi-agent reasoning-and-verification pipeline into an LDSim blueprint centered on verified training tuples and testbench-driven debugging (Chen et al., 10 Feb 2026). This suggests that, in current usage, “LDSim” may denote a family of systems in which LLM-derived knowledge or reasoning is distilled into a more efficient simulator or student model, while the specific architecture remains task-dependent.