Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors

Abstract: Research on reasoning in LMs predominantly focuses on improving the correctness of their outputs. But some important applications require modeling reasoning patterns that are incorrect. For example, automated systems that can reason about and simulate student errors are useful for providing real-time feedback in the classroom or offline practice for educators-in-training. This paper presents a new method, MISTAKE, that (1) constructs high-quality synthetic examples of reasoning errors by leveraging cycle consistency between incorrect answers and latent misconceptions; and (2) uses the generated data to learn models for student simulation, misconception classification, and answer generation. We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers based on specific misconceptions, (2) increased performance inferring latent misconceptions from observed incorrect answers, and (3) higher alignment with expert-written distractor answers when generating incorrect answers (e.g., for multiple-choice tests).

• The paper introduces an unsupervised, cycle-consistent MISTAKE framework that models incorrect student reasoning without human annotation.
• Cycle consistency generates synthetic misconceptions and improves student simulation accuracy by about 9% and distractor precision by 64.6%.
• Iterative joint training with LoRA fine-tuning enables scalable applications in automated feedback, teacher training, and adaptive assessments.

Modeling Incorrect Student Reasoning via Cycle Consistency: The MISTAKE Framework

Introduction

The paper "Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors" (2510.11502) addresses the challenge of simulating and understanding incorrect reasoning patterns in educational contexts using LMs. While most LM research focuses on generating correct outputs, this work targets the generation and modeling of plausible, human-like errors, which are critical for applications such as automated feedback, teacher training, and distractor generation for assessments. The authors introduce the MISTAKE framework, an unsupervised, cycle-consistent approach for synthesizing and learning from incorrect reasoning traces, misconceptions, and answers, without requiring human-annotated error data.

Methodology

Cycle-Consistent Data Generation

The core innovation is the use of cycle consistency to generate high-quality synthetic data representing student errors. The process involves:

• Sampling Incorrect Answers: For each question, a base LM is prompted to produce several plausible incorrect answers.
• Misconception Inference: For each sampled incorrect answer, a misconception inference model ($M_m$) generates a latent misconception and a reasoning trace that could have led to the error.
• Student Simulation: A student simulation model ($M_s$) is conditioned on the inferred misconception to simulate the reasoning and answer a student would produce.
• Cycle Consistency Check: The simulated answer is compared to the original sampled incorrect answer. If they match, the example is upweighted; if not, it may be discarded or downweighted depending on the variant.

This cycle ensures that the inferred misconception both explains the observed error and, when simulated, reproduces it, providing a strong unsupervised filter for data quality.

Iterative Model Training

The framework employs an EM-style iterative training loop:

1. Initialization: Both $M_s$ and $M_m$ are seeded with a pretrained LM (Llama-3.1-8B-Instruct).
2. Data Generation: Synthetic data is generated using the current models.
3. Model Update: $M_s$ and $M_m$ are fine-tuned on the new data (using LoRA, $r=8$).
4. Repeat: The process is repeated for $T$ rounds, with each iteration improving the models' ability to simulate and infer misconceptions.

Variants of the method differ in the strictness of the cycle consistency check (e.g., requiring exact answer match vs. only incorrectness).

Experimental Evaluation

Dataset

Experiments are conducted on the EEDI Mining Misconceptions in Mathematics dataset, which contains K–12 math questions, expert-written distractors, and misconception annotations. The dataset is split by question to ensure generalization.

Tasks

Three key educational tasks are evaluated:

• Student Simulation: Given a misconception, simulate the incorrect answer a student would produce.
• Misconception Inference: Given an incorrect answer, infer the underlying misconception.
• Distractor Generation: Generate plausible distractor answers for multiple-choice questions.

Metrics

• Student Simulation: Accuracy of simulated answers matching ground truth distractors.
• Misconception Inference: MAP@25 using Instructor-XL embeddings to measure retrieval of true misconceptions.
• Distractor Generation: Precision of generated distractors matching expert-written distractors, judged by GPT-4o-mini.

Results

• Student Simulation: The best MISTAKE variant (cycle+correct) improves accuracy by ~9% over the base model (40.83% → 44.43%). Even large models (GPT-4o, GPT-4.1) struggle, with a significant drop in accuracy compared to solving the problems correctly.
• Misconception Inference: MAP@25 improves by ~15% (0.178 → 0.204) with cycle+correct. Joint training of $M_s$ and $M_m$ is essential; ablations show degraded performance when only one model is trained.
• Distractor Generation: Cycle consistency filtering yields a 64.6% increase in precision (22.56% → 37.14%) for distractor alignment. The improvement is consistent across all tested models, including GPT-4o and GPT-4.1.

Comparison with closed-source models shows that MISTAKE-trained Llama-3.1-8B-Instruct models approach or surpass GPT-3.5-turbo on simulation and inference tasks, and cycle consistency filtering improves distractor generation for all model scales.

Implementation Considerations

• Computational Requirements: All experiments are run on a single H100 GPU. LoRA fine-tuning is used for efficiency.
• Prompt Engineering: Few-shot prompts with manually written reasoning traces are critical for effective simulation and inference.
• Data Filtering: Empty outputs and format errors are filtered out to maintain data quality.
• Scalability: The unsupervised nature of the framework allows for large-scale data generation without human annotation, addressing a major bottleneck in educational AI.

Implications and Future Directions

The MISTAKE framework demonstrates that cycle consistency is a powerful unsupervised criterion for generating and modeling incorrect reasoning. The approach is effective for simulating student errors, inferring misconceptions, and generating high-quality distractors, all of which are essential for adaptive assessment, teacher training, and robust evaluation of educational AI systems.

Theoretically, the work suggests that modeling the bidirectional relationship between errors and misconceptions is crucial for understanding human reasoning. Practically, the framework can be extended to other domains requiring user simulation, such as chat-based tutoring, psychology, and economics, where latent cognitive patterns drive behavior.

Future research may explore:

• Integration with chat-based LLMs for real-time tutoring tailored to misconceptions.
• Application of cycle consistency to user simulation in non-educational domains.
• Extension to multimodal reasoning and error modeling.
• Automated curriculum design leveraging simulated misconceptions.

Conclusion

The MISTAKE framework provides an effective, scalable method for modeling incorrect reasoning in educational settings. By leveraging cycle consistency and unsupervised data generation, it advances the state-of-the-art in student simulation, misconception inference, and distractor generation. The results highlight both the difficulty of modeling human-like errors and the promise of cycle-consistent approaches for improving educational AI.