Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation

Abstract: LLMs have strong potential for use in intelligent tutoring systems, but they often fail to follow effective pedagogical strategies, such as guiding students without revealing final answers. We study the application of a two-stage alignment pipeline for math mistake remediation, combining supervised fine-tuning on tutoring dialogs with Direct Preference Optimization on synthetic preference pairs. We construct a dataset that integrates existing tutoring corpora with synthetic data generated along pedagogical dimensions, such as scaffolding and factuality, and study different input configurations that incorporate solution correctness and gold answers. Experiments show that this approach improves both factual accuracy and pedagogical quality over base models and existing tutoring models. Human evaluation further indicates that our best model is competitive with a strong proprietary baseline, while providing additional benefits in terms of openness, transparency, and reproducibility. Our results highlight the effectiveness of preference-based pedagogical alignment, while also revealing challenges in reliably evaluating tutoring quality.

• The paper introduces a two-stage alignment pipeline that combines supervised fine-tuning with Direct Preference Optimization to enhance tutoring responses.
• It employs synthetic preference pairs and input signals like correctness flags and gold solutions to improve factuality, mistake identification, and guided feedback.
• Empirical results show that the DPO-optimized Qwen3-8B model outperforms open-source baselines and rivals proprietary systems on key pedagogical metrics.

Pedagogical Alignment of LLM Tutors for Math Mistake Remediation via SFT and DPO

Motivation and Problem Formulation

Pedagogical alignment remains a critical barrier for LLM-based intelligent tutoring systems (ITS), especially within math mistake remediation scenarios. While contemporary LLMs demonstrate proficiency in language generation and task reasoning, they frequently fail to adhere to instructional best practices, such as providing guided scaffolding without prematurely revealing final answers, targeted feedback for student misconceptions, and fostering student engagement through Socratic techniques. The absence of explicit modeling and control over these pedagogical attributes impedes reliable tutoring deployment at scale. This paper presents a two-stage alignment pipeline: supervised fine-tuning (SFT) on pedagogical dialogs and Direct Preference Optimization (DPO) on synthetic preference pairs, with the aim to directly align tutor responses to pedagogical desiderata.

Figure 1: Two-stage alignment pipeline—SFT on tutoring dialogs followed by DPO using synthetic preference pairs for mistake remediation; input configurations vary in correctness flag and gold solution access.

Dataset Construction and Synthetic Preference Generation

A rigorous dataset was compiled for math mistake remediation, leveraging MathDial and SocraTeach dialog corpora and augmenting them with synthetically generated preference pairs from MR-GSM8K and PRM800K. Student solutions (both correct and incorrect) were constructed from LLM outputs. High-quality tutor responses were synthesized using GPT-5, while degraded variants across five pedagogical aspects (Factuality, Mistake Identification, Targetedness, Revealing Answer, Clarity) were created with minimal perturbations via GPT-4.1, following the RMBoost approach. Importantly, the Clarity aspect was weighted less in DPO due to its ambiguous effect on pedagogical quality. Preference pairs enable offline alignment along explicit instructional dimensions without dependence on costly human annotation, thereby addressing scalability requirements for research in ITS.

Model Architecture and Training Paradigm

Open-source (Qwen3-4B-Instruct-2507, Qwen3-8B) and proprietary (GPT-4.1-nano) models were utilized as base backbones. Training comprised two stages:

1. Supervised Fine-Tuning (SFT): Cross-entropy loss on tutor turns extracted from dialog datasets, conditioned on varying context configurations (V1: context only; V2: + correctness flag; V3: + gold solution; V4: + both).
2. Direct Preference Optimization (DPO): Weighted preference objective on synthetic pairs, with higher weights for factuality, mistake identification, targetedness, and answer-revealing avoidance.

This modular pipeline allows explicit disentanglement and control of pedagogical subtasks (error detection vs. response generation), as well as systematic investigation of the value of additional input signals (correctness flags, gold solutions).

Evaluation Design and Pedagogical Metrics

Pedagogical effectiveness is measured using both LLM-based evaluators (GPT-5, LoMTL from AITutor-EvalKit) and human annotators. Four instructional dimensions are targeted: Mistake Identification, Mistake Location, Guidance Provision, and Actionability. Factuality is explicitly pre-filtered via specialized prompts before downstream pedagogical analysis. Human evaluation emphasizes preference testing within pairwise comparisons, using domain-expert annotators and explicit rubric-based feedback.

Figure 2: Introduction page of the human evaluation form for pairwise dialog comparison.

Figure 3: Explicit reasons provided to annotators for selecting preferred tutor responses.

Empirical Results and Numerical Highlights

Automatic evaluation establishes that SFT and DPO considerably improve feedback quality and factuality over base models, with further gains observed when both correctness flag and gold solution are incorporated (V4). The Qwen3-8B + DPO V4$^*$ configuration, reinforced with additional perturbed preference pairs, achieves a 70.1% factuality rate, outperforming all open-source baselines and rivaling proprietary backbones. Disaggregated analysis shows that DPO enhances factuality for Qwen variants, whereas it decreases it for GPT-4.1-nano relative to SFT. On pedagogical metrics, DPO V4$^*$ robustly outperforms SocraticLM and TutorRL-7B, and in many cases surpasses GPT-5 on targeted guidance and mistake identification as evaluated by LoMTL.

Human evaluation further confirms pipeline efficacy. SFT V4 is preferred over Base in 67.6% of cases; DPO V4$^*$ is preferred over SFT V4 in 35.3% of cases. Strikingly, DPO V4$^*$ is preferred over GPT-5 in 54.3% of comparisons, providing evidence of competitive open-source tutoring systems when properly aligned. Annotators consistently cite factuality, clarity, answer-revealing, and guidance as primary reasons for model preference.

Figure 4: Pairwise human preferences—Qwen3-8B SFT V4 vs. Base model.

Figure 5: Pairwise human preferences—Qwen3-8B SFT V4 vs. DPO V4$^*$.

Figure 6: Pairwise human preferences—Qwen3-8B DPO V4$^*$ vs. GPT-5.

Implications, Limitations, and Future Directions

The results substantiate the efficacy of explicit preference-based pedagogical alignment in LLM tutors. Access to correctness signals and gold solutions during feedback generation is crucial for factual accuracy, suggesting modular ITS design wherein task-solving and feedback are decoupled—a paradigm echoing human tutoring practices. Preference optimization generalizes across backbone and model families, but evaluation remains a challenge: substantial disagreement exists between LLM-based and human evaluators, and actionability interpretation is sometimes ambiguous.

Limitations include reliance on synthetic preference data, limited scale of human evaluation, and lack of direct measurement of student learning outcomes. Further research should address user studies with real learners, enlarge scope and diversity of evaluation, and pursue multi-component, human-in-the-loop pipeline architectures. Comprehensive bias and safety concerns for real-world deployment must also be considered.

Conclusion

This work demonstrates that combining supervised dialog fine-tuning with preference-based alignment effectively enhances pedagogical quality and factual correctness of LLM tutors in math mistake remediation. With systematic disentanglement of input signals and explicit instructional aspect modeling, the open-source Qwen3-8B + DPO V4$^*$ model achieves competitive performance vis-à-vis proprietary systems, and sets a replicable precedent for scalable pedagogical alignment in AI tutoring.

The pipeline, datasets, and models are openly released, facilitating reproducible research in pedagogical LLM alignment and advancing prospects for robust, interpretable, and transparent AI tutors (2606.21502).