Math Tutor Bench: Benchmarking Math Tutoring LLMs

Updated 8 October 2026

MathTutorBench is a benchmark designed to assess the pedagogical capabilities of large language models (LLMs) in interactive mathematics dialogues.
It separates mathematical expertise, student-understanding analysis, and pedagogical response generation to measure various aspects of tutoring effectiveness.
The benchmark includes tasks such as problem solving, Socratic questioning, student-solution verification, mistake localization, scaffolding generation, and instruction following.

MathTutorBench is an open-source benchmark for evaluating the pedagogical capabilities of LLM tutors in multi-turn mathematics dialogues. It separates mathematical expertise, student-understanding analysis, and pedagogical response generation, thereby treating mathematical correctness as necessary but insufficient for effective tutoring. Its tasks include problem solving, Socratic questioning, student-solution verification, mistake localization, mistake correction, scaffolding generation, and pedagogical instruction following. The benchmark combines conventional mathematical metrics with a learned Scaffolding reward model intended to evaluate open-ended teacher responses (Macina et al., 26 Feb 2025).

1. Purpose and conceptual foundations

MathTutorBench addresses the distinction between solving a mathematical problem and teaching a student how to solve it. A tutor may produce a correct solution while revealing the answer prematurely, failing to identify a misconception, addressing the wrong step, providing non-actionable feedback, overwhelming the learner, or preventing self-correction.

The benchmark is motivated by limitations in several established evaluation paradigms:

Text-overlap metrics such as BLEU and BERTScore assume reference-like wording despite the open-ended nature of tutoring.
Problem-solving and question-answering benchmarks measure subject expertise but not teaching quality.
Human evaluation captures pedagogical nuance but is expensive, slow, and difficult to reproduce.
LLM-as-a-judge methods may be prompt-sensitive, position-biased, difficult to reproduce, and dependent on proprietary judges.
Student-learning studies measure ultimate educational outcomes but require costly and ethically constrained experiments.

MathTutorBench is therefore intended as a rapid, reproducible screening and development tool rather than a replacement for controlled studies of student learning. Its central pedagogical principles are correctness, scaffolding rather than answer revelation, encouragement of self-correction, and avoidance of cognitive overload.

The benchmark is related to three approaches to constructing LLM tutors: prompt-based elicitation, supervised fine-tuning on teacher-authored or synthetic tutoring conversations, and pedagogical alignment through post-training on pedagogical preferences. Its purpose is to provide a common evaluation infrastructure for comparing these approaches.

2. Capability model and formal setup

MathTutorBench organizes tutoring ability into three high-level capabilities:

Math expertise
Student understanding
Pedagogical ability and teacher-response generation

For a problem $x$ , the benchmark represents a reference solution as a sequence of steps,

$s=(s_1,\ldots,s_N),$

where $s_N$ contains the final answer. A student solution is represented as

$\widehat{s}=(\widehat{s}_1,\ldots,\widehat{s}_M).$

The first erroneous step is indexed by

$e\in\{0,1,\ldots,M\},$

with $e=0$ denoting a correct student solution. A dialogue history is

$\mathcal{H}=(u_1,\ldots,u_{T-1}),$

and the tutor generates the next utterance $u_T$ conditional on the history, optional mathematical context $\mathcal{K}$ , and a teacher intent $i_T$ :

$s=(s_1,\ldots,s_N),$ 0

This formulation separates three operations that are frequently conflated in ordinary mathematical benchmarks: solving the problem, interpreting the learner’s reasoning, and selecting an appropriate instructional intervention.

The benchmark primarily uses GSM8K for mathematical problems, Bridge and MathDial for tutoring dialogues, and StepVerify for annotations of erroneous student solutions. The source data represent different pedagogical regimes: Bridge contains approximately 700 snippets of novice-teacher conversations revised by expert teachers, MathDial contains approximately 2.9k human-written tutoring conversations with simulated students, and StepVerify extends incorrect MathDial solutions with first-error annotations. The NCTE dataset was excluded because its multiple-persona format did not fit the one-to-one teacher–student setting.

3. Tasks and datasets

MathTutorBench contains seven principal tasks.

Capability	Task	Dataset	Instances
Math expertise	Problem solving	GSM8K	1,319
Math expertise	Socratic questioning	GSM8K	1,319
Student understanding	Solution correctness	StepVerify	2,004
Student understanding	Mistake location	StepVerify	2,004
Student understanding	Mistake correction	StepVerify	1,002
Pedagogy	Scaffolding generation	MathDialBridge	1,150
Pedagogy	Pedagogical instruction following	MathDialBridge[hard]	327

Problem solving

The model receives a GSM8K word problem and generates a solution. Evaluation uses final numerical answer accuracy. This task measures mathematical expertise but is not treated as evidence of tutoring quality.

Socratic questioning

The model receives a correct solution step $s=(s_1,\ldots,s_N),$ 1 and generates a guidance question that could lead a student toward that step. Evaluation uses BLEU against reference questions. The authors characterize this as a relatively surface-level metric that does not fully determine whether a question is pedagogically useful.

Student-solution correctness

The model determines whether a student solution is correct. This is balanced binary classification evaluated with F1 on StepVerify. The ground-truth correctness label is

$s=(s_1,\ldots,s_N),$ 2

because $s=(s_1,\ldots,s_N),$ 3 indicates that no mistake is present.

Mistake location

The model identifies the exact location of the student’s first error. The output is the error index $s=(s_1,\ldots,s_N),$ 4, including $s=(s_1,\ldots,s_N),$ 5 for a correct solution. Evaluation uses micro-F1 over possible error locations. This task emphasizes that useful feedback should address the origin of a misconception rather than a later consequence.

Mistake correction

The model receives a dialogue history containing an incorrect student trajectory and generates a reasoning chain ending in the correct numerical answer. Unlike ordinary problem solving, this task conditions on the student’s error path. Evaluation uses final numerical answer accuracy over 1,002 examples with an average of 3.04 turns.

Scaffolding generation

The model generates the next tutor utterance in an ongoing dialogue, generally responding to an incorrect student solution. The basic instruction asks it to respond as “an experienced math teacher in a useful and caring way.” Responses are scored by the learned Scaffolding reward model, and performance is reported as the win rate over corresponding human teacher responses.

Pedagogical instruction following

This task adds the LearnLM “extended” prompt, which requests nudging, guiding questions, avoidance of answer revelation, avoidance of information overload, and actionable feedback. It is evaluated on both ordinary dialogues and MathDialBridge[hard], whose conversations average 5.78 turns rather than approximately 3.08 turns in the ordinary split.

4. Scaffolding reward model

Open-ended tutoring cannot be evaluated reliably by exact reference matching because multiple responses may be pedagogically acceptable. MathTutorBench therefore trains a reward model that assigns scalar pedagogical scores to tutor responses.

Preference data are constructed from synthetic criteria scores. For a response $s=(s_1,\ldots,s_N),$ 6, the criteria score is

$s=(s_1,\ldots,s_N),$ 7

Whenever $s=(s_1,\ldots,s_N),$ 8, the pair $s=(s_1,\ldots,s_N),$ 9 is treated as a preference pair, with margin

$s_N$ 0

Preferred responses tend to ask Socratic questions, probe student understanding, focus on the erroneous step, provide actionable hints, and avoid revealing the complete solution. Dispreferred responses tend to state the answer, reproduce solution steps, provide incorrect feedback, address the wrong step, or exhibit fewer desired pedagogical behaviors.

The reward model assigns a score $s_N$ 1 to a context–response pair and is trained with a pairwise ranking objective:

$s_N$ 2

where $s_N$ 3 is the preferred response, $s_N$ 4 is the rejected response, and $s_N$ 5 is the sigmoid function.

The selected reward model is based on Qwen2.5-1.5B-Instruct. It is fine-tuned for one epoch with learning rate $s_N$ 6, batch size 16, AdamW, and approximately one hour of training on one NVIDIA A100 80GB GPU. Its validation set contains 482 Bridge examples pairing expert and novice teacher responses. The best model achieves 0.84 accuracy in ranking the expert response above the novice response.

The reward-model training mixture shows that human-annotated pedagogical preference data are more useful than synthetic inpainting alone. Training on MRBench and MathDial produces the strongest reported validation performance, whereas adding a margin to the loss does not improve the result in that experiment. General-purpose LLM judges and chat-oriented reward models perform below the specialized pedagogical reward model in the reported comparisons.

MRBench contributes eight tutoring criteria: guidance, actionability, answer revelation, mistake identification, mistake location, coherence, tone, and humanness. The reward model assigns low scores to responses that confirm incorrect answers, state false facts, reveal complete solutions, provide only generic praise, or merely say that an answer is wrong. Higher scores are assigned to responses that target the precise erroneous step, invite reconsideration, provide a manageable next question, and encourage self-correction.

5. Evaluation results and capability trade-offs

MathTutorBench evaluates general-purpose, specialized tutoring, and mathematical-reasoning models under common prompts, temperature 0, and a maximum generation length of 2048 tokens. The response-generation scores are reward-model win rates over human teacher responses.

Model	Problem solving	Correctness F1	Mistake location	Mistake correction	Scaffolding	Pedagogical instruction following
LLaMA 3.2-3B-Instruct	0.60	0.67	0.41	0.13	0.64	0.63
LLaMA 3.1-70B-Instruct	0.91	0.71	0.56	0.19	0.63	0.70
GPT-4o	0.90	0.67	0.37	0.84	0.50	0.82
LearnLM-1.5-Pro	0.94	0.75	0.57	0.74	0.64	0.68
Qwen2.5-7B SocraticLM	0.73	0.05	0.39	0.23	0.39	0.39
Qwen2.5-Math-7B-Instruct	0.88	0.43	0.47	0.49	0.06	0.07

The benchmark’s principal result is a dissociation between mathematical problem solving and tutoring quality. Qwen2.5-Math-7B-Instruct obtains 0.88 on problem solving but only 0.06 on ordinary scaffolding, 0.07 on pedagogical instruction following, and 0.05 on both hard response-generation measures. GPT-4o obtains 0.90 on problem solving but 0.50 on ordinary scaffolding.

The results also indicate a specialization trade-off. SocraticLM improves scaffolding over Qwen2.5-Math but has very low solution-correctness F1 and weaker mistake-correction accuracy. ScienceTutor has lower problem-solving accuracy but stronger solution correctness and instruction following than SocraticLM. LearnLM provides the strongest reported balance across solving, student-understanding, mistake correction, and scaffolding. The authors characterize the relationship between expertise and pedagogy as a trade-off navigated by the degree and type of tutoring specialization.

Student understanding constitutes a separate capability. LLaMA 3.1-70B-Instruct achieves 0.91 problem-solving accuracy but only 0.19 mistake-correction accuracy. Qwen2.5-Math-7B-Instruct achieves 0.88 problem solving but 0.43 solution-correctness F1 and 0.49 correction accuracy. These results indicate that mathematical competence does not guarantee reliable interpretation of a learner’s reasoning.

Longer dialogues are generally more difficult. Performance tends to decline on MathDialBridge[hard], where dialogues average 5.78 turns. This suggests that effective tutoring requires adaptation to the interaction history rather than repeated application of a fixed questioning template. LearnLM is described as retaining essentially consistent performance on the longer setting, whereas several other models decline substantially.

Explicit pedagogical instructions help unevenly. GPT-4o improves from 0.50 ordinary scaffolding to 0.82 pedagogical instruction following and achieves 0.70 on the hard instruction-following setting. The same explicit prompting produces similar or lower scores for several other models, indicating that listing desired behaviors does not guarantee reliable implementation.

MathTutorBench has been used as an external evaluation suite in subsequent work on prompt optimization, reinforcement learning, student simulation, and educational-impact diagnostics.

Training-free prompt optimization evaluates MathTutorBench as an out-of-distribution suite with two sub-benchmarks, abbreviated Sc and Pd. In that study, MIPROv2 obtains the highest reported MathTutorBench average, 8.16, while ParetoGrad ranks first on the in-domain reward but ninth on MathTutorBench. The correlation between in-domain reward and MathTutorBench average is reported as $s_N$ 7, suggesting that local tutoring rewards do not necessarily predict transfer to held-out pedagogical evaluation (Lee et al., 26 May 2026).

PEARL evaluates open-ended Socratic mathematical tutoring with a controllable student simulator and a multidimensional reward model. Its reward dimensions include accuracy, leakage, completion, cognitive load, guidance, metacognition, adaptivity, and emotion. PEARL-30B obtains a reported MathTutorBench composite score of 0.79, compared with 0.80 for Gemini 3.1 Pro. These values summarize the cited paper’s evaluation and do not reconstruct the official MathTutorBench task definitions (Chang et al., 28 May 2026).

EDUARDO uses masked near-transfer post-tests to distinguish teaching from solution handover. Its reported MathTutorBench composite score is 0.79 for Eduardo-27B and 0.80 for Gemini 3.1 Pro. The paper does not provide uncertainty estimates for this comparison, and the reported 2.4–6.2-fold thinking-token efficiency advantage is measured primarily on TutorMoments rather than directly on the MathTutorBench composite (Macina et al., 5 Oct 2026).

A diagnostic reanalysis divides MathTutorBench’s public task scores into solving-oriented and pedagogy-oriented composites. Across eight models, the correlation between these composites is 0.421. Qwen2.5-Math-7B has a solving score of 0.568 and a pedagogy score of 0.116, while LearnLM-1.5-Pro scores 0.750 and 0.594, respectively. The analysis treats these composites as descriptive diagnostics rather than measures of downstream learning (Yao et al., 15 Jun 2026).

Several mathematics resources provide possible sources for extending the benchmark’s curriculum. The calculus textbook Integral Calculus of One-Dimensional Functions: Personal Tasks and Samples supplies 30 individualized task variants, 40 problems per task, worked solutions, visualization, Mathematica-generated variants, and 20 video lessons. Its content spans antiderivatives, substitution, integration by parts, rational and trigonometric integration, definite and improper integrals, areas, arc lengths, surfaces, and volumes. However, source formulas and worked solutions contain typographical and mathematical inconsistencies, so every item requires normalization and independent verification before benchmark use (1908.10069).

The Spanish-language text Matemáticas Básicas (con Software Libre) provides pre-calculus material on algebra, modular arithmetic, primes, induction, functions, exponentials and logarithms, trigonometry, analytic geometry, and complex numbers. Its repeated chapter structure combines definitions, applications, exercises, and Maxima or GeoGebra activities. The text is relevant for generating conceptual, proof-oriented, modeling, graph-interpretation, and computational-verification tasks, but it is an instructional resource rather than an evaluation benchmark (Morante et al., 2020).

VideoMathQA offers a complementary multimodal direction. It contains 420 video–question pairs, 2,945 expert-annotated reasoning steps, ten mathematical domains, and videos ranging from 10 seconds to over one hour. Its categories include direct problem solving, conceptual transfer, and deep instructional comprehension. Unlike MathTutorBench, it evaluates single-turn mathematical reasoning from video and does not evaluate live tutoring, learner modeling, hints, or adaptive dialogue (Rasheed et al., 5 Jun 2025).

MathTutorBench has several substantive limitations:

No direct learning outcome: reward-model win rates are proxies for pedagogical quality, not evidence of increased retention, transfer, or student achievement.
Limited dialogue length: the benchmark contains no dialogues longer than ten turns.
Restricted interaction format: it focuses on one-to-one tutoring involving hints, nudges, and questions.
Missing social-emotional functions: rapport, trust, motivation, classroom management, and support for disengaged learners are not evaluated.
No safety evaluation: harmful, biased, manipulative, or unsafe tutoring behavior lies outside the current benchmark.
Reward hacking risk: a model may learn to ask questions mechanically or imitate preferred surface patterns without genuinely helping the student.
Potential dataset artifacts: models may exploit regularities in the training or evaluation data.
Reward-model uncertainty: a single scalar may omit valid context-dependent teaching strategies.
Narrow mathematical scope: the principal problems are middle- and high-school-style multi-step mathematics, without comprehensive coverage of advanced mathematics, formal proof, or multimodal mathematical work.

MathTutorBench is therefore best interpreted as a multidimensional diagnostic rather than a single ranking. Its central contribution is to make mathematical expertise, student-reasoning analysis, and open-ended pedagogical response generation separately measurable. The benchmark supports the conclusion that solving mathematics, understanding a learner’s reasoning, and teaching effectively are distinct capabilities. Its scores should be combined with qualitative analysis, symbolic or numerical verification where appropriate, and controlled studies of student learning.