MathBuddy: Affective Multimodal Math Tutor

• MathBuddy is a multimodal, affect-aware mathematics tutor that uses text and facial expression analysis to instantly adapt teaching strategies.
• It fuses affect signals from student conversations and webcam input to determine whether to challenge or motivate through tailored pedagogical responses.
• Evaluation results indicate improved tutoring outcomes, with enhanced metrics such as Win Rate and DAMR in controlled user studies.

MathBuddy is a multimodal, affect-aware LLM-based mathematics tutor that detects student emotion from conversational text and facial expressions, fuses those signals into primitive affective states, and maps the result to pedagogical strategies that condition tutor tone, scaffolding, and answer style. In the reported system, affect modeling is not treated as a cosmetic layer: it is used to modulate challenge versus motivation, to shape step-by-step guidance, and to improve multiple pedagogical dimensions including coherence, human-likeness, and tutor tone. The system is described as model-agnostic, supports multiple LLM and computer-vision backends, and was evaluated through automatic metrics and a real-time user study (Kar et al., 27 Aug 2025).

1. Problem formulation and scope

MathBuddy is motivated by a specific limitation in contemporary LLM-based tutoring: most systems treat learners uniformly and ignore affective state. The underlying premise is drawn from educational psychology as summarized in the source material: confusion, frustration, boredom, engagement, and curiosity materially affect motivation, problem-solving, and learning gains, and math anxiety is especially detrimental. The system therefore targets a gap in which existing tutors rarely integrate multimodal affect in real time to drive tutoring adaptations (Kar et al., 27 Aug 2025).

The core contribution is a pipeline that captures student emotion from both conversational text and facial expression, aggregates those signals into a joint affect estimate, and then converts that estimate into pedagogical intent. The intended output is not merely emotionally colored dialogue, but an “empathetic” response that remains mathematically correct and pedagogically constrained. In the reported user study, the tutoring domains were geometry and probability, while the broader architecture is described as supporting generic LLM-backed tutoring for algebraic manipulation, functions, equations, and basic calculus (Kar et al., 27 Aug 2025).

MathBuddy is also situated against several adjacent lines of research. Prior art mentioned in the source includes LLM math tutors such as Llemma, pedagogical evaluation taxonomies associated with Maurya et al. (2025), and unimodal affect recognition in NLP and CV. The novelty claim is narrower than a general-purpose “AI tutor”: real-time multimodal affect integration, mapping to primitive emotions, then to pedagogy grounded in education theory, followed by evaluation across eight pedagogical dimensions and measurement of user affect (Kar et al., 27 Aug 2025).

2. System architecture and interaction loop

The reported implementation is organized as a web-based frontend and a Python REST backend. The frontend is a web client with chat UI, optional webcam input for face emotion detection, a simple visual workspace, and in-memory storage. Webcam processing can occur in-browser via face-api.js or via a backend API. The backend handles inference, tutor response generation, and optional face emotion recognition, and supports plug-in LLM backends and CV models (Kar et al., 27 Aug 2025).

The end-to-end interaction loop is explicitly defined in six steps. First, the student sends a math utterance and may optionally stream webcam frames. Second, text-based affect recognition infers affect from the latest student turn. Third, face-based affect recognition runs continuously and aggregates over the interval between messages. Fourth, multimodal fusion maps both modalities to primitive emotions $\{Positive, Neutral, Negative\}$ and merges them with a rule-based preference. Fifth, the system selects a pedagogical strategy: Positive maps to challenge, while Neutral or Negative map to motivate. Sixth, the LLM tutor is prompted with the selected strategy and affect context and generates an empathetic, math-correct response (Kar et al., 27 Aug 2025).

The system is described as model-agnostic. In evaluation, it used Qwen2.5-7B-Socratic-LM (QSLM), LlemmaMM, and LearnLM 2.0 as tutor backends; the user study used LearnLM as the tutor. For silver labeling of text affect, the reported system used DeepSeek-R1-Distill-LLaMA-70B. Automatic evaluation employed LLM judges including Llama-3-70B-Instruct, DeepSeek-V3, Mixtral-8x22B-Instruct, and Phi-4, together with an ensemble formed by majority vote (Kar et al., 27 Aug 2025).

The tutoring behavior is explicitly scoped to guided mathematical assistance rather than unrestricted answer production. Stepwise scaffolding, guided questions, error identification and location, adaptive hinting, and LaTeX rendering when appropriate are all part of the documented behavior. The mathematical examples given in the source include triangle area $A = (1/2)bh$, the Pythagorean theorem $a^2 + b^2 = c^2$, the independence rule $P(A\cap B) = P(A)P(B)$, and the complement rule $P(\bar{A}) = 1 - P(A)$ (Kar et al., 27 Aug 2025).

3. Multimodal affect modeling

MathBuddy’s emotion model is built from two distinct channels: text affect and facial affect. The text model uses multi-label intensity labels for Boredom, Engagement, and Neutral on a $0$–$2$ scale, where $0$ denotes low intensity and $2$ denotes high intensity. The reported annotation set contains 224 student turns from the MathDial-Bridge (hard) split, manually labeled and supplemented with silver training data generated by DeepSeek-R1-Distill-LLaMA-70B. Among the fine-tuned BERT-family classifiers, the best DistilBERT model achieved Accuracy $0.618$ and F1 $A = (1/2)bh$0 on the silver-labeled test set; other variants included bert-base-uncased, roberta-base, distilroberta-base, and Appendix results (Kar et al., 27 Aug 2025).

Text labels are then mapped to primitive emotions. Positive corresponds to $A = (1/2)bh$1, Neutral corresponds to $A = (1/2)bh$2, and Negative is defined as otherwise. Facial affect is processed either by an in-browser face-api.js model or by separately trained CV models. The in-browser model recognizes Happy, Sad, Angry, Surprised, Fearful, Disgusted, and Neutral, and is reported at test Accuracy $A = (1/2)bh$3 and F1 $A = (1/2)bh$4 over 151 FERAC and 38 FER images. Additional trained models included a CNN baseline, an attention-enhanced CNN, and Vision Transformers fine-tuned on small and large data configurations; the best ViT achieved up to Accuracy $A = (1/2)bh$5 and F1 $A = (1/2)bh$6 on test (Kar et al., 27 Aug 2025).

Face signals are not used instantaneously; they are temporally aggregated. Continuous samples are discretized into emotion segments and grouped into consecutive runs with identical recognized emotion. For each group $A = (1/2)bh$7 of emotion $A = (1/2)bh$8, with duration $A = (1/2)bh$9 and age $a^2 + b^2 = c^2$0, and half-life $a^2 + b^2 = c^2$1, the weighted score is

$a^2 + b^2 = c^2$2

and the final face emotion is selected by

$a^2 + b^2 = c^2$3

This decay mechanism privileges more recent affect while preserving interval-level evidence (Kar et al., 27 Aug 2025).

Multimodal fusion is rule-based and confidence-aware. Let $a^2 + b^2 = c^2$4 with confidences $a^2 + b^2 = c^2$5. The design preference is explicit: Positive or Negative are always preferred over Neutral, and confidence breaks ties when both modalities are non-neutral but disagree. Neutral is returned only if both modalities are Neutral, or if there is no face signal. The source notes a direct consequence of this choice: reduced Neutral recall in the user study, which is treated as a limitation rather than an intended optimization (Kar et al., 27 Aug 2025).

4. Pedagogical adaptation and tutoring behavior

MathBuddy converts fused affect into pedagogical strategy through a minimal rule set grounded in education theory. Positive affect triggers challenge, defined as increasing cognitive demand through probing questions and deeper justification. Neutral or Negative affect triggers motivate, defined as encouragement, normalization of struggle, and gentler scaffolding. When the text labels specifically distinguish boredom from engagement, boredom maps to motivate and engagement maps to challenge (Kar et al., 27 Aug 2025).

The paper evaluates tutoring quality along eight pedagogical dimensions: Mistake Identification, Mistake Location, Revealing of the Answer (Answer Disclosure), Providing Guidance, Actionability, Human-likeness, Coherence, and Tutor Tone. These dimensions operationalize the claim that tutoring quality is broader than answer correctness. In particular, the system aims to avoid premature answer disclosure, maintain coherent step-by-step support, and preserve an encouraging, supportive, non-offensive style (Kar et al., 27 Aug 2025).

Prompting is implemented in simple and complex template variants. The simple template instructs the model to be “an experienced math teacher,” to “Gently nudge the student with guiding questions,” and to consider emotion classes such as “Positive: engagement, joy,” “Neutral includes neutral, surprise,” and “Negative includes anger, boredom, confusion, contempt, disgust, fear, frustration, sadness.” The complex template instructs the model to be “a friendly, supportive tutor,” to “Ask one guiding question per turn,” to use boredom $a^2 + b^2 = c^2$6 motivate and engagement $a^2 + b^2 = c^2$7 challenge, and to “Wrap up once the student shows understanding” (Kar et al., 27 Aug 2025).

The tutoring policy examples reveal how affect is intended to alter the instructional move rather than simply the wording. Under motivate, the tutor validates feelings, lowers task complexity, offers hints, and reminds the student of prior success. Under challenge, it poses deeper questions, requests justification or alternative methods, and connects the solution to extensions. The illustrative responses are correspondingly differentiated. For Negative or Neutral affect: “It’s okay to find this tricky—many people do. Let’s break it down. What is the area formula for a right triangle? Can you identify the base and height here? I’ll stay with you through each step.” For Positive affect: “Great reasoning so far! Can you justify why your substitution is valid for all x? What happens if we consider the boundary case x=0—does your expression still hold?” (Kar et al., 27 Aug 2025).

A common simplification is to treat affect adaptation as a pure tone-control layer. The reported design and ablations do not support that reading. The intervention is defined as a mapping from affect to pedagogical strategy, and the prompting layer additionally enforces answer-disclosure control, coherent and actionable guidance, and step-by-step support. This suggests that, in MathBuddy, affect awareness is implemented as a tutoring-policy variable rather than only as a stylistic modifier (Kar et al., 27 Aug 2025).

5. Evaluation methodology and reported results

Automatic evaluation was conducted on MathTutorBench, described as the hard split of MathDial-Bridge with 327 human-annotated conversations, and measured across the eight pedagogical dimensions using emotion from text only. The full multimodal system was then tested in a real-time user study with LearnLM as the backend. The automatic metrics were Win Rate (WR), defined as the preference rate of a reward model over ground-truth tutor responses, and Desired Annotation Match Rate (DAMR), defined as the percentage match between LLM-as-Judge labels and desired labels (Kar et al., 27 Aug 2025).

The metric definitions given in the source are

$a^2 + b^2 = c^2$8

and

$a^2 + b^2 = c^2$9

where $P(A\cap B) = P(A)P(B)$0 is the desideratum for dimension $P(A\cap B) = P(A)P(B)$1, and $P(A\cap B) = P(A)P(B)$2 is the judge label such as “Yes,” “No,” or “To some extent” (Kar et al., 27 Aug 2025).

Under the simple prompt setting, MathBuddy-feature “+” variants consistently outperformed their corresponding baselines. The reported highlights are LearnLM versus LearnLM+, where WR improved from $P(A\cap B) = P(A)P(B)$3 and overall DAMR mean improved from $P(A\cap B) = P(A)P(B)$4; LlemmaMM versus LlemmaMM+, where overall DAMR improved from $P(A\cap B) = P(A)P(B)$5; and QSLM versus QSLM+, where the overall score was $P(A\cap B) = P(A)P(B)$6 with strong gains in Human-likeness, Actionability, Coherence, and Tutor Tone. The overall summary reported in the paper is a $P(A\cap B) = P(A)P(B)$7 point WR improvement and a $P(A\cap B) = P(A)P(B)$8 point overall DAMR improvement (Kar et al., 27 Aug 2025).

Under the complex prompt setting, LearnLM+ achieved overall DAMR $P(A\cap B) = P(A)P(B)$9 versus $P(\bar{A}) = 1 - P(A)$0 for baseline, with consistent gains in Providing Guidance and Actionability, while smaller models showed variability under complex prompts. The LLM-as-Judge ensemble achieved the highest correlations across 4 of 8 dimensions and competitive results elsewhere. The ablation on Qwen2.5-7B is particularly important for interpretation: adding emotion alone could misguide a small model, while adding education-theory strategy mapping—explicitly summarized in the source as “motivate if bored,” “challenge if engaged”—yielded marked improvements, for example Prompt1 WR from $P(\bar{A}) = 1 - P(A)$1 (Kar et al., 27 Aug 2025).

The user study used a within-subject design with $P(\bar{A}) = 1 - P(A)$2 participants aged 15–55 across diverse demographics, two problems, and two conditions: Emotion OFF and Emotion ON. It included pre/post tests, a 15-item Likert survey after each session, a final satisfaction survey, and participant review and correction of emotion predictions. In the affect measurements, positive facial affect duration increased from mean $P(\bar{A}) = 1 - P(A)$3 in Emotion OFF to $P(\bar{A}) = 1 - P(A)$4 in Emotion ON, with Wilcoxon paired test $P(\bar{A}) = 1 - P(A)$5, while Neutral and Negative differences were not significant. Against participant gold labels, the system’s multimodal emotion prediction achieved Overall Accuracy $P(\bar{A}) = 1 - P(A)$6, Negative Recall $P(\bar{A}) = 1 - P(A)$7, Neutral Recall $P(\bar{A}) = 1 - P(A)$8, and Positive F1 $P(\bar{A}) = 1 - P(A)$9, directly exposing the neutral-suppression effect of the fusion rule. Participants also reported better learning experiences with Emotion ON, especially in Human-likeness, Actionability, Coherence, and Tutor Tone (Kar et al., 27 Aug 2025).

6. Ethics, limitations, and broader MathBuddy ecosystem

The reported system includes explicit privacy and ethics constraints. GDPR-related principles listed in the source are purpose limitation, data minimization, confidentiality, and transparency, together with informed consent and the right to withdraw. Facial images are used only for real-time inference and are not stored beyond inference; data are anonymized. Bias mitigation included mixed-age and neurodiverse faces in test sets, although continued work on demographic robustness is identified as necessary (Kar et al., 27 Aug 2025).

Several limitations are identified in the primary study. First, there is no gold standard for pedagogical quality, and user ratings may partly reflect UX rather than instruction quality. Second, child participation was limited, so generalization to K–12 remains open. Third, text plus face may be insufficient as the full affect signal; the documented future directions include audio prosody, handwriting, and biometrics, as well as replacing hard fusion with probabilistic or attention-based models. Fourth, smaller LLMs can be overwhelmed by complex prompts, motivating adaptive prompt compression or curriculum prompting (Kar et al., 27 Aug 2025).

Beyond the affective tutoring system itself, the supplied literature presents a broader technical ecosystem for MathBuddy. One proposed extension is step-level algebra diagnosis via math operation embeddings, in which each student step is represented as a transition in expression space. In that work, TE+C achieved $0$0 on OK, $0$1 on ERROR, and $0$2 on BUG for intended-operation identification, and the synthesis for MathBuddy states that TE+C achieves approximately $0$3 accuracy on BUG feedback classification (Zhang et al., 2021). Another proposed extension is semantic math retrieval over zbMATH Open and ARQMath-scale corpora using Mathematical Objects of Interest (MOIs), a design centered on text-plus-formula indexing, canonicalization, and learning-to-rank across BM25, formula similarity, MOI scoring, and entity linking (Scharpf et al., 2020).

A further strand concerns reading support rather than dialogue tutoring. Formula Evolution Map (FEM) and the PDF Reader with Math-Assistant (PRMA) model formula understanding as trajectory exploration over prerequisite and descendant formulas, then recommend Open Educational Resources by heterogeneous graph mining. In the reported evaluation, the best ranking model $0$4 achieved NDCG@3 $0$5, MAP $0$6, and MRR $0$7, while $0$8 of participants found FEM+OER precise/useful (Jiang et al., 2018). This suggests a MathBuddy variant oriented toward mathematical reading and concept dependency navigation rather than only conversational tutoring.

For math word problems, two additional design lines appear in the supplied material. SUMC-Solver reduces output non-determinism by mapping equivalent expressions to a structure-unified M-tree representation; it reported answer accuracy $0$9 on Math23K and $2$0 on MAWPS with PLM encoders (Wang et al., 2022). MeSys instead uses role-tagged logic forms and meaning-based inference, and reported $2$1 on AI2, $2$2 on IL, and about $2$3 on the noisy dataset NDS, with explicit robustness to irrelevant quantities (Liang et al., 2018). These systems indicate that a broader MathBuddy platform could combine affect-aware tutoring with dedicated symbolic or meaning-based subsolvers when the task is a structured word problem rather than open dialogue.

Tool augmentation offers a separate trajectory. MathSensei composes an LLM with Bing Web Search, Python plus Sympy, and the WolframAlpha API. Its best fixed sequence, PG+WA+SG, achieved $2$4 accuracy on MATH and was reported as a $2$5 absolute improvement over gpt-3.5-turbo with Chain-of-Thought on that benchmark, while also showing that tool augmentation was not as effective for simpler GSM-8K problems (Das et al., 2024). A plausible implication is that MathBuddy can be understood not as a single monolithic tutor, but as a family of architectures in which affect modeling, symbolic execution, retrieval, formula search, and educational scaffolding are composable modules specialized to different mathematical interactions.