From Answer Generators to Reasoning Facilitators: Designing AI Tutors for Mathematical Reasoning in High-Stakes Environments

Abstract: The rapid integration of LLMs into educational technology threatens to reduce mathematical learning to mere answer generation. This paper presents a generative study, usability study, and 12-participant field deployment of AITutor, an interactive system that translates theoretical pedagogical mechanisms into concrete user interface features. We explore how junior-high students preparing for high-stakes exams (Zhongkao) interact with AI tutoring. Through mixed-methods triangulation (7,379 telemetry events, 8 contextual observations, 10 interviews), we reveal that students actively resist traditional Socratic dialogue under time pressure, repurposing "answer-first" shortcuts as vital diagnostic checkpoints. We demonstrate how features like layered worked examples, step-linked visual grounding, and metacognitive scaffolding lower the interaction cost of reasoning repair. We contribute a "Reasoning-Centered Product Loop," offering actionable implications for designing AI that structurally supports the inspection, local repair, curriculum verification, and delayed retrieval of mathematical reasoning in the wild.

• The paper introduces AI tutor prototypes that shift focus from answer generation to reasoning facilitation using layered worked examples and metacognitive scaffolding.
• It employs mixed-methods research in high-stakes exam contexts to analyze authentic help-seeking workflows and diagnose students' interaction patterns.
• Findings reveal that curriculum alignment and step-linked visual grounding are critical for reducing cognitive costs and building trust in AI math assistants.

AI Tutors for Mathematical Reasoning: From Answer Generators to Reasoning Facilitators

Context and Motivation

The integration of LLMs into educational technology for mathematics has spawned a wave of AI-based tutoring solutions. While legacy Intelligent Tutoring Systems (ITS) supported stepwise, inspectable feedback, recent AI math-assistants—especially in high-pressure exam environments—tend to deliver polished, monolithic solutions, neglecting pedagogical mechanisms known to foster durable mathematical learning such as cognitive struggle, self-explanation, and metacognitive repair. This work examines the sociotechnical tension between these opposing demands—speed and curriculum-aligned answer verification versus reasoning depth—and situates itself within the empirical realities of China's highly competitive Zhongkao exam preparation environment.

The systemic review of Chinese AI math-help products highlights the current competitive landscape, revealing that existing solutions optimize for either speed, flexibility, or diagnostic review, but rarely achieve all three concurrently.

Figure 1: Landscape analysis of Chinese AI math-help products, each optimizing one axis of the speed-reasoning-design space, but none systematically supporting stepwise reasoning inspection or repair.

Generative Study: Help-Seeking Under High-Stakes Constraints

Through mixed-methods research—including interviews, observations, and telemetry—this study elucidates authentic help-seeking workflows among junior-high students. The analysis identifies four core user profiles: the "Missing Step Hunter," "Cautious Curriculum Validator," "Efficiency-Driven Answer Checker," and "Transfer-Oriented Learner," each representing distinct cognitive and motivational stances toward AI help.

A decisive finding is that answer-first behaviors, often dismissed as "gaming the system," in reality underpin rational cognitive regulation strategies in time-constrained contexts—students need rapid orientation before committing to deeper reasoning investments. The demand for curriculum-constrained methods and multimodal grounding is particularly acute in geometry, where unfounded visual moves or out-of-scope mathematical methods break both trust and engagement.

System Design: AITutor as a Reasoning Facilitator

The AITutor prototype operationalizes three theoretically grounded interventions: Layered Worked Examples, Step-Linked Visual Grounding, and Metacognitive Scaffolding.

• Layered Worked Examples: Solutions are mediated through a progressive disclosure UI: concise overviews for curriculum-fit checking, labeled sub-questions, granular expandable explanations, and exam-format constraints. This design allows students to access the answer as a diagnostic checkpoint, then pivot into steps, substeps, or visualizations as needed.
• Step-Linked Visual Grounding: Key for spatial and geometric reasoning, textual steps are dynamically synchronized with diagrammatic highlights, mitigating split-attention effects.
• Metacognitive Scaffolding: Local, contextually anchored prompts offer one-tap follow-ups (e.g., "Explain this step"), minimizing the interaction cost of reasoning repair in contrast to generic chatbot interfaces.

Field Deployment and Behavioral Analysis

Over a 12-day field deployment involving 12 instrumented students, the study collected 7,379 backend events across 104 sessions, complemented by interviews and contextual observations.

Key Quantitative Results:

• System activity spiked during weekends and homework bursts, indicative of just-in-time, need-based engagement rather than habitual use.

  Figure 2: System usage tracks periods of homework intensity, reflecting opportunistic rather than routine AI engagement.

• The core solve-completion funnel exhibited a 56.4% success rate across all initiated solves, with abandonment often attributable to technical latency (mean latency: 32.0s; 90th percentile: 68.5s) and system stability events.

  Figure 3: Funnel analysis of problem-solving attempts—latency and system failures are major bottlenecks independent of user disinterest.

• Follow-up and transfer-practice features saw modest uptake (∼8–10% immediate utilization), suggesting tension between the theoretical value of metacognitive scaffolding and the immediate practical priorities of homework completion.

Figure 4: Funnel for follow-up and transfer-practice features—initiation rates are low, with all submitted follow-ups receiving answers.

• Students preferentially saved more difficult or uncertain problems for later review, validating the demand for delayed retrieval workflows.

  Figure 5: Distribution of history-save targets—saved interactions cluster on difficult items, evidencing implicit retrieval-practice strategies.

Strong Claims:

• The data refute the assumption that answer-first behaviors are indicative of shallow learning in high-stakes environments. Instead, these serve as diagnostic orientation and error localization mechanisms.
• Trust in AI explanations is not grounded in surface correctness but in explicit curriculum-fit. Methods not strictly aligned with syllabus scope or exam-conventional formats are summarily rejected, even if mathematically correct.
• Visual grounding and local step repair are not auxiliary features but essential for the usability and trustworthiness of AI math assistance in authentic practice.

Theoretical and Practical Implications

The authors propose a "Reasoning-Centered Product Loop" as a normative framework. Central functions for AI math-assistants are: (i) rapid orientation (answer checkpointing), (ii) coordinated visualization of reasoning steps, (iii) localized, frictionless reasoning repair, (iv) delayed, structured retrieval for transfer, and (v) curriculum-based trust calibration.

This approach mandates that AI educational interfaces move beyond answer generation or superficial Socratic dialogue, focusing instead on structurally lowering the cost of reasoning inspection, repair, and long-term retrieval—without adding metacognitive burdens to learners.

The study provides evidence that simply exposing AI-explanations or increasing transparency does not suffice for calibrated trust; explicit signals regarding knowledge scope, extracted conditions, and syllabus alignment are preconditions for student engagement.

Implications for Future AI Tutoring Systems:

• Dual-mode explanation loading (instant "Answer+Core Strategy" followed by stepwise elaboration) is necessary to bridge latency sensitivity and cognitive depth.
• Automated organization of solved problems by knowledge point ("wrong-book" workflows) democratizes the metacognitive scaffolding typically reserved for students with access to private tutors.
• Localization of help-seeking affordances to explanation steps dramatically reduces help-seeking friction without encouraging unproductive shortcutting.

Limitations and Future Research Directions

• The short field deployment and small sample size preclude population-level generalization. The interplay of novelty effects and technical outages with observed patterns limits the inferential scope.
• No direct assessment of learning gains, transfer, or long-term retention was conducted. Future work should integrate controlled pre/post assessment and measure whether durable reasoning transfer occurs, particularly for students operating without high-resource supports.
• Telemetry limitations conflate system reliability and user abandonments, suggesting the need for more nuanced instrumentation that captures partial progress and parallel action during AI waits.

Conclusion

This study systematically evidences that robust UI and interaction design, grounded in the learning sciences, are prerequisites for transforming LLMs from answer generators into authentic reasoning facilitators in mathematics education. By lowering the cognitive and operational cost of reasoning inspection, local repair, and delayed retrieval, AI tutors can democratize access to high-quality metacognitive scaffolding—an effect with major implications for educational equity, particularly in high-stakes exam environments. The Reasoning-Centered Product Loop unifies these insights, providing a blueprint for the next generation of AI-enhanced learning systems—not only in mathematics but across domains where reasoning, not mere answers, remains the goal.

---

Reference:

"From Answer Generators to Reasoning Facilitators: Designing AI Tutors for Mathematical Reasoning in High-Stakes Environments" (2607.01692)