DrawEduMath: Handwritten Math Benchmark

• DrawEduMath is a benchmark dataset of K–12 student handwritten math work designed to evaluate vision-language models’ ability to interpret and reason in realistic educational settings.
• The dataset comprises 2,030 images with rich annotations including 11,661 teacher-written and 44,362 synthetic QA pairs to support automated feedback and detailed error analysis.
• Evaluation tasks such as image description, visual Q&A, error diagnosis, and feedback generation highlight key challenges in model performance on noisy, real-world student work.

DrawEduMath is a publicly released benchmark and dataset specifically designed for evaluating the ability of vision–LLMs (VLMs) to interpret and reason over K–12 students’ hand-drawn mathematical work in realistic educational settings. Distinct from datasets of clean, typed content, DrawEduMath targets the domain of naturalistic, noisy, and often idiosyncratic handwritten student responses encountered in classrooms and digital learning platforms. It serves as a testbed for advancing VLMs in tasks ranging from automated feedback to pedagogically-oriented error analysis, bringing together richly annotated real-world student work and expert-designed evaluation protocols (Baral et al., 24 Jan 2025, Lucy et al., 1 Mar 2026).

1. Dataset Construction and Annotation Protocols

DrawEduMath comprises 2,030 distinct images, each typically consisting of a K–12 math problem (source: Eureka Math, Open Up Resources, Illustrative Math) juxtaposed with a student’s handwritten response. The problems cover grades 2–12 and span key mathematical domains: arithmetic, operations and fractions, geometry (including transformations and sketches), ratios and proportions, the number system, and symbolic expressions. The dataset reflects authentic classroom variations, with mixed handwriting styles, variable pen and pencil line quality, multi-modal diagrams (arrays, number lines, tape diagrams, geometric sketches), and prevalent real-world noise such as lighting artifacts and cropping.

Annotation involved three experienced K–12 teachers generating free-form textual descriptions for each image with the goal of reproducibility—providing sufficient detail for a peer educator to reconstruct the student’s work from the narrative. Eight additional teachers then authored 11,661 teacher-written question–answer (QA) pairs, yielding an average of 5.74 QAs per image. These questions range across problem-solving strategy (“What method did the student use?”), compositional features (“How was the array labeled?”), and diagnostic checks for misconceptions (“Are the place-value columns aligned correctly?”).

To increase scale and diversity, synthetic QAs were generated from teacher descriptions using LLMs (Claude 3.5 Sonnet, GPT-4o) in a two-stage pipeline: (1) decompose the teacher-written description into atomic facets reflecting individual concepts, and (2) transform each into a closed-ended question and short answer. This yielded 44,362 synthetic QAs, with expert review indicating >90% answerability and >85% correct binary answers.

Quality assurance was embedded at multiple stages. Spot-checks during QA generation showed inter-annotator agreement with Cohen’s κ > 0.8 for answerability and ≈ 0.75 for correctness. Median edit distances on descriptions reveal stable, reproducible annotation. Approximately 334 images were discarded for excessive blur and 4 for residual identifiers, resulting in a curated, privacy-preserving collection (Baral et al., 24 Jan 2025).

2. Task Design and Evaluation Methodologies

DrawEduMath defines several core tasks to probe VLMs’ capabilities on pedagogically relevant axes:

1. Image Description Quality: Models must generate natural language descriptions of student responses, matching teacher-crafted references.
2. Visual Question Answering (VQA): Closed-ended questions about labels, strategies, diagram elements, and correctness are posed, using either teacher-written or synthetic QAs.
3. Error Diagnosis: Classification of student responses for correctness, identification of specific misconceptions, or detection of procedural errors.
4. Feedback Generation: Generation of hints or corrective statements aligned to curriculum standards.

Evaluation metrics include accuracy for binary/categorical tasks, BLEU-n and METEOR for short-answer and feedback generation, and BERTScore/embedding-based similarity as well as LLM-based pairwise judgments (e.g. Mixtral 8x22B) for open-ended outputs. For rigorous benchmarking, a recommended split is 70/15/15% (train/validation/test), stratified by problem ID to prevent information leakage among student responses for the same underlying problem.

State-of-the-art closed models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) are compared to open-source baselines (Llama 3.2-11B Vision Instruct). Input images are minimally cropped to preserve all relevant student work, with all models receiving the full problem plus response at up to 1024 px width (Baral et al., 24 Jan 2025).

3. Model Performance, Failure Modes, and Error Analysis

Performance on DrawEduMath reveals persistent challenges for contemporary VLMs. Even top proprietary models achieve only 60–70% accuracy on teacher-written QAs evaluating correctness and misconceptions, with the most difficult questions (e.g., “What error did the student make?”) yielding F₁ scores often below 0.5. On synthetic QAs, although absolute accuracy is higher and model rankings mirror those found on teacher-written QAs (Spearman ρ ≈ 0.8), this is attributed to the synthetic questions’ focus on low-level composition rather than high-level reasoning.

Substantial failure modes include:

• Default-to-correct bias: On student work containing errors, models frequently provide answers more characteristic of the correct solution than the observed erroneous one. Analysis indicates that 29–35% of model mistakes on erroneous work replicate the “majority correct” solution for problem-matched responses, suggesting brittle adaptation to unexpected input (Lucy et al., 1 Mar 2026).
• Sensitivity to Pen-and-Paper Artifacts: Models misread low-contrast images, invert foreground/background on chalkboard photos, or overlook essential marks such as ticks or arrows.
• Prompt Overreliance: Responses to strategy questions often reflect canonical procedures rather than diagnosing the student’s actual (potentially flawed) approach.

Across 11 VLMs benchmarked in 2026, an 8–10 percentage point accuracy gap is consistently observed between student work labeled as error-free and that exhibiting mistakes. On content-description QAs, this gap is robust, with β₁ coefficients ranging from 0.0585 to 0.1013 (all p < 1e⁻¹²) in OLS regressions indicating significant main effects of student correctness (Lucy et al., 1 Mar 2026).

4. Integration with Advanced Handwriting Understanding and Visualization Systems

DrawEduMath is positioned for integration with state-of-the-art handwriting mathematics evaluation models. VEHME exemplifies such a system with a vision–language backbone (Qwen2.5-VL-7B-Instruct or similar), an expression-aware visual prompting module for localizing handwritten math tokens (Yolov11 backbone), and a structured reasoning output format including step-by-step chains-of-thought, correctness labels, and error localization fields. VEHME’s two-phase supervised fine-tuning and reinforcement learning regime (Group Relative Policy Optimization) optimizes for correctness, detailed localization, and interpretability of reasoning traces (Nguyen et al., 26 Oct 2025). Key assessment metrics aligned with DrawEduMath include error detection (ED) and error localization (EL) accuracy and F₁, as well as correlation with human teacher scoring.

Complementing assessment, automated diagram generation is supported by pipelines that use LLMs to convert math hints and textual descriptions into scalable vector graphics (SVGs). Key workflow steps include in-context learning prompts, strict SVG schema enforcement, and visual question answering verification loops. These automated pipelines yield high diagrammatic fidelity and support scalable, accurate mathematical visualizations within the DrawEduMath ecosystem (Lee et al., 10 Mar 2025).

Systems such as Wisdom Computing Perspective introduce dense, step-by-step visualizations for matrix operations and other mathematical calculations. Handwritten input recognition is achieved via a Mamba backbone, YOLO-Mamba with CoordAttention, and dynamic matrix reconstruction, with detailed animation of calculation steps using Manim. The entire recognition to visualization workflow can be modularly integrated into DrawEduMath, supporting real-time feedback and interactive understanding of mathematical structure (Yu, 2 May 2025).

5. Geometry Construction, Reasoning, and Proof Integration

DrawEduMath supports advanced geometry tasks by leveraging constraint-based interactive construction engines and externalized reasoning loops. The Draw2Think paradigm enables agentic LLM interaction with the GeoGebra engine via a Propose–Draw–Verify loop: the LLM proposes construction or query actions, the GeoGebra engine executes constraint-checked algebraic operations, and the results feed back for the next reasoning cycle (Hu et al., 20 May 2026). This enables auditability via construction fidelity (fraction of ground-truth predicates realized in the canvas) and measurement faithfulness (precision of derived geometric quantities), with predicate- and problem-level success metrics. Intermediate states are fully externalized, allowing direct measurement and stepwise feedback.

"Proofs without words" functionality builds on the Geometric Deduction Database and JGEX reasoning layer integrated with GeoGebra. Users construct diagrams in-browser, invoke the proof engine, and receive animated, slider-driven demonstrations of stepwise geometric inference, with each step expressed in LaTeX and dynamically visualized (Thaller et al., 2022).

6. Multimodal Tutoring, Whiteboard Feedback, and Interaction Paradigms

Enabling collaborative, multimodal tutoring is a central design goal for DrawEduMath. The Interactive Sketchpad system demonstrates the effectiveness of tightly coupled text–visual feedback, where large multimodal models fine-tuned for hint and visualization tasks interactively guide learners through successive problem-solving steps. Diagram generation is performed via code-execution (e.g., matplotlib, scipy) within the reasoning loop; both the LMM and code interpreter contribute to the accuracy and clarity of generated visuals (Chen et al., 12 Feb 2025). The user experience is centered on an interactive whiteboard where students can annotate over model-generated diagrams; each loop through problem, hint, diagram, and sketch is designed to maximize engagement and conceptual understanding.

Empirical studies on such multimodal systems show improved task comprehension, solution accuracy, and learner engagement, highlighting the importance of incremental, visual scaffolding over text-only or final-answer approaches.

7. Limitations, Future Directions, and Open Challenges

DrawEduMath benchmarks reveal that VLMs significantly underperform on tasks involving error diagnosis, especially in the context of real, struggling student work. These deficiencies are linked to training data biases (models overly exposed to correct solutions), prompt sensitivity (tendency to hallucinate correct answers), and visual recognition limitations in low-quality or artifact-laden inputs (Lucy et al., 1 Mar 2026, Baral et al., 24 Jan 2025). To address these gaps, research calls for:

• Inclusion of error-rich and procedurally diverse training examples, including synthetic perturbations representing common student mistakes.
• Optimizing directly for pedagogical tasks such as error localization, misconception categorization, and targeted feedback suggestion, rather than merely solving canonical math problems.
• Multitask objective fine-tuning, integrating VQA, error classification, graphical step-tracking, and generation tasks.
• Expansion of DrawEduMath to multilingual data, new grade levels and problem types, and deeper feedback taxonomies for nuanced instructional interventions.

The current ecosystem offers a platform for continual evaluation and model development, with direct implications for real-time, reliable, pedagogically meaningful math tutoring at classroom scale (Baral et al., 24 Jan 2025, Lucy et al., 1 Mar 2026).