MathReal: Real-Scene MLLM Math Benchmark

• MathReal is a benchmark that assesses multimodal LLMs on K–12 math problems captured in authentic mobile photos, incorporating OCR noise and real-world image degradations.
• It uses a carefully curated dataset of 2,000 Chinese math questions with varied challenges including blur, glare, rotation, and handwritten interference to test multi-step reasoning.
• The benchmark’s stratified evaluation highlights that even top-performing models struggle with integrated visual perception and mathematical reasoning under realistic conditions.

MathReal is a real-scene benchmark for evaluating multimodal LLMs (MLLMs) on K–12 mathematics problems presented as authentic mobile photographs rather than clean, curated images. Introduced in "MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal LLMs" (Feng et al., 8 Aug 2025), it comprises 2,000 Chinese mathematical questions with images captured by handheld mobile devices in authentic scenarios. Its defining premise is that practical educational deployment requires simultaneous robustness to OCR noise, figure perception errors, layout irregularity, and multi-step mathematical reasoning, all under the visual degradations common in real educational use.

1. Motivation and problem setting

MathReal was created to address a gap in multimodal mathematical evaluation: existing benchmarks largely used clean or processed images and therefore did not reflect the conditions under which students and parents actually photograph textbooks, worksheets, and exam papers. The benchmark is explicitly framed as a test of math reasoning under realistic visual acquisition noise, not merely a test of symbolic or diagrammatic reasoning in sanitized settings (Feng et al., 8 Aug 2025).

The underlying critique has four parts. First, prior benchmarks mostly used clean or processed images, which reduces the burden on OCR, layout parsing, and figure interpretation. Second, many earlier settings separate the question text from the image, whereas real K–12 use often embeds the entire question in a single photograph. Third, messy capture conditions such as blur, glare, rotation, shadows, perspective distortion, handwritten annotations, and reverse-side bleed-through are common in practice but were underrepresented in earlier benchmark design. Fourth, benchmark distributions had not adequately reflected the real user distribution of everyday educational assistance, where a model is expected to answer from imperfect smartphone photos (Feng et al., 8 Aug 2025).

This design choice makes MathReal a joint perception-comprehension-reasoning benchmark. A model must read the question from the image, interpret figures and markings, distinguish relevant from irrelevant visual content, and then solve the mathematical problem. A plausible implication is that MathReal measures the interaction of subsystems that are often decoupled in cleaner evaluation pipelines.

2. Corpus construction and annotation pipeline

The dataset construction began from a repository of 1.5 million photographed math questions. A two-stage filtering process was then applied. In the first stage, a domain-specific classifier selected math-related samples containing figures. In the second stage, GPT-4o, Doubao-1.5-vision-pro-32k, and Qwen2.5-VL-Instruct-72B independently checked whether each sample contained a single complete question, had a figure essential to solving the question, and was not a dialogue-style or irrelevant visual input. Only samples approved by all three models were retained (Feng et al., 8 Aug 2025).

After automated screening, a Gradio-based annotation system supported three fully manual stages. The first stage re-screened samples to remove incomplete questions, multi-question images, and irrelevant or decorative figures. The second stage annotated image conditions by assigning one or more real-scene degradation types. The third stage annotated question metadata, including question text, school level, question type, knowledge category, answer, figure description, and a clean image version when available. Each question was independently verified by at least two expert annotators (Feng et al., 8 Aug 2025).

The resulting corpus contains 2,000 questions. Average question length is 122.03 words and average answer length is 27.25 words. The benchmark therefore emphasizes not only visual realism but also substantive textual and reasoning load (Feng et al., 8 Aug 2025).

3. Dataset structure and real-scene taxonomy

MathReal organizes problems along three orthogonal axes: mathematical content, answer format, and educational difficulty. The five core knowledge and ability categories are Plane Geometry, Solid Geometry, Logical Reasoning, Function Graphs, and Statistical Charts. The three question types are multiple-choice, fill-in-the-blank, and constructed-response. The three difficulty levels are elementary school, middle school, and high school (Feng et al., 8 Aug 2025).

The benchmark also includes a fine-grained taxonomy of real-world image conditions. These are grouped into three primary categories with 14 subcategories. Image quality degradation includes blur, underexposure / overexposure, shadow coverage, and glare. Perspective variation includes rotation, in-plane tilt, non-planar capture, and background distortion. Irrelevant content interference includes handwritten questions, reverse-side content, question marking, figure marking, handwritten answers for multiple-choice or fill-in-the-blank questions, and handwritten process for constructed-response questions (Feng et al., 8 Aug 2025).

The principal corpus statistics are as follows.

Aspect	Subdivision	Count
Question type	Multiple-choice	104
Question type	Fill-in-the-blank	475
Question type	Constructed-response	1,421
Split	testmini questions	480
School level	Elementary	779
School level	Middle school	883
School level	High school	338
Visual structure	Only real images	745
Visual structure	Real images and clean images	1,255
Visual structure	Single figure	1,296
Visual structure	Multiple figures	704
Structure	Single sub-question	829
Structure	Multiple sub-questions	1,171

This structure matters analytically. Statistical charts are visually regular and often layout-stable, while function graphs and logical reasoning problems require tighter visual-symbol grounding. Likewise, constructed-response questions impose different demands from fill-in-the-blank or multiple-choice formats. The benchmark is therefore not merely a noise stress test; it is a stratified evaluation substrate for studying how visual degradation interacts with specific mathematical competencies (Feng et al., 8 Aug 2025).

4. Experimental settings and evaluation metrics

MathReal defines six experimental settings intended to disentangle perception from reasoning. The first is raw image only. The second, denoted $I\_\mathrm{UER}$, is unified end-to-end reasoning from the image, where OCR, figure understanding, and reasoning are performed in one interaction. The third is $I + QM$, image plus model-generated question text. The fourth is $I + QM + DM$, image plus model-generated question text and model-generated figure description. The fifth is $I + QG$, image plus ground-truth question text. The sixth is $I + QG + DG$, image plus ground-truth question text and ground-truth figure description (Feng et al., 8 Aug 2025).

These settings expose where failure occurs. Performance differences between raw-image settings and $I + QG$ isolate OCR-like extraction errors; differences between $I + QG$ and $I + QG + DG$ isolate figure-understanding deficits. This suggests a decomposition of multimodal math performance into recognition, comprehension, and reasoning components.

Two answer-level metrics are used. Strict accuracy requires all sub-answers in a question to be correct:

$$Acc_{str} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[ \forall j \in \{1, \dots, K_i\},\ a_{i,j}^{pred} \equiv a_{i,j}^{gt} \right].$$

Loose accuracy gives partial credit for partially correct multi-part responses:

$$Acc = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{K_i} \sum_{j=1}^{K_i} \mathbb{I}\left[ a_{i,j}^{pred} \equiv a_{i,j}^{gt} \right] \right).$$

An OCR-specific score is also defined:

$I + QM$0

The OCR formulation reflects the paper’s view that mathematical OCR is not reducible to character recognition alone; layout fidelity, numeric correctness, and semantic preservation are all material (Feng et al., 8 Aug 2025).

5. Empirical performance and error patterns

The central empirical finding is that MathReal is substantially harder than existing clean multimodal math benchmarks. On the main image-only evaluation, the best-performing model, Doubao-1.5-thinking-vision-pro, reaches 53.9% loose accuracy and 41.0% strict accuracy. The authors interpret this as evidence that current MLLMs remain significantly challenged in realistic educational contexts (Feng et al., 8 Aug 2025).

Closed-source models generally outperform open-source models, especially under noise, but both groups remain far from robust. Doubao-1.5-thinking-vision-pro is best overall, while Doubao-seed-1.6 and Gemini-2.5-pro-thinking are also strong. Open-source systems such as ERNIE-4.5-Turbo-VL-Preview and Qwen2.5-VL-72B trail behind. The paper also observes that reasoner-style models often perform especially poorly on raw real images, suggesting that reasoning-oriented inference stacks do not compensate for weak perception front ends (Feng et al., 8 Aug 2025).

Real visual noise strongly hurts performance. Blur, rotation, and handwritten interference all reduce accuracy. Several models improve markedly on clean images relative to real photos; Llama-4-Maverick improves by about $I + QM$1, and Claude-sonnet-4-thinking by about $I + QM$2. Conversely, Doubao-1.5-thinking-vision-pro and Gemini-2.5-flash-thinking perform better on real than clean images, which the authors interpret as possible adaptation to real-world texture and noise (Feng et al., 8 Aug 2025).

Difficulty varies across content categories. Statistical charts are generally easiest, whereas logical reasoning and function graphs are hardest. The paper attributes this to the structural regularity of charts, versus the abstract inference required by logical reasoning and the precise visual-symbol alignment required by function graphs. Question type also matters: fill-in-the-blank tends to have the highest overall accuracy, constructed-response is next, and multiple-choice is often surprisingly hard because it remains vision-centric and distractor-sensitive in this benchmark (Feng et al., 8 Aug 2025).

The error analysis separates failures into OCR error, figure perception error, calculation error, reasoning error, hallucination, and reject error. In an analysis of 100 failed cases per model, reasoning error is the most common category, accounting for over one-third of failures. OCR and figure perception errors together contribute roughly 40–50% of failures. The benchmark therefore indicates that weaknesses are distributed across both visual parsing and downstream mathematical inference; improving only one stage is insufficient (Feng et al., 8 Aug 2025).

6. Significance, misconceptions, and name ambiguity

MathReal’s main significance lies in shifting evaluation from idealized multimodal math inputs to authentic educational photos. It therefore challenges a common misconception: strong performance on clean multimodal benchmarks does not imply reliable performance in real educational deployment. The benchmark also complicates another assumption, namely that multiple-choice should be easy; in MathReal, multiple-choice often remains visually demanding and error-prone because perception and discrimination dominate the task (Feng et al., 8 Aug 2025).

The authors propose several directions for improvement: more robust visual encoders, better OCR under noise, stronger handling of perspective and layout distortions, improved robustness to handwritten interference and image artifacts, and architectures that jointly handle perception, parsing, and multi-step reasoning. This suggests that future progress will likely require cross-stack optimization rather than isolated gains in either OCR or symbolic reasoning (Feng et al., 8 Aug 2025).

The name is also ambiguous in the broader arXiv literature. In Klazar’s work on Countable Mathematical Analysis, the paper’s “MathReal” construction denotes HMC-compatible real numbers defined as rational Cauchy sequences themselves, with $I + QM$3 and asymptotic equality $I + QM$4 used as the equality surrogate (Klazar, 2023). In another distinct context, the query label “MathReal” is applied to real-valued quantum mechanics, whose universal validity is experimentally challenged in a three-party entanglement-swapping test under strict locality conditions; the reported value $I + QM$5 exceeds the real number bound $I + QM$6 by 5.30 standard deviations (Wu et al., 2022). These usages are conceptually unrelated to the MLLM benchmark.

In contemporary multimodal evaluation, however, MathReal most specifically denotes the real-scene K–12 benchmark that tests whether MLLMs can solve mathematics from noisy, handheld mobile photos. Its contribution is not a new mathematical formalism, but an evaluation regime in which recognition, comprehension, and reasoning are inseparable.