EGE-Math Solutions Assessment Benchmark

• EGE-Math Solutions Assessment Benchmark is a testbed for VLMs to grade handwritten math solutions by analyzing intermediate reasoning and detecting errors.
• It uses 122 scanned EGE math exam solutions annotated with official expert scores across diverse domains and rubric-based partial-credit criteria.
• Empirical results reveal VLM limitations in rubric alignment, handwriting recognition, and contextual reasoning for reliable automated grading.

Searching arXiv for the benchmark and closely related assessment/benchmark papers. I’m unable to access the arXiv search tool in this session, so I’m relying on the arXiv records and data provided here, citing them directly. The EGE-Math Solutions Assessment Benchmark is a benchmark for evaluating vision-LLMs (VLMs) on the assessment of handwritten mathematical solutions, rather than on the generation of solutions from scratch. Introduced in "CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam" (Khrulev, 29 Jul 2025), it is built from scanned solutions from the Russian Unified State Exam (EGE) in mathematics and uses official expert grades and task-specific rubrics as ground truth. Its central object of evaluation is not answer production but process analysis, including the reading of handwritten work, the interpretation of intermediate reasoning, the detection of mistakes or omissions, and the assignment of partial-credit scores aligned with official human grading.

1. Definition, scope, and assessment target

The benchmark is designed to measure whether a VLM can perform five linked functions: read handwritten mathematical work from scanned images, understand the student’s reasoning process, detect mistakes or missing steps, assign a score according to a fixed rubric, and align its judgment with official human expert grading (Khrulev, 29 Jul 2025). In this formulation, the model acts as a grader rather than as a solver.

This distinguishes the benchmark from standard mathematical reasoning datasets such as MATH, GSM8K, or MathVista, which primarily test whether a model can produce the correct answer to a problem (Khrulev, 29 Jul 2025). The paper explicitly frames the task as a shift from answer verification to process analysis and diagnostic assessment. The benchmark therefore targets a capability that is structurally closer to educational evaluation than to conventional math QA.

The assessment problem is also tied to the realities of high-stakes examination. In such settings, graders must decide whether a method is valid, whether intermediate steps are correct, whether an incorrect final answer reflects a minor computational slip or a deeper conceptual error, and how much partial credit is warranted (Khrulev, 29 Jul 2025). This makes rubric alignment a primary criterion. A plausible implication is that benchmark success depends not only on mathematical competence, but also on the ability to operationalize institutional scoring conventions.

2. Dataset composition and task structure

The benchmark contains 122 scanned handwritten solutions from the Russian Unified State Exam (EGE) in mathematics (Khrulev, 29 Jul 2025). Its source is the official EGE expert guide / grading materials. The core supervision consists of official expert-assigned grades, and for many tasks the dataset additionally includes the problem statement, the official score, a reference or “gold standard” solution, and the task-specific rubric / scoring criteria (Khrulev, 29 Jul 2025).

The 122 solutions are distributed across seven EGE task types:

Task ID	Domain	Count / Score Range
13	Trigonometric equations	21 / 0–2
14	Stereometry	18 / 0–3
15	Logarithmic inequalities	19 / 0–2
16	Financial mathematics problems	17 / 0–2
17	Planimetry	15 / 0–3
18	Parameterised equations	16 / 0–4
19	Number theory / combinatorics	16 / 0–4

The task distribution spans algebraic, geometric, financial, and combinatorial material, but always under the same operational condition: the model must grade a completed handwritten solution rather than solve a fresh prompt (Khrulev, 29 Jul 2025). This matters because the input modality is heterogeneous. It combines handwritten symbolic notation, natural-language explanation, and, in some cases, diagrammatic or spatial content. The paper reports that geometry tasks such as stereometry and planimetry were harder and showed poorer agreement with human graders, which the authors attribute to difficulty in interpreting free-hand diagrams, spatial relations, and geometric reasoning chains from handwriting and figures (Khrulev, 29 Jul 2025).

3. Rubrics, partial credit, and grading semantics

A defining feature of the benchmark is its use of official human expert scoring according to official exam criteria (Khrulev, 29 Jul 2025). The benchmark is therefore not grounded in approximate annotation or crowd judgments, but in formal grading practice. This is especially important because the central challenge is not binary correctness but partial correctness.

The paper gives a concrete rubric example for Task 18, which uses a 0–4 scale (Khrulev, 29 Jul 2025). The rubric is summarized as follows:

• 4 points: correct, complete, well-reasoned solution
• 3 points: correct set of parameter values, but only differs from the correct set by boundary-point inclusion/exclusion
• 2 points: correct interval obtained with correct reasoning but possibly wrong boundary points, or a computational mistake despite correct logic
• 1 point: roots are found and the problem is reduced correctly, but further investigation is incomplete or wrong
• 0 points: none of the above

This rubric exemplifies the benchmark’s semantics. The model must discriminate among several nearby states of correctness: exact solution, boundary-point error, computational error under correct logic, and incomplete but substantively relevant reduction. Such distinctions are typical of human mathematical grading and are not reducible to final-answer matching.

This feature also places the benchmark in continuity with broader critiques of overly reductive mathematics assessment. "Assessments in Mathematics, undergraduate degree" argues that mathematical assessment should not be reduced to fast, automated marking when the goal is to evaluate reasoning, writing, and conceptual understanding, and it states that MCQs can test knowledge and understanding, but not the ability to reason, justify, write, or argue mathematically (David, 2017). The EGE-Math benchmark’s emphasis on handwritten production, partial credit, and rubric-based interpretation is consistent with that assessment perspective. This suggests that the benchmark is not merely a multimodal engineering task, but also an instantiation of a specific theory of mathematical validity in assessment.

4. Evaluation protocol and performance measures

The evaluation in (Khrulev, 29 Jul 2025) covers seven VLMs in three inference modes. The tested models are:

1. Arcee AI Spotlight
2. Google Gemini 2.0 Flash
3. Google Gemini 2.0 Flash Lite
4. Google Gemini 2.5 Flash Preview
5. Google Gemini 2.5 Flash Preview:thinking
6. OpenAI o4-mini
7. Qwen 2.5 VL 32B

The three inference modes are:

• Without Answer: the model receives the handwritten solution image and the problem statement
• With Answer: the model receives the handwritten solution image, the problem statement, and the correct final numerical answer
• With True Solution: the model receives the handwritten solution image, the problem statement, the correct final answer, and a full reference / gold-standard solution

These modes probe whether additional grading context improves rubric alignment. The benchmark therefore tests not only standalone assessment ability, but also the model’s capacity for comparative grading and conditional use of supervision.

The paper evaluates three main metrics (Khrulev, 29 Jul 2025). Accuracy (Exact Match) is the percentage of cases where the predicted score exactly matches the expert score. Quality Score is a normalized closeness measure that rewards predictions near the correct score, even if not exact. Average Score Distance is the mean absolute difference between predicted and true scores. Together, these measures distinguish exact rubric agreement from approximate grade proximity.

The strongest result reported is for OpenAI o4-mini in With Answer mode, with Accuracy: 56.56%, Quality Score: 78.17%, and Average Score Distance: 0.60 (Khrulev, 29 Jul 2025). The paper identifies this as the best result across the benchmark. It also reports that Google Gemini 2.0 Flash and Google Gemini 2.5 Flash Preview performed reasonably well, while Arcee AI Spotlight and Qwen 2.5 VL 32B generally had lower accuracy and larger score distances (Khrulev, 29 Jul 2025).

5. Empirical findings and observed failure modes

The principal empirical conclusion is that current VLMs remain limited in mathematical reasoning and human-rubric alignment when grading handwritten exam solutions (Khrulev, 29 Jul 2025). Even the best exact-match performance leaves substantial disagreement with expert scorers. The benchmark therefore identifies assessment as a harder problem than answer production alone.

One major finding is that extra context does not always help (Khrulev, 29 Jul 2025). Some models improved when given the correct answer or a reference solution, but others showed little benefit or even degradation. The paper gives specific examples: Gemini 2.0 Flash improved notably when given the correct answer, whereas Arcee AI Spotlight got slightly worse with more context. It further notes that With True Solution was not always the best mode. This indicates that current systems do not yet exhibit robust comparative reasoning over student work and expert solutions.

The Gemini 2.5 Flash Preview:thinking variant did not consistently outperform the regular version (Khrulev, 29 Jul 2025). The paper interprets this as evidence that increased deliberative runtime or nominally deeper reasoning does not automatically yield better rubric-based grading. This is an important result because it separates generic reasoning depth from institutional scoring competence.

The paper identifies several recurrent limitations (Khrulev, 29 Jul 2025). Handwriting/OCR difficulty produces recognition errors that propagate into grading mistakes. Weak rubric alignment leads models to mishandle partial credit and boundary cases. Difficulty with comparative reasoning persists even when the correct answer or full solution is supplied. The benchmark is also constrained by a small dataset of 122 solutions and by mostly zero-shot evaluation, meaning that models were not optimized specifically for this task. Finally, cost-performance trade-offs matter for practical use.

The appendix includes a representative parameter problem in which most models assigned the correct score, but others either overestimated the score or underestimated it badly, and some failed to complete a coherent grading response (Khrulev, 29 Jul 2025). The significance of this example is diagnostic rather than anecdotal: model failures are heterogeneous, arising from rubric misinterpretation, incomplete reasoning, or response-generation breakdowns.

6. Relation to other mathematical benchmarks and assessment traditions

The EGE-Math benchmark occupies a distinct position among mathematical evaluation suites. In GeoEval, the task is geometry problem-solving for LLMs and multi-modal models, measured by accuracy on final answers across text-only, text+diagram, and challenge subsets (Zhang et al., 2024). GeoEval stresses backward reasoning, rephrasing robustness, contamination sensitivity, and difficulty scaling, but its target remains solution production rather than grading. In DocMath-Eval, the task is document-grounded numerical reasoning in long financial reports, requiring retrieval of evidence from text and tables and then generation of a correct numeric answer (Zhao et al., 2023). Again, the emphasis is on solving rather than assessment.

By contrast, EGE-Math evaluates whether a model can judge how good a student’s solution is, and why (Khrulev, 29 Jul 2025). The output is a grade / score, not a final numerical answer, and correctness is defined by alignment with official human expert rubrics. This makes the benchmark closer to educational assessment than to standard reasoning leaderboards.

The benchmark also resonates with pedagogical arguments against reducing mathematics evaluation to item formats that privilege recognition over production. The undergraduate assessment study in (David, 2017) concludes that MCQs are “totally unsuitable” for mathematics assessment except for mechanical or purely computational tasks, and that classical written subjects remain the best option for reasoning, writing, or genuine mathematical production. The EGE-Math benchmark does not itself make that curricular claim, but it is built precisely around written, step-by-step solutions and official scoring criteria. A plausible implication is that it operationalizes, in benchmark form, the distinction between computational correctness and the evaluation of mathematical production that (David, 2017) argues is central to valid assessment.

Two misconceptions are explicitly countered by this comparison. The first is that strong performance on math-solving benchmarks implies strong performance on math assessment benchmarks. EGE-Math rejects that equivalence by construction (Khrulev, 29 Jul 2025). The second is that supplying the correct answer or a gold solution necessarily resolves the assessment problem. The reported mode effects show that this is false for current models (Khrulev, 29 Jul 2025).

7. Applications, limits, and research directions

The paper presents the benchmark as a step toward AI as a grading assistant, not as a replacement for teachers (Khrulev, 29 Jul 2025). The applications it discusses include helping teachers triage student work, providing preliminary grading suggestions, detecting likely mistakes, supporting rubric-based assessment, and improving feedback quality. At the same time, it states that current models are not yet reliable enough for fully autonomous grading in high-stakes settings (Khrulev, 29 Jul 2025).

The future research directions identified in the paper are concrete. These include hybrid systems combining VLMs with specialized handwritten-math recognition (HMER), fine-tuning on rubric-based grading data, larger and more diverse datasets, better reasoning over student work vs. reference solutions, more explainable grading outputs, adaptation to other curricula and countries, and interactive or human-in-the-loop grading systems (Khrulev, 29 Jul 2025).

In this sense, the benchmark functions as both an evaluation resource and a problem specification. It defines a testbed in which progress would require simultaneous advances in multimodal parsing, symbolic and informal mathematical reasoning, rubric semantics, and alignment with expert human judgment. Its strongest contribution is therefore not only the dataset of 122 scanned handwritten EGE solutions, but the precise formulation of grading as a benchmarkable capability: a model must read, interpret, diagnose, and score mathematical work under official criteria, and current systems remain substantially below the level required for dependable autonomous use (Khrulev, 29 Jul 2025).