---
title: EGE-Math Solutions Assessment Benchmark
url: https://www.emergentmind.com/topics/ege-math-solutions-assessment-benchmark
type: topic
---

# EGE-Math Solutions Assessment Benchmark

Searching arXiv for the benchmark and closely related assessment/benchmark papers.
I’m unable to access the arXiv search tool in this session, so I’m relying on the arXiv records and data provided here, citing them directly.
The **EGE-Math Solutions Assessment Benchmark** is a benchmark for evaluating **vision-language models (VLMs)** on the assessment of **handwritten mathematical solutions**, rather than on the generation of solutions from scratch. Introduced in "CHECK-MAT: Checking Hand-Written Mathematical Answers for the Russian Unified State Exam" [2507.22958], it is built from scanned solutions from the **Russian Unified State Exam (EGE)** in mathematics and uses **official expert grades** and task-specific rubrics as ground truth. Its central object of evaluation is not answer production but **process analysis**, including the reading of handwritten work, the interpretation of intermediate reasoning, the detection of mistakes or omissions, and the assignment of **partial-credit scores** aligned with official human grading.

## 1. Definition, scope, and assessment target

The benchmark is designed to measure whether a VLM can perform five linked functions: **read handwritten mathematical work from scanned images**, **understand the student’s reasoning process**, **detect mistakes or missing steps**, **assign a score according to a fixed rubric**, and **align its judgment with official human expert grading** [2507.22958]. In this formulation, the model acts as a grader rather than as a solver.

This distinguishes the benchmark from standard mathematical reasoning datasets such as **MATH**, **GSM8K**, or **MathVista**, which primarily test whether a model can produce the correct answer to a problem [2507.22958]. The paper explicitly frames the task as a shift from **answer verification** to **process analysis** and **diagnostic assessment**. The benchmark therefore targets a capability that is structurally closer to educational evaluation than to conventional math QA.

The assessment problem is also tied to the realities of high-stakes examination. In such settings, graders must decide whether a method is valid, whether intermediate steps are correct, whether an incorrect final answer reflects a minor computational slip or a deeper conceptual error, and how much partial credit is warranted [2507.22958]. This makes rubric alignment a primary criterion. A plausible implication is that benchmark success depends not only on mathematical competence, but also on the ability to operationalize institutional scoring conventions.

## 2. Dataset composition and task structure

The benchmark contains **122 scanned handwritten solutions** from the **Russian Unified State Exam (EGE)** in mathematics [2507.22958]. Its source is the **official EGE expert guide / grading materials**. The core supervision consists of **official expert-assigned grades**, and for many tasks the dataset additionally includes the **problem statement**, the **official score**, a **reference or “gold standard” solution**, and the **task-specific rubric / scoring criteria** [2507.22958].

The 122 solutions are distributed across seven EGE task types:

| Task ID | Domain | Count / Score Range |
|---|---|---|
| 13 | Trigonometric equations | 21 / 0–2 |
| 14 | Stereometry | 18 / 0–3 |
| 15 | Logarithmic inequalities | 19 / 0–2 |
| 16 | Financial mathematics problems | 17 / 0–2 |
| 17 | Planimetry | 15 / 0–3 |
| 18 | Parameterised equations | 16 / 0–4 |
| 19 | Number theory / combinatorics | 16 / 0–4 |

The task distribution spans algebraic, geometric, financial, and combinatorial material, but always under the same operational condition: the model must **grade a completed handwritten solution** rather than solve a fresh prompt [2507.22958]. This matters because the input modality is heterogeneous. It combines handwritten symbolic notation, natural-language explanation, and, in some cases, diagrammatic or spatial content. The paper reports that geometry tasks such as **stereometry** and **planimetry** were harder and showed poorer agreement with human graders, which the authors attribute to difficulty in interpreting **free-hand diagrams**, **spatial relations**, and **geometric reasoning chains from handwriting and figures** [2507.22958].

## 3. Rubrics, partial credit, and grading semantics

A defining feature of the benchmark is its use of **official human expert scoring according to official exam criteria** [2507.22958]. The benchmark is therefore not grounded in approximate annotation or crowd judgments, but in formal grading practice. This is especially important because the central challenge is not binary correctness but **partial correctness**.

The paper gives a concrete rubric example for **Task 18**, which uses a **0–4** scale [2507.22958]. The rubric is summarized as follows:

- **4 points**: correct, complete, well-reasoned solution  
- **3 points**: correct set of parameter values, but only differs from the correct set by boundary-point inclusion/exclusion  
- **2 points**: correct interval obtained with correct reasoning but possibly wrong boundary points, or a computational mistake despite correct logic  
- **1 point**: roots are found and the problem is reduced correctly, but further investigation is incomplete or wrong  
- **0 points**: none of the above  

This rubric exemplifies the benchmark’s semantics. The model must discriminate among several nearby states of correctness: exact solution, boundary-point error, computational error under correct logic, and incomplete but substantively relevant reduction. Such distinctions are typical of human mathematical grading and are not reducible to final-answer matching.

This feature also places the benchmark in continuity with broader critiques of overly reductive mathematics assessment. "Assessments in Mathematics, undergraduate degree" argues that mathematical assessment should not be reduced to fast, automated marking when the goal is to evaluate **reasoning, writing, and conceptual understanding**, and it states that MCQs can test **knowledge and understanding**, but not the ability to **reason, justify, write, or argue mathematically** [1707.04758]. The EGE-Math benchmark’s emphasis on handwritten production, partial credit, and rubric-based interpretation is consistent with that assessment perspective. This suggests that the benchmark is not merely a multimodal engineering task, but also an instantiation of a specific theory of mathematical validity in assessment.

## 4. Evaluation protocol and performance measures

The evaluation in [2507.22958] covers **seven VLMs** in **three inference modes**. The tested models are:

1. **Arcee AI Spotlight**  
2. **Google Gemini 2.0 Flash**  
3. **Google Gemini 2.0 Flash Lite**  
4. **Google Gemini 2.5 Flash Preview**  
5. **Google Gemini 2.5 Flash Preview:thinking**  
6. **OpenAI o4-mini**  
7. **Qwen 2.5 VL 32B**

The three inference modes are:

- **Without Answer**: the model receives the handwritten solution image and the problem statement  
- **With Answer**: the model receives the handwritten solution image, the problem statement, and the correct final numerical answer  
- **With True Solution**: the model receives the handwritten solution image, the problem statement, the correct final answer, and a full reference / gold-standard solution  

These modes probe whether additional grading context improves rubric alignment. The benchmark therefore tests not only standalone assessment ability, but also the model’s capacity for **comparative grading** and conditional use of supervision.

The paper evaluates three main metrics [2507.22958]. **Accuracy (Exact Match)** is the percentage of cases where the predicted score exactly matches the expert score. **Quality Score** is a normalized closeness measure that rewards predictions near the correct score, even if not exact. **Average Score Distance** is the mean absolute difference between predicted and true scores. Together, these measures distinguish exact rubric agreement from approximate grade proximity.

The strongest result reported is for **OpenAI o4-mini** in **With Answer** mode, with **Accuracy: 56.56%**, **Quality Score: 78.17%**, and **Average Score Distance: 0.60** [2507.22958]. The paper identifies this as the best result across the benchmark. It also reports that **Google Gemini 2.0 Flash** and **Google Gemini 2.5 Flash Preview** performed reasonably well, while **Arcee AI Spotlight** and **Qwen 2.5 VL 32B** generally had lower accuracy and larger score distances [2507.22958].

## 5. Empirical findings and observed failure modes

The principal empirical conclusion is that current VLMs remain limited in **mathematical reasoning** and **human-rubric alignment** when grading handwritten exam solutions [2507.22958]. Even the best exact-match performance leaves substantial disagreement with expert scorers. The benchmark therefore identifies assessment as a harder problem than answer production alone.

One major finding is that **extra context does not always help** [2507.22958]. Some models improved when given the correct answer or a reference solution, but others showed little benefit or even degradation. The paper gives specific examples: **Gemini 2.0 Flash** improved notably when given the correct answer, whereas **Arcee AI Spotlight** got slightly worse with more context. It further notes that **With True Solution** was not always the best mode. This indicates that current systems do not yet exhibit robust comparative reasoning over student work and expert solutions.

The **Gemini 2.5 Flash Preview:thinking** variant did not consistently outperform the regular version [2507.22958]. The paper interprets this as evidence that increased deliberative runtime or nominally deeper reasoning does not automatically yield better rubric-based grading. This is an important result because it separates generic reasoning depth from institutional scoring competence.

The paper identifies several recurrent limitations [2507.22958]. **Handwriting/OCR difficulty** produces recognition errors that propagate into grading mistakes. **Weak rubric alignment** leads models to mishandle partial credit and boundary cases. **Difficulty with comparative reasoning** persists even when the correct answer or full solution is supplied. The benchmark is also constrained by a **small dataset** of 122 solutions and by **mostly zero-shot evaluation**, meaning that models were not optimized specifically for this task. Finally, **cost-performance trade-offs** matter for practical use.

The appendix includes a representative parameter problem in which most models assigned the correct score, but others either **overestimated** the score or **underestimated** it badly, and some failed to complete a coherent grading response [2507.22958]. The significance of this example is diagnostic rather than anecdotal: model failures are heterogeneous, arising from rubric misinterpretation, incomplete reasoning, or response-generation breakdowns.

## 6. Relation to other mathematical benchmarks and assessment traditions

The EGE-Math benchmark occupies a distinct position among mathematical evaluation suites. In **GeoEval**, the task is geometry problem-solving for **LLMs and multi-modal models**, measured by **accuracy** on final answers across text-only, text+diagram, and challenge subsets [2402.10104]. GeoEval stresses **backward reasoning**, **rephrasing robustness**, **contamination sensitivity**, and **difficulty scaling**, but its target remains solution production rather than grading. In **DocMath-Eval**, the task is document-grounded numerical reasoning in long financial reports, requiring retrieval of evidence from text and tables and then generation of a correct numeric answer [2311.09805]. Again, the emphasis is on solving rather than assessment.

By contrast, EGE-Math evaluates whether a model can judge **how good a student’s solution is, and why** [2507.22958]. The output is a **grade / score**, not a final numerical answer, and correctness is defined by alignment with **official human expert rubrics**. This makes the benchmark closer to educational assessment than to standard reasoning leaderboards.

The benchmark also resonates with pedagogical arguments against reducing mathematics evaluation to item formats that privilege recognition over production. The undergraduate assessment study in [1707.04758] concludes that MCQs are **“totally unsuitable”** for mathematics assessment except for **mechanical or purely computational tasks**, and that **classical written subjects remain the best option** for reasoning, writing, or genuine mathematical production. The EGE-Math benchmark does not itself make that curricular claim, but it is built precisely around **written, step-by-step solutions** and **official scoring criteria**. A plausible implication is that it operationalizes, in benchmark form, the distinction between computational correctness and the evaluation of mathematical production that [1707.04758] argues is central to valid assessment.

Two misconceptions are explicitly countered by this comparison. The first is that strong performance on math-solving benchmarks implies strong performance on math assessment benchmarks. EGE-Math rejects that equivalence by construction [2507.22958]. The second is that supplying the correct answer or a gold solution necessarily resolves the assessment problem. The reported mode effects show that this is false for current models [2507.22958].

## 7. Applications, limits, and research directions

The paper presents the benchmark as a step toward **AI as a grading assistant**, not as a replacement for teachers [2507.22958]. The applications it discusses include helping teachers **triage student work**, providing **preliminary grading suggestions**, **detecting likely mistakes**, supporting **rubric-based assessment**, and improving **feedback quality**. At the same time, it states that current models are **not yet reliable enough** for fully autonomous grading in **high-stakes settings** [2507.22958].

The future research directions identified in the paper are concrete. These include **hybrid systems** combining VLMs with specialized **handwritten-math recognition (HMER)**, **fine-tuning** on rubric-based grading data, **larger and more diverse datasets**, better **reasoning over student work vs. reference solutions**, more **explainable** grading outputs, adaptation to **other curricula and countries**, and **interactive or human-in-the-loop grading systems** [2507.22958].

In this sense, the benchmark functions as both an evaluation resource and a problem specification. It defines a testbed in which progress would require simultaneous advances in multimodal parsing, symbolic and informal mathematical reasoning, rubric semantics, and alignment with expert human judgment. Its strongest contribution is therefore not only the dataset of **122 scanned handwritten EGE solutions**, but the precise formulation of grading as a benchmarkable capability: a model must read, interpret, diagnose, and score mathematical work under official criteria, and current systems remain substantially below the level required for dependable autonomous use [2507.22958].

Source: https://www.emergentmind.com/topics/ege-math-solutions-assessment-benchmark