---
title: 'IMO-GradingBench: Proof Grading Benchmark'
url: https://www.emergentmind.com/topics/imo-gradingbench
type: topic
---

# IMO-GradingBench: Proof Grading Benchmark

IMO-GradingBench is an evaluation benchmark for automated grading of proof-based solutions to Olympiad-level mathematics problems, specifically targeting the International Mathematical Olympiad (IMO). Developed as part of the IMO-Bench initiative [2511.01846], its principal contribution is a large, rigorously annotated dataset that enables direct, quantitative assessment of autograders—AI systems capable of assigning partial credit to mathematical proofs in a manner analogous to expert human graders. This resource aims to address the essential *meta-reasoning* task of evaluating not just final answers or solution existence, but the quality, rigor, and partial progress reflected in long-form mathematical reasoning.

## 1. Motivation and Evaluation Scope

IMO-GradingBench was created to measure and promote robustness in mathematical reasoning evaluation while filling methodological gaps in existing benchmarks, which traditionally focus on short-answer correctness or solution generation rather than grading free-form proofs. Its design responds to critical needs:

- **Scalability**: Human grading is labor-intensive and slow for large collections of complex proofs;
- **Rigorous meta-reasoning**: Automated system evaluation demands the ability to recognize nuance, partial progress, and minor errors—mirroring human grader standards;
- **Educational and research feedback**: The benchmark facilitates training and evaluation of autograders for use in mathematical education and research.

It targets Olympiad-grade problems, advancing beyond saturated benchmarks such as MATH and GSM8K by addressing full proofs rather than short answers [2511.01846].

## 2. Dataset Construction and Annotation Methodology

IMO-GradingBench comprises 1,000 solutions to IMO-level proof problems, sampled from the Advanced IMO-ProofBench set. Each data point consists of:

- The original problem statement;
- A candidate solution, typically generated by cutting-edge large language models, exhibiting varied correctness and quality;
- A human-assigned grade (0–7 points), mapped to four discrete categories for robust metric computation.

Human graders—experienced mathematicians with IMO scoring expertise—assigned grades according to standard Olympiad protocol:

| Category         | IMO Points | Solution Quality                                         |
|------------------|------------|---------------------------------------------------------|
| Correct          | 7          | Fully correct, rigorous, and complete                   |
| Almost Correct   | 6–4        | Minor errors or incomplete justification                |
| Partial          | 1–3        | Some progress but incomplete or flawed                  |
| Incorrect        | 0          | Irrelevant or incorrect                                |

Problems were sampled so that all four categories are represented roughly equally, maximizing diagnostic value when benchmarking autograders [2511.01846].

## 3. Grading Rubric and Guidelines

The benchmark employs a simplified rubric modeled on IMO standards. Graders assign solutions to precisely one of the following categories:

- **Correct**: Full solution, with all steps justified and arguments complete;
- **Almost**: Solution is correct in principle but contains minor omissions or inaccuracies;
- **Partial**: Substantial work done, but significant gaps or errors remain;
- **Incorrect**: Solution is essentially wrong, off-topic, or trivial.

This discretization enables standardized accuracy and error calculation. Model prompts mimic IMO grading scenarios: the AI only receives the problem and proposed solution—no reference answer or guideline—mirroring blind peer review [2511.01846].

## 4. Evaluation Protocols and Metrics

IMO-GradingBench introduces two principal metrics for autograder evaluation:

1. **Accuracy**: Fraction of solutions where the model prediction matches the human-assigned category.
2. **Mean Absolute Error (MAE)**: Average absolute difference between model and human grades, mapped to canonical category scores (7 for Correct, 6 for Almost, 1 for Partial, 0 for Incorrect).

The protocol mandates that model outputs must end with a single rubric word ("correct", "almost", "partial", "incorrect"), facilitating automated score extraction and direct comparison with expert scores.

## 5. Use in Autograder Development and Validation

The benchmark serves as a testbed for autograder systems—AI agents or LLMs tasked with **grading** mathematical proof solutions. It supports development and fine-tuning of models intended for:

- Educational feedback applications, where rapid, nuanced assessment is required;
- Research into model "understanding" of mathematical argumentation and error patterns;
- Quantitative benchmarking of mathematical meta-reasoning, distinct from answer-finding or proof-writing.

For evaluation on IMO-GradingBench, autograders are **not given reference solutions or detailed grading guidelines**; they must infer correctness and assign a categorical grade based solely on the presented argument [2511.01846].

## 6. Results, Analysis, and Limitations

Reported autograder performance highlights the difficulty of human-aligned grading:

| Model                | Accuracy     | MAE        |
|----------------------|-------------|------------|
| Gemini 2.5 Pro       | 44.3%       | 30.2%      |
| SOTA non-Gemini      | 50.2%       | 18.4%      |
| Gemini 2.5 Deep Think| 52.5%       | 20.5%      |
| o3 (OpenAI)          | 54.0%       | 20.2%      |

Key observations:

- State-of-the-art models match human grading at best ~54% of the time, underscoring the challenge of nuanced proof evaluation;
- Most errors occur between "partial" and "incorrect", revealing limits in recognizing borderline or pedagogical distinctions;
- Models rarely confuse "correct" and "incorrect", suggesting reliable recognition of gross validity but difficulty with subtler errors.

When autograders are supplied with reference solutions and grading guidelines (as in the related IMO-ProofBench evaluation), far higher correlations with expert judgment are reported (up to 0.96 Pearson), indicating that the core difficulty lies in the blind evaluation setting provided by IMO-GradingBench [2511.01846].

## 7. Impact and Future Directions

IMO-GradingBench marks a critical advance in automated evaluation protocols for proof-based mathematical reasoning at the Olympiad level:

- It enables scalable research on autograder reliability, partial credit assignment, and diagnostic coverage;
- By establishing "north-star" metrics for meta-reasoning, it provides the community with a concrete target for progress beyond answer-centric evaluation;
- Its open release (https://imobench.github.io/) encourages replication, fine-tuning, and benchmarking for both educational and research AI systems.

*This suggests that future work will focus on improving nuance recognition, pedagogical understanding, and robustness of automated graders, toward expert-level performance in mathematical solution assessment.*

## References

- Towards Robust Mathematical Reasoning [2511.01846]  
- GenesisGeo: Technical Report [2509.21896]  
- Wu's Method can Boost Symbolic AI to Rival Silver Medalists and AlphaGeometry to Outperform Gold Medalists at IMO Geometry [2404.06405]  
- Towards Solving More Challenging IMO Problems via Decoupled Reasoning and Proving [2507.06804]  
- CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning [2510.27094]  
- A Lean Dataset for International Math Olympiad: Small Steps towards Writing Math Proofs for Hard Problems [2411.18872]  
- Gemini 2.5 Pro Capable of Winning Gold at IMO 2025 [2507.15855]  
- RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning [2509.07711]

Source: https://www.emergentmind.com/topics/imo-gradingbench