---
title: 'Nemotron-IMO-Bench: An Overview'
url: https://www.emergentmind.com/topics/nemotron-imo-bench
type: topic
---

# Nemotron-IMO-Bench: An Overview

Nemotron-IMO-Bench is a benchmark for natural-language proof generation on difficult olympiad-level mathematics. Introduced in “An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics” [2609.10712], it consists of 200 novel problems written specifically for the benchmark in collaboration with Professor Titu Andreescu. The benchmark is designed to evaluate proof generation, proof verification, verification-guided refinement, multi-model search, test-time compute scaling, and checkpoint specialization without formal provers, external tools, or internet access. It should be distinguished from the official International Mathematical Olympiad (IMO), IMO-AnswerBench, and IMO-Proof Bench. Earlier Nemotron papers discuss mathematical benchmarks and IMO-related evaluations, but do not define a benchmark named Nemotron-IMO-Bench [2508.14444] [2603.19220].

## 1. Scope and purpose

Nemotron-IMO-Bench evaluates the generation of complete mathematical proofs from natural-language olympiad problems. The model receives a LaTeX problem statement and produces a natural-language solution. The system described with the benchmark operates entirely in natural language and does not use a formal theorem prover, symbolic geometry engine, external tools, or internet access.

The benchmark was created to provide a relatively fresh evaluation resource for olympiad mathematics. Its problems were written specifically for the benchmark and had not been published previously, which the authors present as a safeguard against contamination from repeated use of public olympiad and mathematics datasets. The benchmark is also intended to support research on:

- proof generation;
- proof verification;
- verification-guided refinement;
- multi-model search;
- test-time compute scaling;
- checkpoint specialization and ensemble design.

The benchmark is not presented as a formalized theorem-proving dataset. Its central evaluation object is a natural-language olympiad problem, and its expected output is a natural-language mathematical proof rather than an exact short answer or a formal proof object.

The benchmark was released publicly on Hugging Face as `nvidia/Nemotron-IMO-Bench` under the CC BY 4.0 license. The supplied description does not specify a JSON schema, serialized record format, example record, or complete metadata schema.

## 2. Construction, novelty, and dataset coverage

The full benchmark contains 200 novel olympiad-level problems. The problems were created in collaboration with Professor Titu Andreescu, a mathematics educator and olympiad problem author. The paper does not specify how many authors contributed, whether every problem received independent review, whether official solutions or answer keys were created, or whether the problems were generated from templates, adapted from unpublished material, or written entirely from scratch.

The benchmark’s principal anti-contamination safeguard is novelty and non-publication. The authors state that the problems were written for the benchmark and had not been published before; consequently, they state that the problems do not appear in the evaluated models’ training data. However, the paper does not report a formal corpus-overlap audit, membership-inference analysis, hash-based contamination test, or quantitative comparison against model pretraining corpora.

The benchmark itself was not the main source of model pretraining or post-training data. The reported SFT and RL corpora were drawn primarily from Nemotron-Math-Proofs-v1 and its AoPS subset. The authors nevertheless used Nemotron-IMO-Bench in exploratory ablations. This distinction is relevant because the benchmark’s novelty claim addresses literal exposure to its problem statements, whereas it does not establish complete separation from mathematically related concepts, proof strategies, or predecessor datasets.

The paper does not provide complete topic, difficulty, or problem-type statistics for all 200 problems. It also does not define conventional train, validation, and test splits. The released collection is therefore a benchmark corpus rather than a formally partitioned supervised-learning dataset.

## 3. Development set and empirical difficulty

The main rapid-iteration experiments use a 30-problem development set assembled from:

- 20 problems from Nemotron-IMO-Bench;
- 10 problems from recent competitions.

The 10 competition problems are publicly available on Hugging Face, and the authors provide a script for assembling the complete development set from those problems and the released benchmark.

A preliminary Nemotron-3-Ultra-GA run was used to estimate difficulty and select problems spanning the observed range. The resulting categories are operational, search-based labels rather than intrinsic human-curated difficulty classes:

| Difficulty category | Number |
|---|---:|
| Easy | 3 |
| Medium | 7 |
| Hard | 10 |
| Unsolved | 10 |
| **Total** | **30** |

The categories are defined by search behavior. Easy problems have a substantial first-round acceptance rate. Medium problems have only one or two accepted proofs in round 1, or obtain their first accepted proof in round 2. Hard problems first obtain an accepted proof between rounds 3 and 8. Unsolved problems receive no accepted proof.

The development-set topic distribution is:

| Topic | Number |
|---|---:|
| Algebra | 6 |
| Combinatorics | 8 |
| Geometry | 8 |
| Number theory | 8 |
| **Total** | **30** |

The paper does not report the corresponding distribution for the remaining 170 benchmark problems. It also does not provide a complete taxonomy of problem types such as inequalities, functional equations, constructions, extremal problems, counting problems, or invariant problems. Categories appearing in prompts—such as lemma-first decomposition, route comparison, counterexample search, formal sanity checking, invariant or extremal reasoning, and strategy auditing—are generation strategies rather than benchmark metadata.

## 4. Verification and grading

Nemotron-IMO-Bench uses several distinct evaluation layers. Internal search acceptance, finalist ranking, independent jury scoring, and official contest grading are not interchangeable.

### Search-time verification

During search, candidate proofs are evaluated by Nemotron-3-Ultra-RL and Nemotron-3-Ultra-SFT. Each verifier assigns a score in $\{0, 0.5, 1\}$:

- $1$: complete and correct proof;
- $0.5$: generally correct proof with minor errors or omissions;
- $0$: fatal errors or severe omissions.

Each verifier checkpoint produces eight independent judgments, giving 16 judgments per proof. A proof is internally accepted only when all 16 judgments are valid and parseable and every judgment assigns score $1$. A non-parseable judgment is invalid and does not count toward the required panel.

Internal acceptance is a search condition and does not by itself imply correctness under independent or official grading. This distinction is important because the benchmark evaluates natural-language proofs, for which automated verification can be both selective and error-prone.

### Finalist selection

Final finalists are evaluated by three checkpoints:

- Nemotron-3-Ultra-GA;
- Nemotron-3-Ultra-RL;
- Nemotron-3-Ultra-SFT.

Each checkpoint produces 16 independent judgments using an IMO-style prompt. Each judgment assigns an integer score from 0 to 7. The finalist score is the mean of 48 judgments:

$$
\text{Finalist score}
=
\frac{1}{48}\sum_{j=1}^{48}s_j,
\qquad
s_j\in\{0,1,\ldots,7\}.
$$

Finalists are ranked by this mean, with ties broken in favor of the shorter proof.

### Independent experimental jury

For development-set experiments, the authors use an independent jury comprising GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.8. Each model assigns a score from 0 to 7 using the same IMO-style judge prompt. The evaluation is reference-free, uses no tool calls, performs no normalization, and does not use rubrics derived from official solutions.

The authors apply a MathArena-style reconciliation procedure. If initial scores differ by at most two points, the minimum score is used. Otherwise, all judges re-grade after seeing the first-round assessments, and the minimum second-round score is used. Incomplete panels are rerun.

A problem is considered solved in the experiments when the selected proof receives full credit—7 points—from the independent jury. The paper reports both cumulative scores for problems with internally accepted proofs and end-to-end scores over all 30 problems, using the highest-ranked fallback proof when no proof is internally accepted.

### Official grading

Official human grading applies only to the model’s 2026 IMO submission, not to the general benchmark. That submission received 30/42 points, above the gold-medal cutoff of 29. Problems 1, 2, 4, and 5 received 7 points each, while Problems 3 and 6 received 1 point each. A later unofficial human regrade of a post-contest Problem 6 proof gave 4/7; this was not an official benchmark or IMO result.

## 5. Search, refinement, and inference-time computation

The associated system uses three 550B-A55B Nemotron-3 Ultra checkpoints:

- GA, the general-availability checkpoint;
- SFT, the supervised fine-tuned checkpoint;
- RL, the reinforcement-learned checkpoint.

The checkpoints participate in generation, verification, and refinement. Search runs for at most eight rounds per problem.

In round 1, eight complementary generation prompts encourage different proof strategies, including standard proof generation, lemma-first decomposition, route comparison, counterexample guarding, formal sanity checking, gap-resistant reasoning, invariant or extremal reasoning, and strategy auditing. Each of the three checkpoints samples 16 attempts for each prompt:

$$
3 \times 8 \times 16 = 384
$$

proof attempts per problem.

Each candidate is then judged by the RL and SFT checkpoints, with eight independent judgments from each:

$$
2 \times 8 = 16
$$

judgments per candidate. A candidate with unanimous 16/16 score 1 is accepted.

If no proof is accepted, up to 16 high-ranked proofs are selected from the global pool. Each receives up to eight verifier critiques, and each refinement prompt is sent to all three generation checkpoints, with four outputs per checkpoint:

$$
16 \times 3 \times 4 = 192
$$

refinement attempts per round.

The process continues for up to eight rounds. Once a generation checkpoint produces an accepted proof, that checkpoint stops sampling further candidates for the current problem, while the other checkpoints continue through the round. This produces up to three finalists, one per generator checkpoint. If no proof is accepted, the highest-ranked proof in the final proof pool becomes the sole finalist.

The search procedure therefore measures a model-plus-verification-and-refinement system rather than a single forward-pass policy. Additional inference-time computation is a central experimental variable.

## 6. Results and ablations

The principal development-set results are cumulative independent-jury points on the 30-problem set. Parentheses indicate the number of problems with an internally accepted proof.

| Round | GA | RL | SFT | Ensemble |
|---|---:|---:|---:|---:|
| R1 | 34 (5) | 47 (7) | 70 (10) | 91 (13) |
| R2 | 72 (12) | 82 (12) | 98 (14) | 160 (23) |
| R3 | 93 (15) | 109 (16) | 112 (16) | 167 (24) |
| R4 | 114 (18) | 137 (20) | 126 (18) | 167 (24) |
| R5 | 128 (20) | 137 (20) | 140 (20) | 167 (24) |
| R6 | 142 (22) | 151 (22) | 147 (21) | 167 (25) |
| R7 | 142 (22) | 151 (22) | 147 (21) | 167 (25) |
| R8 | 145 (23) | 152 (23) | 147 (21) | 167 (25) |
| R8 + fallback | 162 (23) | 180 (23) | 165 (21) | 188 (25) |

The SFT checkpoint is strongest in round 1, whereas RL is the strongest single-checkpoint system overall. The ensemble reaches its final accepted-only score within three rounds and outperforms every individual checkpoint pipeline. Including fallback proofs after round 8 raises the RL score to 180 and the ensemble score to 188 out of a possible 210 points.

### Verifier audit

The authors audit 300 proofs: 25 accepted proofs and 275 non-accepted proofs stratified by search-verifier score. An independent-jury score of 7 is treated as the correctness ground truth.

| Panel or rule | False accept | False reject |
|---|---:|---:|
| GA, 8/8 | 31.6% [17.9, 45.4] | 22.8% [11.0, 35.5] |
| RL, 8/8 | 12.4% [4.7, 20.4] | 64.2% [53.8, 72.5] |
| SFT, 8/8 | 4.5% [0.6, 9.0] | 69.1% [62.1, 75.3] |
| GA+SFT, 16/16 | 3.4% [0.0, 7.3] | 75.6% [67.6, 81.7] |
| RL+SFT, 16/16 | 1.1% [0.0, 2.7] | 81.3% [75.8, 85.0] |
| GA+RL+SFT, 24/24 | 1.1% [0.0, 2.7] | 82.9% [77.5, 86.7] |
| RL+SFT, 14/16 | 17.0% [6.5, 27.9] | 59.4% [48.4, 69.0] |
| RL+SFT, 12/16 | 25.4% [10.5, 42.3] | 30.9% [20.0, 42.2] |

The intervals are 95% intervals from 2,000 problem-clustered bootstrap resamples. SFT is the most selective verifier, GA is more permissive, and RL lies between them. The unanimous RL+SFT panel has the lowest false-accept rate among the tested rules without adding GA judgments.

Among 25 internally accepted proofs, 23 received jury score 7, one received 6 for a minor quantifier formality, and one received 0 because it used an invalid column-permutation symmetry argument contradicted by an explicit counterexample.

### Attempt-scaling and negative ablations

An attempt-scaling ablation compares additional sampling from one checkpoint with diversity from multiple checkpoints. Doubling RL attempts from 128 to 256 produces only a small gain, whereas adding SFT attempts is substantially more useful. Among 18 accepted problems from an RL+SFT pool, six are accepted by both checkpoints, seven only by RL, and five only by SFT. Doubling RL attempts recovers only one of the five problems uniquely reached by SFT. GA adds no new round-1 accepted problem beyond RL and SFT, although it contributes refinement diversity later.

Several alternatives did not improve final aggregate performance. Cross-proof context and zero-first ranking sometimes accelerated early progress but did not improve the final number of accepted problems. Refining one parent proof and synthesizing from four diverse proofs accepted the same number of problems in the reported comparison, differing on only two problems. A triage system reduced nominal routing verification cost by 95.3%, but discarded the round-1 ancestor of an eventually accepted proof in 44% of relevant problems; human reviewers found 13 of 14 such discarded-lineage proofs fully correct, so triage was rejected for production.

The benchmark consequently exposes a trade-off between verifier precision and recall. Strict unanimous panels reduce false acceptance but produce substantial false rejection, while less stringent rules admit more potentially correct proofs at the cost of weaker reliability.

## 7. Interpretation, limitations, and related resources

Nemotron-IMO-Bench measures the performance of a proof-generation system coupled with search, verification, refinement, checkpoint ensembling, and substantial inference-time computation. Its results should not be interpreted as measurements of a single model sample or of unaided mathematical competence.

The paper does not provide a complete formal contamination audit. Novelty and non-publication reduce the likelihood of literal memorization, but they do not establish disjointness from all related mathematical sources or proof strategies. The development set also includes 10 recent competition problems in addition to 20 benchmark problems, so development-set results are not results on the 200 benchmark items alone.

The benchmark lacks a conventional train/dev/test partition. The full 200-item release is not accompanied, in the supplied description, by a complete schema, official solutions, proof rubrics, full topic statistics, full difficulty metadata, representative problem statements, or a formal held-out test definition. The 30-problem development set uses difficulty categories derived from preliminary system search behavior rather than independently established human judgments.

Automated and model-based grading introduces further limitations. Internal verifier acceptance is reference-free and can produce both false accepts and false rejects. The independent jury is also reference-free and uses model judgments rather than official mathematical marking. Consequently, high scores can reflect interactions among generator, verifier, refinement policy, and judge models. The official 30/42 IMO 2026 result is based on human contest grading, but it is separate from the general benchmark evaluation.

The benchmark is related to, but distinct from, several other Nemotron mathematical resources. Earlier work evaluated GSM8K, MATH, MATH Level 5, AIME, MATH-500, and MGSM rather than Nemotron-IMO-Bench [2508.14444]. Nemotron-Cascade 2 reported results on the official 2025 IMO, IMO-AnswerBench, and IMO-Proof Bench, none of which is the same as Nemotron-IMO-Bench [2603.19220]. Nemotron-IMO-Bench instead emphasizes newly authored olympiad-level problems and natural-language proof generation under an explicit test-time search and verification framework.

The released benchmark and associated artifacts include the dataset, post-trained SFT and RL checkpoints, training data, inference pipeline, development-set assembly code, and training recipes. These resources are intended to support reproducible study of proof-generation systems, verifier design, refinement policies, ensemble composition, and the allocation of inference-time computation.

Source: https://www.emergentmind.com/topics/nemotron-imo-bench