RIMO Benchmark: Olympiad-Level Reasoning

• RIMO is a dual-track benchmark derived from International Mathematical Olympiad problems, featuring a numeric (RIMO-N) and a proof (RIMO-P) track.
• The benchmark minimizes evaluation noise by enforcing unique integer answers and structured proof decomposition for stepwise grading.
• Empirical results reveal a significant gap between end-to-end numeric accuracy and sustained proof competence, highlighting challenges in high-level reasoning.

Searching arXiv for the exact RIMO benchmark paper and closely related mathematical reasoning benchmarks. RIMO, short for “Remade International Mathematical Olympiad,” is a two-track benchmark derived from International Mathematical Olympiad problems from 1959–2023 for evaluating advanced mathematical reasoning in LLMs. Its central design objective is to preserve Olympiad-level difficulty while reducing evaluation noise: RIMO-N rewrites 335 problems so that each has a single, unique integer answer with deterministic grading, whereas RIMO-P contains 456 proof problems with expert-checked solutions decomposed into sub-problems for step-wise automated assessment (Chen et al., 9 Sep 2025).

1. Benchmark rationale and positioning

RIMO was introduced in a period when strong LLMs already achieve very high scores on earlier mathematical benchmarks such as GSM8K and MATH, leaving limited headroom for differentiating systems on high-end reasoning tasks (Cobbe et al., 2021, Hendrycks et al., 2021, Chen et al., 9 Sep 2025). The benchmark is explicitly motivated by limitations in recent Olympiad-level evaluations, including heterogeneous output formats, dependence on LLM judges or symbolic normalization heuristics, potentially flawed reference solutions, and hidden or dynamic test sets that complicate reproducibility (Chen et al., 9 Sep 2025).

The benchmark therefore adopts a dual criterion: it is intended to remain hard to solve while becoming easy to evaluate. In RIMO-N, this is realized by enforcing a single integer answer and exact-match grading; in RIMO-P, it is pursued through proof decomposition and strict step-local judging against expert-validated reference solutions. The benchmark comparison table lists both tracks as having 5 difficulty levels, and RIMO-N is described there as having noise level 0 (Chen et al., 9 Sep 2025).

This design places RIMO at the intersection of two benchmark desiderata that often conflict in mathematical evaluation: preserving non-routine Olympiad reasoning, and ensuring that measurement error does not dominate the reported scores. A plausible implication is that RIMO is intended not merely as another difficult dataset, but as an instrument for high-resolution measurement of genuine reasoning progress.

2. Two-track structure

RIMO consists of a numeric track and a proof track, each targeting a different aspect of mathematical competence (Chen et al., 9 Sep 2025).

Track	Content	Evaluation
RIMO-N	335 remade IMO problems; 236 shortlist and 99 contest problems	Exact matching to a single unique integer answer
RIMO-P	456 proof problems decomposed into 1–4 sub-problems	Sequential step-wise grading with DeepSeek-R1

RIMO-N preserves the standard Olympiad topical spread. Its 335 problems are distributed across algebra, geometry, number theory, and combinatorics, with counts of 96, 95, 86, and 58 respectively. The final answer is always a single integer, including a binary subset of 96 problems whose answers lie in $\{0,1\}$ (Chen et al., 9 Sep 2025).

RIMO-P retains original proof-style formulations rather than converting them to short answers. Each problem is decomposed into 1–4 sub-problems according to proof length and structure, and the final sub-problem always corresponds to the original main claim. This makes the track explicitly process-oriented: it evaluates whether a model can establish intermediate lemmas and sustain a logically coherent derivation, not only whether it can guess or infer the final conclusion (Chen et al., 9 Sep 2025).

The two tracks are complementary. RIMO-N emphasizes end-to-end answer production under a large output space, whereas RIMO-P targets rigorous multi-step proof construction. This suggests that the benchmark is designed to separate at least two capabilities that are often conflated in mathematical evaluation: correct final-answer generation and durable proof-level reasoning.

3. Construction, remaking, and verification

The source material for RIMO comes from scanned IMO booklets and PDFs covering public contest papers and confidential shortlists from 1959–2023. The processing pipeline collects statements and available solutions, digitizes and normalizes LaTeX text, reconciles solutions across sources, and then determines whether a problem should remain in proof form for RIMO-P or be rewritten into a single-integer format for RIMO-N (Chen et al., 9 Sep 2025).

For RIMO-N, the remaking process is constrained by three requirements: the final answer must be a single integer, the integer must be unique, and the logical core and difficulty of the original problem must be preserved. The benchmark documentation describes several transformation patterns. A concurrency statement may be reformulated as a question about the number of common points; a classification problem may be rewritten so that all valid solutions induce the same value of a requested sum; a true/false claim may become a question with answer in $\{0,1\}$ (Chen et al., 9 Sep 2025).

Answer verification is stringent. For shortlist problems, official solutions typically provide the target integer directly. For contest problems, inclusion requires agreement from at least two of three independent community sources: AoPS Wiki, YouTube blackboard expositions, and ParSe transcripts. Disagreement triggers manual adjudication and usually exclusion. Problems with unresolved discrepancies in either statements or answers are dropped (Chen et al., 9 Sep 2025).

For RIMO-P, each retained proof problem is paired with an expert-verified complete solution assembled from official shortlist solutions and community write-ups. Decomposition is then aligned to the structure of that solution. Longer arguments are typically partitioned into key lemmas, intermediate reductions, and a final synthesis step, while shorter arguments may remain in one or two parts (Chen et al., 9 Sep 2025).

The construction pipeline therefore operates as both a curation and normalization procedure. Its emphasis is not only on difficulty preservation, but also on reducing ambiguities in reference answers and proof structure that would otherwise propagate into benchmark noise.

4. Evaluation methodology

Both tracks are evaluated under a uniform inference protocol: greedy decoding with temperature $T=0$, a single pass per problem, and no external tools or calculators. The benchmark is thus intended to measure internal reasoning rather than retrieval or tool-assisted execution (Chen et al., 9 Sep 2025).

For RIMO-N, scoring is exact and deterministic. If $\hat{y}_i$ denotes a model prediction and $y_i$ the ground-truth integer for problem $i$, accuracy is computed as

$$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[\hat{y}_i = y_i],$$

with $N=335$. Because the output space is reduced to unique integers, correctness reduces to exact equality; no symbolic simplification or model-based adjudication is required (Chen et al., 9 Sep 2025).

For RIMO-P, evaluation is step-wise rather than holistic. For each problem $i$, let $X_i$ be the number of sub-problems and $\{0,1\}$0 the number of consecutively correct sub-solutions from the start. If the first incorrect step occurs at position $\{0,1\}$1, then only the preceding $\{0,1\}$2 steps count, even if later responses would independently be judged correct. The overall score is

$\{0,1\}$3

where $\{0,1\}$4 (Chen et al., 9 Sep 2025).

The RIMO-P judge is DeepSeek-R1, prompted as a strict mathematician and constrained to return JSON of the form {"verdict": "correct", "reason": "..."} or {"verdict": "incorrect", "reason": "..."}. The judge is given the original problem, the official complete solution, the candidate sub-solution, and the step index within the decomposition. It is instructed to reject any mathematical error, logical gap, or unjustified step (Chen et al., 9 Sep 2025).

The methodological distinction between the two tracks is important. RIMO-N removes nearly all grading ambiguity by construction. RIMO-P still uses an LLM judge, but narrows the evaluation problem to local proof steps anchored by reference solutions. This suggests a deliberate hierarchy of evaluation certainty: exact matching where possible, constrained model-based assessment where proof equivalence cannot be reduced to string comparison.

5. Empirical findings

RIMO was benchmarked on ten models ranging from 7B to 671B parameters, including Qwen3-8B, GPT-4o-2024-08-06, InternLM3-8B-instruct, DeepSeek-R1-671B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-math-7B-instruct, NuminaMath-7B-TIR, Gemini-2.5-flash, and Mathstral-7B-v0.1. The resulting RIMO-N scores are substantially lower than the same models’ scores on GSM8K and MATH (Chen et al., 9 Sep 2025).

The strongest reported RIMO-N result is 62.96% from DeepSeek-R1-671B, followed by 58.81% from Gemini-2.5-flash. Representative smaller or non-reasoning-specialized systems perform markedly worse: Qwen3-8B reaches 36.72%, GPT-4o-2024-08-06 reaches 33.43%, and several math-specialized 7B systems remain below 24%. The paper characterizes this as a roughly 30–50 percentage point drop relative to MATH for top systems (Chen et al., 9 Sep 2025).

The release-date analysis reports a Spearman rank correlation of only 0.21 between first-release date and RIMO-N accuracy. Parameter scale helps, but does not determine outcomes on its own. DeepSeek-R1-671B leads the table, yet GPT-4o performs far below Gemini-2.5-flash despite also being a large proprietary model, and Qwen3-8B outperforms several nominally math-specialized 7B–14B systems (Chen et al., 9 Sep 2025).

The benchmark also isolates answer-space effects through the binary subset of 96 problems with answers in $\{0,1\}$5. All evaluated models improve on this subset by 8–30 percentage points. DeepSeek-R1-671B rises from 63% on full RIMO-N to 73% on the binary subset; Qwen3-8B rises from 37% to 67%; DeepSeek-R1-Distill-Qwen-7B rises from 16% to 24%. Because random guessing would yield 50% on a strict binary task, these results indicate genuine reasoning signal, but they also show that exact integer identification in a large output space contributes materially to difficulty (Chen et al., 9 Sep 2025).

Reasoning-optimized systems outperform their non-reasoning counterparts. The paper reports that QwQ-32B exceeds Qwen-2.5-32B by 19.4 percentage points on RIMO-N, while DeepSeek-R1-671B exceeds DeepSeek-V3 by 4.2 points. This indicates that explicit reasoning optimization remains effective even at Olympiad difficulty (Chen et al., 9 Sep 2025).

RIMO-P is more severe still. Only three models are reported there—Qwen3-8B, InternLM3-8B-instruct, and Mathstral-7B-v0.1—and all obtain very low $\{0,1\}$6 scores. Mathstral-7B, which has domain-specific mathematical fine-tuning, outperforms similarly sized generic models, but the general pattern is that even systems capable of nontrivial performance on RIMO-N rarely sustain full proof trajectories on RIMO-P (Chen et al., 9 Sep 2025).

6. Interpretation, limitations, and future directions

The principal empirical conclusion of RIMO is the existence of a substantial gap between current LLM performance on established math benchmarks and actual Olympiad-level reasoning. The gap appears in two forms. First, end-to-end numeric problem solving remains far from saturated: even the best reported model solves only 62.96% of RIMO-N. Second, proof construction remains substantially weaker than answer finding: the very low RIMO-P scores indicate frequent failure to maintain stepwise logical validity across a full derivation (Chen et al., 9 Sep 2025).

This split between RIMO-N and RIMO-P exposes a common misconception in mathematical evaluation: strong final-answer performance does not imply proof competence. The benchmark’s structure suggests that qualitative judgment, local lemma discovery, and exact quantitative completion are partially separable abilities, and that current systems remain much weaker on long-horizon proof organization than on shorter answer-oriented reasoning (Chen et al., 9 Sep 2025).

The benchmark has explicit limitations. It is confined to IMO-style high-school Olympiad mathematics and does not cover university mathematics, other STEM disciplines, or multimodal geometry settings. RIMO-P still depends on an LLM judge, so it is not fully machine-verifiable. Its decomposition into 1–4 steps follows expert judgment based on reference proofs, which may not align with every valid proof route. The experiments also cover only ten models overall and only three on RIMO-P, owing to GPU and API constraints (Chen et al., 9 Sep 2025).

The proposed future directions are correspondingly concrete. The authors suggest formalizing RIMO-P in Lean to obtain machine-checkable proof evaluation, expanding the leaderboard as new models appear, and performing finer-grained error analysis to identify which proof steps fail most often. The benchmark is distributed through a GitHub repository and a HuggingFace dataset, which positions it as a static and reproducible long-term evaluation suite rather than a transient competition artifact (Chen et al., 9 Sep 2025).

In that sense, RIMO functions less as a general mathematics corpus than as a calibrated stress test for advanced reasoning. Its significance lies in combining Olympiad difficulty with a deliberate effort to minimize grading ambiguity, thereby turning benchmark design itself into part of the measurement apparatus.