---
title: 'EEFSUVA: A Novel Math Olympiad Benchmark'
url: https://www.emergentmind.com/topics/eefsuva
type: topic
---

# EEFSUVA: A Novel Math Olympiad Benchmark

Searching arXiv for the cited paper to ground the article in the current record.
EEFSUVA is a mathematical Olympiad benchmark for large language models designed to test performance on hard, low-contamination, numerical-answer problems drawn from Eastern Europe and the countries of the former Soviet Union, together with a small set of problems from Vladimir I. Arnold’s texts. It was introduced to probe whether strong scores on widely used Olympiad-style benchmarks reflect genuine mathematical reasoning or, instead, partial dependence on data contamination and familiarity with overrepresented problem traditions. The benchmark emphasizes problems that are both difficult and unfamiliar, especially those requiring an extra nontrivial step and nonstandard problem-solving techniques, and it evaluates models by exact numerical correctness rather than proof quality [2510.01227].

## 1. Rationale and conceptual position

The benchmark was introduced against the backdrop of claims that large language models match gold medal Olympiad to graduate level proficiency on mathematics benchmarks. Its central premise is that existing evaluations, especially those drawing heavily from the International Mathematical Olympiad, USAMO, and related contests, may overstate reasoning ability because those problems are heavily scraped, translated, and discussed online, and because many current benchmarks and platforms rely strongly on IMO-type contest problems [2510.01227].

The paper identifies two distinct mechanisms by which such overstatement can occur. The first is **data contamination**: models may have seen many of the benchmark problems or their solutions during pretraining. The second is **narrow problem style**: IMO-style problems emphasize certain patterns, structures, and tricks, so models may learn reusable templates rather than robust, transferable reasoning. EEFSUVA is therefore positioned not as a replacement for established Olympiad benchmarks, but as a complement intended to broaden evaluation.

Within the benchmark description, “EEFSUVA” refers to **Eastern Europe and Former Soviet Union Variable Answer** problems. The “variable answer” designation reflects a restriction to deterministic outputs, usually numeric answers. This design choice is methodological rather than merely cosmetic: it permits unambiguous automatic checking and avoids the subjectivity inherent in grading proof-style outputs. A plausible implication is that the benchmark isolates one specific dimension of mathematical competence—answer-finding under nonstandard reasoning demands—while deliberately excluding proof production.

## 2. Corpus composition and source traditions

EEFSUVA contains **45 problems** in total: **39 problems** from regional, national, and international Olympiads in Eastern Europe and former Soviet Union countries, and **6 problems** from Vladimir I. Arnold’s texts *Ordinary Differential Equations*, *Mathematical Methods of Classical Mechanics*, and *The Mathematical Trivium* [2510.01227]. The contest sources listed in the benchmark description include the Balkan Mathematical Olympiad, Junior Balkan Mathematical Olympiad and its Romanian selection tests, Bulgarian Mathematical Competitions used as selection tests for IMO, All Soviet Union Math Competitions from 1961 to 1992, the International Zhautykov Olympiad, the International Olympiad “Tuymaada” for 2019, 2021, and 2022, various “National Contests from Around the World” collections, and a small subset from Putnam Exam 2022.

These sources are characterized as well known among Olympiad specialists but substantially less present in mainstream English-language datasets and widely scraped educational websites. That positioning is crucial to the benchmark’s intended novelty: the problems are not secret, but they are comparatively under-circulated in the web ecosystems most likely to shape general-purpose LLM pretraining.

| Component | Count | Notes |
|---|---:|---|
| Olympiad and competition problems | 39 | Eastern Europe and former Soviet Union sources |
| Arnold text problems | 6 | From MMCM, *Mathematical Trivium*, and ODE |
| Total | 45 | Purely evaluative benchmark |

The topic distribution is markedly uneven by design. Among the Olympiad problems, **Combinatorics** accounts for **61.5%**, **Number Theory** for **17.9%**, **Graph Theory** for **10.3%**, **Algebra** for **5.1%**, and **Functional equations** for **5.1%**; there is explicitly **no geometry**. Among the Arnold problems, **57.1%** come from *Mathematical Methods of Classical Mechanics*, **28.6%** from *The Mathematical Trivium*, and **14.3%** from *Ordinary Differential Equations*. The authors state that the emphasis on combinatorics and number theory reflects their view that these areas most strongly expose weaknesses in template-based pattern recognition, because solutions usually demand novel constructions or clever simplifications rather than routine computation.

## 3. Problem design and mathematical character

The benchmark is deliberately restricted to **numerical-answer problems** and explicitly excludes proof problems [2510.01227]. Two reasons are given. First, proof problems often correspond to well-known theorems or standard arguments that LLMs can recognize and mimic without deep reasoning. Second, proof evaluation involves subjectivity and substantial human effort. By selecting only determinate outputs—numbers or concrete expressions—the benchmark prioritizes crisp scoring and reproducibility.

The benchmark’s problems are described as demanding an **“extra, nontrivial step in mathematical reasoning”** and as being **“intentionally non-algorithmic.”** In Olympiad combinatorics, this often means that direct application of known formulas is ineffective, and that the solver must invent structures such as bespoke invariants or recursive configurations. In number theory, the paper emphasizes subtle inequalities, clever substitutions, and simplifying tricks whose discovery is decisive. This problem profile is intended to test whether a model can construct reasoning in situ rather than recover a memorized schema.

Two representative examples illustrate the intended difficulty profile. One combinatorics problem asks for the maximum number of color-preserving automorphisms of a ten-level binary tree under arbitrary leaf colorings, with answer
$$
M = 2^{2^7}.
$$
Its solution proceeds through definitions of \(k\)-level \(2\)-trees, isomorphisms and automorphisms, lower bounds such as
$$
\chi(k) \ge (\chi(k-1))^2
$$
and
$$
\chi(k) \ge 2^{2^{k-3}},
$$
and a construction showing sharpness. The point of the example is not merely the final value, but the structural sophistication of the reasoning path. A second example asks for the greatest integer \(n>10\) such that the remainder of \(n\) when divided by each square between \(2\) and \(n/2\) is an odd integer; the answer is
$$
n = 505.
$$
The solution uses parity constraints, inequalities of the form
$$
3 \le \frac{n}{(2k-1)^2} < 4,
$$
and a sequence of bounds involving odd squares before a final verification. Both examples are presented as instances of reasoning chains that are intricate, non-obvious, and poorly captured by rote formula application.

A plausible implication is that EEFSUVA targets a specific failure regime of current LLMs: the inability to discover or maintain the single decisive transformation that converts a superficially opaque problem into a tractable one.

## 4. Construction workflow and evaluation protocol

The curation workflow has six stated stages: identify overused Olympiads in existing benchmarks; search for Eastern European and former Soviet contests, especially older and underused ones; extract problems with deterministic numerical answers; solve each problem independently; verify those solutions against official or published sources; and then run LLMs and evaluate their outputs [2510.01227]. This workflow is intended to ensure problem quality, answer correctness, and minimal ambiguity in scoring.

Problems were taken from PDFs and web sources, then translated or adapted into English where necessary. Notation was standardized while retaining mathematical rigor. Although LaTeX is used in the paper, the benchmark tasks presented to models use plain-text mathematical notation that the models can parse. Each problem is given to the model **as is**, without mention of competition source, year, or contest identity, in order to avoid hinting at cached solutions.

The benchmark has no train/test split because it is not a training resource; it is purely evaluative. The time horizon for included problems is **at least 5 years old**, often **5–40 years back**. The paper notes an important tension here: age increases the chance that a problem exists somewhere on the web, but the selected corpus is less aligned with fashionable recent benchmarks. The benchmark therefore does not claim zero contamination. Instead, it is explicitly framed as **low contamination and structurally diverse**.

The evaluation protocol is likewise tightly specified. A **fresh chat session** is opened for each problem to prevent cross-problem context carryover. The tested models are **GPT-5 Thinking** in high reasoning mode and **Gemini 2.5 Pro**. Each problem is run **twice**, in separate sessions, and a problem is counted as solved if **either attempt** yields the correct numerical answer. The primary metric is **accuracy**. Although the benchmark does not impose a step-by-step reasoning metric, the authors report reading chain-of-thought carefully in order to analyze how the models reason and where they fail.

## 5. Empirical performance and failure modes

The central empirical result is a large performance drop on EEFSUVA relative to other Olympiad-style benchmarks [2510.01227]. On MathArena benchmarks, **Gemini 2.5 Pro** scores **82.50%** on HHMT, **84.91%** on SMT, **90.00%** on BRUMO, **31.55%** on IMO, **24.40%** on USMO, and **0%** on EEFSUVA. **GPT-5 Thinking/High** scores **88.33%** on HHMT, **91.98%** on SMT, **91.67%** on BRUMO, **38.10%** on IMO, **n/a** on USMO, and **35.89%** on EEFSUVA.

| Model | Selected benchmark results | EEFSUVA |
|---|---|---:|
| GPT-5 Thinking/High | HHMT 88.33%, SMT 91.98%, BRUMO 91.67%, IMO 38.10% | 35.89% |
| Gemini 2.5 Pro | HHMT 82.50%, SMT 84.91%, BRUMO 90.00%, IMO 31.55%, USMO 24.40% | 0% |

Within EEFSUVA itself, **GPT-5 Thinking** achieves **35.89** overall, with **25.00** in Combinatorics, **42.85** in Number Theory, **25.00** in Graph Theory, **100.00** in Algebra, **0.00** in Functional Equations, and **28.57** on the Arnold problems. **Gemini 2.5 Pro** scores **0** across all of these categories. The algebra score for GPT-5 is based on only a handful of problems, so it should not be overgeneralized.

The qualitative error analysis is as important as the percentages. In combinatorics, failures often involve hallucinated intermediate steps, miscounting, or inability to construct the required invariant or recursive argument. In number theory, typical errors include missing the key inequality or substitution and mishandling divisibility or parity constraints. In graph theory, errors include miscounting and misunderstanding extremal relationships. The paper reports that models frequently **hallucinate** intermediate reasoning, often **lock onto the wrong template** when a problem resembles a known contest problem, and seldom reconcile contradictions once such a template has been engaged.

A particularly revealing example concerns a coin-box game similar, but not identical, to an IMO shortlist problem. Gemini 2.5 Pro produces the answer
$$
n =
\begin{cases}
k + \left\lfloor \dfrac{2}{k} \right\rfloor + 1, & k \ge 2, \\
3, & k = 1,
\end{cases}
$$
and then explicitly remarks that
> “The value \( n = k + \left\lfloor \frac{k}{2} \right\rfloor + 1 \) is the value from the official IMO Shortlist, which suggests that my failure analysis for \(k=2,3\) or my subsequent analysis of player strategy might be flawed.”

The significance of this case is methodological. The model recognizes resemblance, anchors on a known IMO formula, and reuses the pattern rather than solving the actual variant afresh. The paper treats this as direct evidence for contamination-driven template reliance and for brittleness under small structural perturbations.

## 6. Interpretation, limitations, and prospective extensions

The benchmark’s main interpretive claim is that strong scores on IMO-based or similarly prominent mathematics benchmarks can give a misleading impression of broad mathematical proficiency [2510.01227]. On EEFSUVA, even GPT-5 Thinking solves only about one-third of the problems, while Gemini 2.5 Pro solves none. The paper argues that this does not necessarily mean EEFSUVA is uniformly harder than IMO in human terms; rather, the problems are less familiar and more structurally diverse. This suggests that current LLMs are strongly tuned to particular problem traditions and that their generalization to equally difficult but less common styles remains limited.

The benchmark also has clear limitations. It is relatively small at **45 problems**, so results are informative but not exhaustive. It is restricted to **numerical-answer problems**, which means it does not test proof-writing or the evaluation of richer mathematical exposition. It covers high-level competition mathematics rather than the full range of mathematical practice, and it explicitly excludes geometry to avoid diagram-interpretation complications.

The practical recommendations associated with the benchmark follow directly from its design. Evaluation should use fresh sessions per problem, multiple attempts if variability is being assessed, strict numerical scoring, and no source hints. Because the problems are old but public, EEFSUVA cannot guarantee absence of contamination; its stated contribution is instead a lower-contamination, structurally diverse alternative to more saturated benchmark traditions.

Several extensions are suggested. These include expanding to other regional contest traditions, incorporating more problems from advanced texts such as *The Mathematical Trivium*, adding detailed solution rationales, introducing multi-step scoring or partial credit for correct intermediate steps, and considering geometry problems with formal diagram encodings. More broadly, the benchmark motivates the construction of evaluation datasets that cover different regions and traditions, include older and obscure problems not widely mirrored online, and emphasize nonstandard problem-solving strategies. A plausible implication is that such datasets are necessary if benchmark performance is to track robust reasoning rather than familiarity with a narrow, highly visible slice of Olympiad culture.

Source: https://www.emergentmind.com/topics/eefsuva