EEFSUVA: A Novel Math Olympiad Benchmark

• EEFSUVA is a benchmark comprising 45 numerical-answer problems sourced from Eastern Europe and former Soviet contests, emphasizing nonroutine problem-solving.
• It evaluates language models by requiring exact numerical outputs and demanding extra, nontrivial mathematical reasoning beyond template-based methods.
• Designed as a complement to traditional Olympiad tests, EEFSUVA minimizes data contamination and focuses on isolated numerical correctness without proof-based assessments.

Searching arXiv for the cited paper to ground the article in the current record. EEFSUVA is a mathematical Olympiad benchmark for LLMs designed to test performance on hard, low-contamination, numerical-answer problems drawn from Eastern Europe and the countries of the former Soviet Union, together with a small set of problems from Vladimir I. Arnold’s texts. It was introduced to probe whether strong scores on widely used Olympiad-style benchmarks reflect genuine mathematical reasoning or, instead, partial dependence on data contamination and familiarity with overrepresented problem traditions. The benchmark emphasizes problems that are both difficult and unfamiliar, especially those requiring an extra nontrivial step and nonstandard problem-solving techniques, and it evaluates models by exact numerical correctness rather than proof quality (Khatibi et al., 23 Sep 2025).

1. Rationale and conceptual position

The benchmark was introduced against the backdrop of claims that LLMs match gold medal Olympiad to graduate level proficiency on mathematics benchmarks. Its central premise is that existing evaluations, especially those drawing heavily from the International Mathematical Olympiad, USAMO, and related contests, may overstate reasoning ability because those problems are heavily scraped, translated, and discussed online, and because many current benchmarks and platforms rely strongly on IMO-type contest problems (Khatibi et al., 23 Sep 2025).

The paper identifies two distinct mechanisms by which such overstatement can occur. The first is data contamination: models may have seen many of the benchmark problems or their solutions during pretraining. The second is narrow problem style: IMO-style problems emphasize certain patterns, structures, and tricks, so models may learn reusable templates rather than robust, transferable reasoning. EEFSUVA is therefore positioned not as a replacement for established Olympiad benchmarks, but as a complement intended to broaden evaluation.

Within the benchmark description, “EEFSUVA” refers to Eastern Europe and Former Soviet Union Variable Answer problems. The “variable answer” designation reflects a restriction to deterministic outputs, usually numeric answers. This design choice is methodological rather than merely cosmetic: it permits unambiguous automatic checking and avoids the subjectivity inherent in grading proof-style outputs. A plausible implication is that the benchmark isolates one specific dimension of mathematical competence—answer-finding under nonstandard reasoning demands—while deliberately excluding proof production.

2. Corpus composition and source traditions

EEFSUVA contains 45 problems in total: 39 problems from regional, national, and international Olympiads in Eastern Europe and former Soviet Union countries, and 6 problems from Vladimir I. Arnold’s texts Ordinary Differential Equations, Mathematical Methods of Classical Mechanics, and The Mathematical Trivium (Khatibi et al., 23 Sep 2025). The contest sources listed in the benchmark description include the Balkan Mathematical Olympiad, Junior Balkan Mathematical Olympiad and its Romanian selection tests, Bulgarian Mathematical Competitions used as selection tests for IMO, All Soviet Union Math Competitions from 1961 to 1992, the International Zhautykov Olympiad, the International Olympiad “Tuymaada” for 2019, 2021, and 2022, various “National Contests from Around the World” collections, and a small subset from Putnam Exam 2022.

These sources are characterized as well known among Olympiad specialists but substantially less present in mainstream English-language datasets and widely scraped educational websites. That positioning is crucial to the benchmark’s intended novelty: the problems are not secret, but they are comparatively under-circulated in the web ecosystems most likely to shape general-purpose LLM pretraining.

Component	Count	Notes
Olympiad and competition problems	39	Eastern Europe and former Soviet Union sources
Arnold text problems	6	From MMCM, Mathematical Trivium, and ODE
Total	45	Purely evaluative benchmark

The topic distribution is markedly uneven by design. Among the Olympiad problems, Combinatorics accounts for 61.5%, Number Theory for 17.9%, Graph Theory for 10.3%, Algebra for 5.1%, and Functional equations for 5.1%; there is explicitly no geometry. Among the Arnold problems, 57.1% come from Mathematical Methods of Classical Mechanics, 28.6% from The Mathematical Trivium, and 14.3% from Ordinary Differential Equations. The authors state that the emphasis on combinatorics and number theory reflects their view that these areas most strongly expose weaknesses in template-based pattern recognition, because solutions usually demand novel constructions or clever simplifications rather than routine computation.

3. Problem design and mathematical character

The benchmark is deliberately restricted to numerical-answer problems and explicitly excludes proof problems (Khatibi et al., 23 Sep 2025). Two reasons are given. First, proof problems often correspond to well-known theorems or standard arguments that LLMs can recognize and mimic without deep reasoning. Second, proof evaluation involves subjectivity and substantial human effort. By selecting only determinate outputs—numbers or concrete expressions—the benchmark prioritizes crisp scoring and reproducibility.

The benchmark’s problems are described as demanding an “extra, nontrivial step in mathematical reasoning” and as being “intentionally non-algorithmic.” In Olympiad combinatorics, this often means that direct application of known formulas is ineffective, and that the solver must invent structures such as bespoke invariants or recursive configurations. In number theory, the paper emphasizes subtle inequalities, clever substitutions, and simplifying tricks whose discovery is decisive. This problem profile is intended to test whether a model can construct reasoning in situ rather than recover a memorized schema.

Two representative examples illustrate the intended difficulty profile. One combinatorics problem asks for the maximum number of color-preserving automorphisms of a ten-level binary tree under arbitrary leaf colorings, with answer

$M = 2^{2^7}.$

Its solution proceeds through definitions of $k$-level $2$-trees, isomorphisms and automorphisms, lower bounds such as

$\chi(k) \ge (\chi(k-1))^2$

and

$\chi(k) \ge 2^{2^{k-3}},$

and a construction showing sharpness. The point of the example is not merely the final value, but the structural sophistication of the reasoning path. A second example asks for the greatest integer $n>10$ such that the remainder of $n$ when divided by each square between $2$ and $n/2$ is an odd integer; the answer is

$n = 505.$

The solution uses parity constraints, inequalities of the form

$k$0

and a sequence of bounds involving odd squares before a final verification. Both examples are presented as instances of reasoning chains that are intricate, non-obvious, and poorly captured by rote formula application.

A plausible implication is that EEFSUVA targets a specific failure regime of current LLMs: the inability to discover or maintain the single decisive transformation that converts a superficially opaque problem into a tractable one.

4. Construction workflow and evaluation protocol

The curation workflow has six stated stages: identify overused Olympiads in existing benchmarks; search for Eastern European and former Soviet contests, especially older and underused ones; extract problems with deterministic numerical answers; solve each problem independently; verify those solutions against official or published sources; and then run LLMs and evaluate their outputs (Khatibi et al., 23 Sep 2025). This workflow is intended to ensure problem quality, answer correctness, and minimal ambiguity in scoring.

Problems were taken from PDFs and web sources, then translated or adapted into English where necessary. Notation was standardized while retaining mathematical rigor. Although LaTeX is used in the paper, the benchmark tasks presented to models use plain-text mathematical notation that the models can parse. Each problem is given to the model as is, without mention of competition source, year, or contest identity, in order to avoid hinting at cached solutions.

The benchmark has no train/test split because it is not a training resource; it is purely evaluative. The time horizon for included problems is at least 5 years old, often 5–40 years back. The paper notes an important tension here: age increases the chance that a problem exists somewhere on the web, but the selected corpus is less aligned with fashionable recent benchmarks. The benchmark therefore does not claim zero contamination. Instead, it is explicitly framed as low contamination and structurally diverse.

The evaluation protocol is likewise tightly specified. A fresh chat session is opened for each problem to prevent cross-problem context carryover. The tested models are GPT-5 Thinking in high reasoning mode and Gemini 2.5 Pro. Each problem is run twice, in separate sessions, and a problem is counted as solved if either attempt yields the correct numerical answer. The primary metric is accuracy. Although the benchmark does not impose a step-by-step reasoning metric, the authors report reading chain-of-thought carefully in order to analyze how the models reason and where they fail.

5. Empirical performance and failure modes

The central empirical result is a large performance drop on EEFSUVA relative to other Olympiad-style benchmarks (Khatibi et al., 23 Sep 2025). On MathArena benchmarks, Gemini 2.5 Pro scores 82.50% on HHMT, 84.91% on SMT, 90.00% on BRUMO, 31.55% on IMO, 24.40% on USMO, and 0% on EEFSUVA. GPT-5 Thinking/High scores 88.33% on HHMT, 91.98% on SMT, 91.67% on BRUMO, 38.10% on IMO, n/a on USMO, and 35.89% on EEFSUVA.

Model	Selected benchmark results	EEFSUVA
GPT-5 Thinking/High	HHMT 88.33%, SMT 91.98%, BRUMO 91.67%, IMO 38.10%	35.89%
Gemini 2.5 Pro	HHMT 82.50%, SMT 84.91%, BRUMO 90.00%, IMO 31.55%, USMO 24.40%	0%

Within EEFSUVA itself, GPT-5 Thinking achieves 35.89 overall, with 25.00 in Combinatorics, 42.85 in Number Theory, 25.00 in Graph Theory, 100.00 in Algebra, 0.00 in Functional Equations, and 28.57 on the Arnold problems. Gemini 2.5 Pro scores 0 across all of these categories. The algebra score for GPT-5 is based on only a handful of problems, so it should not be overgeneralized.

The qualitative error analysis is as important as the percentages. In combinatorics, failures often involve hallucinated intermediate steps, miscounting, or inability to construct the required invariant or recursive argument. In number theory, typical errors include missing the key inequality or substitution and mishandling divisibility or parity constraints. In graph theory, errors include miscounting and misunderstanding extremal relationships. The paper reports that models frequently hallucinate intermediate reasoning, often lock onto the wrong template when a problem resembles a known contest problem, and seldom reconcile contradictions once such a template has been engaged.

A particularly revealing example concerns a coin-box game similar, but not identical, to an IMO shortlist problem. Gemini 2.5 Pro produces the answer

$k$1

and then explicitly remarks that

“The value $k$2 is the value from the official IMO Shortlist, which suggests that my failure analysis for $k$3 or my subsequent analysis of player strategy might be flawed.”

The significance of this case is methodological. The model recognizes resemblance, anchors on a known IMO formula, and reuses the pattern rather than solving the actual variant afresh. The paper treats this as direct evidence for contamination-driven template reliance and for brittleness under small structural perturbations.

6. Interpretation, limitations, and prospective extensions

The benchmark’s main interpretive claim is that strong scores on IMO-based or similarly prominent mathematics benchmarks can give a misleading impression of broad mathematical proficiency (Khatibi et al., 23 Sep 2025). On EEFSUVA, even GPT-5 Thinking solves only about one-third of the problems, while Gemini 2.5 Pro solves none. The paper argues that this does not necessarily mean EEFSUVA is uniformly harder than IMO in human terms; rather, the problems are less familiar and more structurally diverse. This suggests that current LLMs are strongly tuned to particular problem traditions and that their generalization to equally difficult but less common styles remains limited.

The benchmark also has clear limitations. It is relatively small at 45 problems, so results are informative but not exhaustive. It is restricted to numerical-answer problems, which means it does not test proof-writing or the evaluation of richer mathematical exposition. It covers high-level competition mathematics rather than the full range of mathematical practice, and it explicitly excludes geometry to avoid diagram-interpretation complications.

The practical recommendations associated with the benchmark follow directly from its design. Evaluation should use fresh sessions per problem, multiple attempts if variability is being assessed, strict numerical scoring, and no source hints. Because the problems are old but public, EEFSUVA cannot guarantee absence of contamination; its stated contribution is instead a lower-contamination, structurally diverse alternative to more saturated benchmark traditions.

Several extensions are suggested. These include expanding to other regional contest traditions, incorporating more problems from advanced texts such as The Mathematical Trivium, adding detailed solution rationales, introducing multi-step scoring or partial credit for correct intermediate steps, and considering geometry problems with formal diagram encodings. More broadly, the benchmark motivates the construction of evaluation datasets that cover different regions and traditions, include older and obscure problems not widely mirrored online, and emphasize nonstandard problem-solving strategies. A plausible implication is that such datasets are necessary if benchmark performance is to track robust reasoning rather than familiarity with a narrow, highly visible slice of Olympiad culture.