Olympiad-Style Evaluation Event

• Olympiad-style evaluation events are competitive benchmarks that use difficult, multi-step tasks to test advanced reasoning, algorithm design, and proof construction.
• These events employ controlled protocols such as sealed exams, hidden tests, and fixed scoring scripts to enhance reliability and auditability.
• The framework addresses issues like contamination and incentive misalignment by emphasizing reproducibility and systematic, domain-spanning task curation.

An Olympiad-style evaluation event is a benchmarking or competition protocol built around olympiad-grade tasks: difficult, multi-step problems intended to probe generalized reasoning, algorithm design, proof construction, or multimodal scientific inference under tightly specified rules. Recent work uses the format in two closely related senses. One is a public benchmark suite with contest-like tasks, hidden tests or formal graders, and fixed evaluation scripts; examples include USACO for competitive programming, OlymMATH and RIMO for mathematics, OIBench for informatics, and domain-specific suites in chemistry, physics, and multi-image reasoning (Shi et al., 2024, Sun et al., 27 Mar 2025, Zhu et al., 12 Jun 2025, Chen et al., 9 Sep 2025, Cui et al., 17 Dec 2025, Yu et al., 9 Sep 2025, Chen et al., 22 Apr 2026). The other is a sealed-exam design in which tasks remain confidential until a single centralized run, submissions are frozen beforehand, and the full task bundle, harness, and logs are released after scoring to support auditability (Cruz et al., 24 Mar 2026).

1. Motivation and evaluative role

The modern rationale for Olympiad-style evaluation is diagnostic rather than merely competitive. Several recent benchmarks argue that standard leaderboards are increasingly saturated or increasingly easy to misread. In mathematics, OlymMATH states that benchmarks such as GSM8K and MATH have been largely “solved” by today’s strongest LLMs, while Omni-MATH reports that OpenAI o1 achieves $94.8\%$ on MATH and therefore no longer faces a truly challenging test of Olympiad-level reasoning (Sun et al., 27 Mar 2025, Gao et al., 2024). In programming, USACO is introduced precisely because computing olympiads demand complex algorithmic reasoning, puzzle solving, and efficient code generation, yet had been understudied as an evaluation domain (Shi et al., 2024).

A second motivation is epistemic reliability. The sealed-exam proposal identifies three core weaknesses of contemporary LLM benchmarking: fragility, contamination, and incentive misalignment. Fragility denotes the fact that small choices such as prompt ordering, decoding settings, or aggregation can flip rankings. Contamination arises because models trained on web-scale corpora often “see” public test sets or near-duplicates. Incentive misalignment refers to leaderboard-driven private trial-and-error and selective disclosure of best results. The Olympiad-style response is to make strong performance harder to manufacture and easier to trust by sealing problems until evaluation, freezing submissions, and running all entries through one harness (Cruz et al., 24 Mar 2026).

A third motivation is coverage of capabilities that conventional datasets underrepresent. EEFSUVA was created because existing Olympiad mathematics benchmarks draw heavily on a small set of well-known competitions and may therefore overstate reasoning ability through contamination and over-representation of familiar templates (Khatibi et al., 23 Sep 2025). HiPhO addresses a parallel gap in physics: existing datasets did not provide systematic, up-to-date coverage of real Olympiad exams and did not enable direct human comparison (Yu et al., 9 Sep 2025). OMIBench makes the same point for multi-image reasoning, arguing that prior Olympiad-level multimodal benchmarks overemphasized single-image analysis (Chen et al., 22 Apr 2026). Taken together, these studies suggest that the format is valued not only for difficulty, but for its ability to isolate failure modes that simpler or more public benchmarks obscure.

2. Event architectures and governance

Two institutional architectures recur. The first is the open benchmark architecture. Here, organizers curate a fixed task set, publish evaluation code or graders, and encourage reproducible model comparison. USACO releases 307 problems from the USA Computing Olympiad with unit tests, reference code, and official analyses (Shi et al., 2024). OlymMATH releases 200 manually verified bilingual problems together with evaluation code and a data visualization tool (Sun et al., 27 Mar 2025). OIBench releases 250 original olympiad-level informatics problems and emphasizes contamination resistance, canonical solutions, and public judging infrastructure (Zhu et al., 12 Jun 2025). RIMO separates a deterministic numeric-answer track from a proof track with automated sequential grading (Chen et al., 9 Sep 2025).

The second architecture is the sealed exam. The “LLM Olympiad” proposal formalizes this as a yearly centrally run evaluation. Problems remain confidential until evaluation; participants submit a frozen artifact or committed endpoint in advance; organizers enforce budgets, output schemas, retry policies, and logging; and, after scoring, the sealed task set, scoring scripts, full harness, and run manifest are released for auditability (Cruz et al., 24 Mar 2026). The protocol also includes a public fingerprint of the encrypted archive to prevent post-hoc tampering, conflict-of-interest rules barring task authors from entering the same round, and a transparent patch-and-rerun process for late scoring-code bugs (Cruz et al., 24 Mar 2026).

Track design and standardization are inherited from older competition traditions. The Fifth ASP Competition partitions tasks by expressivity and complexity into tracks such as Basic Decision, Advanced Decision, Optimization, and Unrestricted, and couples this with a fixed modeling language to ensure fair comparison and community convergence on a stable syntax and semantics (Calimeri et al., 2014). This precedent shows that an Olympiad-style event need not be limited to natural-language reasoning or school-style contests; it can also be a systems evaluation framework whose essential features are controlled task selection, explicit scoring, and standardized interfaces.

3. Task curation, benchmark assets, and domain coverage

Recent Olympiad-style events span programming, mathematics, chemistry, physics, and multi-image STEM reasoning. Representative instantiations reported in the literature include the following (Shi et al., 2024, Sun et al., 27 Mar 2025, Cui et al., 17 Dec 2025, Yu et al., 9 Sep 2025, Chen et al., 22 Apr 2026, Zhu et al., 12 Jun 2025, Chen et al., 9 Sep 2025, Zhang et al., 9 Jun 2026).

Benchmark or event	Domain and scale	Distinctive evaluation assets
USACO	307 programming problems	hidden unit tests, reference Python 3, official analyses
OlymMATH	200 math problems, EN/ZH	EASY/HARD tiers, rule-based numeric verification
USNCO-V	473 chemistry questions	one image + four answer choices
HiPhO	13 physics exams, 519 subquestions	official marking schemes, medal thresholds
OMIBench	~1,300 multi-image STEM tasks	annotated rationales, exact and semantic matching
OIBench	250 original informatics problems	canonical C++17/O2, Time/Space Completion Curves
RIMO	335 numeric + 456 proof math problems	deterministic integer grading, sequential proof grading
ComBench	100 combinatorics problems	proof rubric, deterministic construction verification

Although the subject matter varies, the asset pattern is strikingly consistent. USACO requires, for each problem, a full statement, exhaustive hidden unit tests, a reference implementation in Python 3, and an official human-written analysis; its hidden tests number 10–17 per problem, and its 307 released items come from an initial 484 scraped from the USACO archive spanning 2011–2023 (Shi et al., 2024). OIBench likewise requires each coach-authored problem to include a canonical C++17/O2 solution and a test battery spanning very small to extreme instances, worst-case time and memory stress, and corner cases such as empty or all-equal inputs (Zhu et al., 12 Jun 2025). HiPhO adds a document-processing pipeline: PDF to Markdown via OCR preserving LaTeX, question–answer matching by indices, human verification of text, answers, and figures, extraction of official step-level rubrics, and post-processing for context completion, subquestion structuring, and unit specification (Yu et al., 9 Sep 2025).

Mathematics benchmarks show a similar concern for verifiability and contamination control. OlymMATH uses printed magazines, textbooks, and official competition materials rather than online scraping, organizes 200 problems into EASY and HARD tiers, and provides fully parallel English and Chinese versions (Sun et al., 27 Mar 2025). RIMO reconstructs all IMO problems from 1959–2023 into two tracks: RIMO-N rewrites 335 problems to admit a single unique integer answer, while RIMO-P preserves 456 proof problems and decomposes them into sequential sub-problems aligned to expert-verified official solutions (Chen et al., 9 Sep 2025). EEFSUVA draws from under-circulated regional and former Soviet Olympiads, focuses on numerical-answer problems, and evaluates each problem in a fresh chat session with no context carry-over (Khatibi et al., 23 Sep 2025).

Multimodal events extend the same logic to visual evidence. USNCO-V is composed of 204 local and 269 national U.S. National Chemistry Olympiad Part-I questions, each with one image and four answer choices, and explicitly labels modalities such as tables, analytical charts, experimental apparatus diagrams, and symbolic molecular structures (Cui et al., 17 Dec 2025). OMIBench contains approximately 1,300 problems from biology, chemistry, mathematics, and physics Olympiads with an average of 3.07 images per problem; its curation criterion requires that each task draw non-redundant evidence from at least two images (Chen et al., 22 Apr 2026). This suggests that Olympiad-style evaluation is increasingly defined by the richness of its supporting assets, not just by the prestige of the source contest.

4. Scoring, verification, and human alignment

A defining characteristic of the format is explicit scoring under standardized verification. In code generation, USACO uses the unbiased estimator from Chen et al. (2021),

$\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$

where $N$ is the total number of samples per problem and $n$ is the number of correct samples; it also enforces time and memory constraints and reports secondary metrics such as compilation-error rate, TLE rate, and wrong-answer rate (Shi et al., 2024). The sealed-exam proposal uses automatically computed per-task metrics such as accuracy, F1, calibration, and stability, with aggregate score normally defined as the macro-average across tasks,

$$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$

All metric and aggregation rules are fixed in the contest syllabus before evaluation (Cruz et al., 24 Mar 2026).

Other events emphasize deterministic or human-aligned grading. HiPhO defines an Answer-Level Score $A_Q \in \{0,1\}$, a Step-Level Score $S_Q = \sum_i w_i c_i$ based on official marking points, and a Final Problem Score $\mathrm{Score}(Q)=\max(A_Q,S_Q)$; medal assignments are then determined from official human medal thresholds or qualification cutoffs (Yu et al., 9 Sep 2025). RIMO-N uses exact string match on the decimal representation of a unique integer answer after normalization, while RIMO-P evaluates proof progress through the fraction of consecutive correct sub-problems before the first failure, averaged over all problems (Chen et al., 9 Sep 2025). ComBench assigns proof scores on a $0/1/6/7$ rubric and then gates construction-centric problems through a deterministic verifier: if the explicit construction fails, a $7$ is demoted to $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$0, a $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$1 to $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$2, and $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$3 remain unchanged (Zhang et al., 9 Jun 2026).

Verification infrastructure can itself be a major research object. OlymMATH accepts exact radicals, decimals, expressions, and intervals and verifies them through high-precision symbolic or numeric checks with acceptance threshold $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$4, or correct interval containment (Sun et al., 27 Mar 2025). OMIBench combines exact matching with semantic matching via a judge model, using exact numeric tolerances, multiple-choice extraction, and LCS-based text normalization, then reports both micro-averaged exact accuracy and GPTScore (Chen et al., 22 Apr 2026). These designs make a common methodological point: Olympiad-style evaluation is not only about harder tasks, but about replacing ambiguous judgment with constrained, inspectable grading protocols whenever possible.

5. Controlled inference protocols and solver interaction

Olympiad-style events usually specify not just the task set but the admissible interaction pattern between solver and task. In USACO, the baseline is zero-shot chain-of-thought prompting, where the model is asked to restate the problem, outline reasoning or pseudocode, and then generate full code. More advanced protocols include self-reflection, in which the model iteratively debugs its own code for up to $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$5 iterations with $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$6 as a typical choice, and retrieval-augmented generation over two corpora: a semantic store of competitive-programming textbook chapters and an episodic store of other USACO problems and solutions. The best automatic performance comes from combining episodic retrieval and self-reflection, and the human-in-the-loop variant permits at most five code generations and at most three feedback turns per problem (Shi et al., 2024).

Long-horizon and search-heavy systems formalize the same idea at greater scale. TongGeometry evaluates geometry solvers under a 90-minute wall-clock limit per problem and defines success as a fully closed proof trace accepted by a symbolic deductive-database engine. Its actor–critic guided tree search ranks candidate constructions by

$\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$7

where $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$8 is the policy model and $\mathrm{pass@}k = 1 - \frac{C(N-n,k)}{C(N,k)},$9 estimates distance to goal (Zhang et al., 2024). SU-01, a 30B-A3B reasoning model, allows up to 10 independent runs and up to 30 solve–verify–refine cycles per run, with response lengths up to 160K tokens and repeated self-verification before acceptance (Li et al., 13 May 2026). In IOI-style programming, GenCluster performs large-scale candidate generation, behavioral clustering on $N$0 synthesized tests, tournament ranking with $N$1 pairwise games per cluster, and round-robin submission under the IOI cap of $N$2 submissions per problem (Samadi et al., 16 Oct 2025).

These protocols turn the event itself into a controlled experiment on test-time compute. A plausible implication is that Olympiad-style evaluation increasingly measures two coupled objects: the base model and the submission policy. This is explicit in several papers, which compare single-pass inference, iterative reflection, retrieval, clustering, ranking, and self-verification under fixed budgets rather than treating decoding as an implementation detail (Shi et al., 2024, Samadi et al., 16 Oct 2025, Li et al., 13 May 2026).

6. Empirical findings, controversies, and open problems

The empirical picture is heterogeneous but consistently discriminative. On USACO, GPT-4 reaches only $N$3 pass@1 under zero-shot chain-of-thought prompting, and the best automatic method—episodic retrieval plus self-reflection—reaches $N$4; Bronze improves from $N$5 to $N$6, Silver from $N$7 to $N$8, Gold from $N$9 to $n$0, and Platinum remains $n$1 (Shi et al., 2024). On OlymMATH-HARD, state-of-the-art slow-thinking models score only $n$2–$n$3 on English and $n$4–$n$5 on Chinese, with a consistent $n$6–$n$7 drop from EN to ZH (Sun et al., 27 Mar 2025). EEFSUVA intensifies this effect: Gemini 2.5 Pro scores $n$8 and GPT-5 Thinking/High $n$9, despite much higher scores on standard Olympiad sets such as HHMT, SMT, and BRUMO (Khatibi et al., 23 Sep 2025). OMIBench reports that even Gemini-3-Pro reaches only about $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$0 overall (Chen et al., 22 Apr 2026). ComBench remains far from saturated as well: the strongest model reaches $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$1 overall Avg. and $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$2 overall Best@4 (Zhang et al., 9 Jun 2026).

Multimodal scientific events reveal additional structure in the failure modes. In chemistry, many models struggle with modality fusion, and in some cases removing the image improves accuracy, which the benchmark interprets as misalignment in vision-language integration; chain-of-thought prompting, by contrast, consistently improves both accuracy and visual grounding (Cui et al., 17 Dec 2025). In OMIBench, human-labeled analysis of 100 errors attributes $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$3 to visual perception failures, $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$4 to cross-image association failures, $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$5 to logical reasoning fallacies, and $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$6 to instruction comprehension biases (Chen et al., 22 Apr 2026). HiPhO shows that closed-source reasoning MLLMs can achieve 6 to 12 gold medals across 13 recent physics Olympiad exams, while most open-source MLLMs remain at or below the bronze level and most models still have a significant gap from full marks (Yu et al., 9 Sep 2025).

There are also striking exceptions. TongGeometry solves all 30 problems in IMO-AG-30 and 183 of 225 problems in MO-TG-225 under a 90-minute limit, surpassing both earlier symbolic-neural systems and average gold medalists on the geometry benchmark (Zhang et al., 2024). OIBench reports that current SOTA models already outperform most human participants in both correctness and efficiency, though they remain suboptimal relative to canonical solutions (Zhu et al., 12 Jun 2025). GenCluster reaches an IOI-2025 gold medal with an open-weight model at large test-time compute, while its submitted score remains below OpenAI’s closed system under the same benchmark (Samadi et al., 16 Oct 2025). SU-01 reaches gold-medal-level performance on IMO 2025, USAMO 2026, and IPhO 2024/2025 when combined with supervised curriculum learning, reinforcement learning, and test-time scaling (Li et al., 13 May 2026). These cases indicate that the format can register frontier progress without ceasing to be discriminative.

The main controversies concern trust, representativeness, and what is being measured. Sealed-exam advocates argue that public leaderboards are vulnerable to contamination, hidden evaluation choices, and selective disclosure, but also acknowledge that contamination cannot be fully eliminated, that closed-endpoint assurance is weaker, and that centralized harness bugs can shift rankings unless transparently patched and rerun (Cruz et al., 24 Mar 2026). Public benchmark papers emphasize complementary problems: OIBench estimates a contamination RiskScore of less than $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$7 for all tested models, OlymMATH avoids online scraping, and EEFSUVA deliberately broadens the source distribution beyond familiar competitions (Zhu et al., 12 Jun 2025, Sun et al., 27 Mar 2025, Khatibi et al., 23 Sep 2025). Human interaction introduces a further interpretive complication. On 15 hard USACO problems unsolved by any automatic method, GPT-4 plus targeted tutoring solved 13, with an $$\mathrm{Score}_{\mathrm{macro}} = \frac{1}{|T|}\sum_{t \in T}\mathrm{metric}_t.$$8 unsolved-to-solved rate, while GPT-3.5 solved 0; this reveals substantial latent capability, but it also motivates future work on automated generation of concise, strategy-level hints rather than direct comparison to purely autonomous systems (Shi et al., 2024).

Open challenges recur across domains. USACO identifies Platinum-tier algorithms, automated hint generation, and integration of test-case creation and counterexample generation into the loop as unresolved (Shi et al., 2024). The sealed-exam proposal stresses annual task rotation, modular governance, and post-hoc audit bundles (Cruz et al., 24 Mar 2026). ComBench argues that rigorous proof reasoning and constructive realization are distinct capabilities, especially for Existence and Construction problems (Zhang et al., 9 Jun 2026). OMIBench suggests that multi-image reasoning remains constrained by cross-panel association and long multi-step argument maintenance (Chen et al., 22 Apr 2026). The cumulative lesson is that an Olympiad-style evaluation event is best understood as a methodological framework for measuring high-end reasoning under controlled conditions, rather than as any single benchmark or leaderboard.