---
title: Formalization of IMO Problems
url: https://www.emergentmind.com/topics/formalization-of-imo-problems
type: topic
---

# Formalization of IMO Problems

The formalization of International Mathematical Olympiad (IMO) problems refers to the rigorous encoding of IMO-level mathematical statements and proofs within formal logical systems, typically using interactive or automated theorem provers. This process transforms inherently informal, natural-language problem statements and human-written arguments into syntactically precise, machine-verifiable objects suitable for automated reasoning, corpus construction, and AI research. The challenge is multi-faceted: it involves translating intricate and diverse mathematical domains (algebra, number theory, geometry, combinatorics) into structured formalisms, engineering supporting libraries and notations, designing benchmarks for evaluating both human and AI solvers, and architecting workflows that accommodate both manual and automated contributions. Over the last several years, a suite of datasets, frameworks, and evaluation harnesses have become essential resources for benchmarking and advancing the state of formalized mathematical reasoning at the Olympiad level.

## 1. Problem Selection and Domain Coverage

The selection of IMO problems for formalization typically spans various sources and mathematical domains, with datasets focusing on shortlist problems, official contests, or handpicked benchmarks. Notable efforts include:

- **FIMO** formalizes 149 algebra and number theory problems from the IMO Shortlist (2006–2021), achieving a 60.8% success rate for machine-assisted formalization (71.8% in algebra, 49.6% in number theory) [2309.04295].
- **CombiBench** covers combinatorics, including all IMO combinatorial problems since 2000 (excluding image-based ones), formalized in Lean 4 [2505.03171].
- **LeanGeo** spans geometry, incorporating all 43 IMO geometry problems since 2000 into Lean 4 [2508.14644].
- **miniF2F** provides a cross-system corpus with 40 IMO problems as part of a 488-problem set, covering algebra, number theory, and inequalities, suitable for Metamath, Lean, Isabelle, and HOL Light [2109.00110].
- **FormalGeo** encodes a smaller number (currently 18) of IMO geometry problems in a custom formal system, expandable via augmentation [2310.18021].
- **Lean-IMO Datasets** (e.g., "Small Steps…" [2411.18872]) provide detailed formalizations and lemma decompositions for all 20 miniF2F IMO test problems and selected recent IMOs.

Problem selection often omits classical geometry and advanced combinatorics in early datasets due to insufficient library support—though recent frameworks (LeanGeo, AlphaGeometry2, FormalGeo) now admit a much broader range, including locus problems, movement, and non-constructive statements [2502.03544, 2310.18021].

## 2. Formalization Languages and Representational Strategies

The predominant target languages are Lean (both v3+mathlib and v4+mathlib4), Isabelle/HOL, and custom DSLs for geometry.

- **Lean (v3/v4):** Theorems are encoded with explicit parameter lists, hypotheses as assumptions, and conclusions as goals. Common imports include real numbers, sets, finsets (for combinatorics), and algebraic and geometric modules. Managed codebases often use naming conventions for easy cross-reference with public competition archives and mathlib [2309.04295, 2505.03171, 2508.14644, 2411.18872].
- **Isabelle/HOL:** Statements utilize nat, int, real, and set types with Isar-style block-structured proofs and abundant use of automation (e.g., sledgehammer, metis) [2010.16015].
- **Custom Geometry DSLs:** Both AlphaGeometry2 and FormalGeo introduce expressive syntaxes with primitive predicates (collinearity, concyclicity, equal angles, ratios, locus queries). AlphaGeometry2 offers 88 predicates and 196 theorems, balancing succinctness and coverage [2502.03544, 2310.18021].
- **Hybrid, Multi-Assistant Benchmarks:** miniF2F and RIMO coordinate formalizations across Metamath, Lean, Isabelle, and HOL Light to ensure cross-system comparability and stress-test formal reasoning capabilities [2109.00110, 2509.07711].

Representative encodings often closely mirror the original LaTeX, but always isolate all parameters, hypotheses, and quantifiers. For answer-seeking (classification) problems, goals are reframed as uniqueness or sum-formulas to admit single-valued outputs, as seen in RIMO-N [2509.07711].

## 3. Methodologies for Formalization: Human-in-the-Loop, Automation, and Decomposition

Formalization methodologies span the spectrum from fully manual to hybrid human–AI pipelines:

- **Manual Formalization:** Practitioners encode problem statements and proofs stepwise, as in miniF2F, Isabelle/HOL-IMO, and the "Small Steps..." dataset [2109.00110, 2010.16015, 2411.18872]. This ensures semantic fidelity but is labor-intensive.
- **Semi-Automatic with Reflection:** FIMO demonstrates an iterative "autoformalization-with-reflection" workflow: GPT-4 proposes Lean statements, Lean is invoked to check/suggest error corrections, and humans verify semantic alignment. Up to five correction rounds boost success rates from 32.6% to 60.8% [2309.04295].
- **Decomposition into Lemmas:** Several benchmarks systematically break down full proofs into networks of intermediate lemmas:
    - RIMO-P decomposes each proof into 1-4 logically ordered subproblems, enabling stepwise grading [2509.07711].
    - The "Small Steps..." dataset extracts and curates 1,329 nontrivial lemmas from full IMO proofs, explicitly avoiding trivial (single-tactic) steps [2411.18872].
    - LEAP and Aristotle leverage blueprint/lemma planning, where informal arguments scaffold sketches of formal statements with auxiliary lemmas inserted as formal subgoals [2606.03303, 2510.01346].
- **Geometry-Specific Engineering:** AlphaGeometry2, FormalGeo, and LeanGeo implement domain-specific logics with hundreds of high-level predicates and theorems. They express constructions, metric relations, and non-constructive constraints (ratios, movement) through concise, extensible DSLs [2502.03544, 2310.18021, 2508.14644].

Typical formalization difficulties arise from quantifier misplacement, implicit/informal assumptions in original phrasing, library limitations for specialized constructs, and subtle mismatches between formal and informal semantics.

## 4. Benchmarking, Evaluation Protocols, and Automated Grading

Evaluation strategies aim to ensure rigor, reproducibility, and diagnostic clarity:

- **Exact-Match, Fill-in-the-Blank, and Deterministic Grading:** RIMO-N uses integer-only answer formats, so solution correctness is established by O(1) string matching; CombiBench's Fine-Eval protocol demands sorry-free Lean files compiling to specified values, with optional normalization for mathematically equivalent answers [2509.07711, 2505.03171].
- **Stepwise and Subproblem Evaluation:** RIMO-P introduces a LLM-based, JSON-formatted judge for subproblem chains; the model can only progress if all prior steps are deemed correct [2509.07711]. Similar decomposition enables fine-grained diagnosis of LLM failures in the "Small Steps..." and LEAP frameworks [2411.18872, 2606.03303].
- **Cross-System and Multi-Language Evaluation:** miniF2F aligns statements across four theorem provers, with baseline metrics like Pass@N (fraction solved after N attempts), average proof length, and per-category breakdowns [2109.00110].
- **Agentic and Self-Refining Search:** LEAP maintains an AND-OR DAG of goal decompositions, employing the Lean compiler as a search oracle and an LLM reviewer to regulate subgoal proposals and discourage unhelpful lemma restatements [2606.03303].
- **Geometry Benchmarks:** AlphaGeometry2, LeanGeo, and FormalGeo assess both coverage (percentage of IMO geometry problems expressible in their systems—88% for AlphaGeometry2) and solving success rates (AlphaGeometry2: 84% on IMO geometry, LeanGeo: near-zero for the hardest Olympiad problems with current LLMs) [2502.03544, 2508.14644].

## 5. Technical, Architectural, and Methodological Innovations

Several engineering and methodology themes recur across successful formalization efforts:

- **Feedback-Reflective Loops:** Iterative interaction between language models and proof assistants (error-correction cycles) substantially increases the yield of machine-consistent formalizations [2309.04295].
- **Lemma-Based Proof Search:** Incorporating autoformalized, human-suggested, or LM-extracted lemmas dramatically improves accessibility, success rates, and proof search tractability over single-stage, tactic-centric approaches [2510.01346, 2606.03303, 2507.06804].
- **Blueprint Scaffolding and AND-OR Structures:** Hierarchical memoization and anticipatory lemma planning—exemplified by LEAP—avoid exponential explosion in DFS-style search and can be coupled tightly to proof assistants' feedback or external reviewers [2606.03303].
- **Geometry DSLs and Engineered Solvers:** For geometry, domain-specific languages (AlphaGeometry2, FormalGeo, LeanGeo) encode constructions, constraints, and determination theorems, closely mirroring natural geometric argumentation and supporting non-constructive, locus-based, and metric relations [2502.03544, 2310.18021, 2508.14644].
- **Benchmark Construction and Ground-Truth Cross-Verification:** RIMO and Lean-IMO-Bench utilize expert-checked or community-validated ground truth for both answer and proof formats, ensuring reliable and reproducible evaluation [2509.07711, 2606.03303].
- **Extensibility and Data Augmentation:** Frameworks such as FormalGeo and AlphaGeometry2 support seamless addition of new predicates and theorems, facilitating broadening of coverage and finer stratification by difficulty or topic [2310.18021, 2502.03544].

## 6. Persistent Challenges, Limitations, and Future Directions

Challenges persist on several fronts:

- **Library Maturity and Expressivity:** Many IMO geometry and combinatorics problems remain outside reach of general-purpose theorem prover libraries, requiring continual extension (especially for advanced combinatorial objects, geometric transformations, and movement/locus queries) [2309.04295, 2505.03171, 2502.03544].
- **Quantifier Handling and Natural-Language Drift:** Autoformalization remains brittle to quantifier order, implicit structural constraints, and subtle distinctions between existence, uniqueness, and classification problems [2309.04295, 2509.07711].
- **Proof Construction Gaps:** LLMs, even with blueprint or lemma guidance, often misapply tactics, introduce mathematically invalid arguments, or conflate similar hypotheses, especially on the most challenging problems (e.g., recent IMO shortlist or combinatorial geometry) [2309.04295, 2508.14644, 2411.18872].
- **Deterministic Scoring vs. Expressive Goals:** Answer-based benchmarks (e.g., RIMO-N) trade off mathematical richness for deterministic evaluation, while proof-based approaches require robust and objective stepwise judging [2509.07711].
- **Geometry-Specific Tactics:** Embedding SMTs in geometry (as in LeanGeo) improves some automation, but leveraging area-method, coordinate, or algebraic approaches remains an open engineering problem [2508.14644].
- **Automated Lemma Utilization:** Provers often fail to call upon pre-proved lemmas unless specifically guided or fine-tuned for context sensitivity, leading to search inefficiency and missed integrations [2507.06804].

Planned extensions include expanded coverage within geometry and combinatorics, cross-assistant pipelines for Lean, Isabelle, and Coq, tighter integration with tactic-guidance tools (Tactic Toe, Sledgehammer, library_search), and dynamic embedding of formal benchmarks into neural proof search agents [2309.04295, 2606.03303].

## 7. Impact, Resources, and Community Practices

The systematic formalization of IMO problems accelerates research into AI-based mathematical reasoning and theorem proving, underpins new agentic architectures, and provides public benchmarks essential for replicable progress and fair comparison. Publicly available datasets (FIMO, miniF2F, RIMO, LeanGeo-Bench, CombiBench, formalgeo7k, IMO-Steps) serve both as evaluation suites and as foundational corpora for training, fine-tuning, and ablation studies.

Key best practices, distilled from current work [2411.18872], include decomposing proofs into nontrivial lemma units, maintaining explicit datasheets for topics and difficulty, careful library management and versioning, and the use of rigorous, publication-grade verification and grading pipelines. The formalization of IMO problems thus constitutes both a technical and community-driven bridge from natural, creative mathematical problem solving to fully machine-verifiable, extensible, and scalable mathematical knowledge systems.

Source: https://www.emergentmind.com/topics/formalization-of-imo-problems