---
title: miniF2F & FIMO Math Benchmarks
url: https://www.emergentmind.com/topics/minif2f-and-fimo-benchmarks
type: topic
---

# miniF2F & FIMO Math Benchmarks

miniF2F and FIMO are formal mathematics benchmarks that define the current frontier for automated theorem proving at advanced competition and olympiad levels. miniF2F is designed as a broad, cross-system suite covering high-school and early undergraduate mathematics, emphasizing coverage across proof assistants and problem types, while FIMO concentrates exclusively on deeply creative algebra and number theory problems sourced from the International Mathematical Olympiad (IMO) Shortlist, pushing toward true IMO-level challenge. Both benchmarks have catalyzed the development of new methodologies in formal autoformalization, automated theorem proving (ATP), and large language model (LLM)-guided mathematical reasoning.

## 1. Benchmark Structure and Problem Sourcing

**miniF2F** comprises 488 formal problem statements (244 test / 244 validation), sourced from the AMC, AIME, IMO, and a wide selection of undergraduate mathematics topics. Of these, only 40 are authentic IMO/IMO Shortlist problems; the rest capture a gradient of difficulty spanning routine high-school exercises to olympiad-style challenges. Problems are meticulously formalized in multiple proof assistants, including Lean, Isabelle/HOL, Metamath, and HOL Light, to enable cross-system benchmarking and direct comparison of ATP approaches across different formal frameworks [2109.00110].

**FIMO** (Formal IMO-level) contains 149 (in some sources, 148) formal Lean statements, exclusively targeting algebra and number theory problems from the IMO Shortlists (2006–2021). Each FIMO entry is delivered as a triple: the LaTeX-formatted informal statement, the official IMO Shortlist proof in LaTeX, and a formal Lean encoding with a placeholder or completed proof. The selection process achieved a 60.8% formalization success rate over all available shortlist items, after human-in-the-loop verification of LLM-formalized problems [2309.04295].

| Benchmark   | # Problems | Math Level                 | System Coverage           | Informal Proofs |
|-------------|------------|----------------------------|---------------------------|------------------|
| miniF2F     | 488        | AMC/AIME/IMO/Undergrad     | Lean, Metamath, Others    | Source text only |
| FIMO        | 149        | IMO Shortlist (Algebra/NT) | Lean                      | Yes (LaTeX)      |

## 2. Formal Encoding and Representation

**miniF2F** statements are fully formalized and reviewed, with each problem represented as a formal theorem or lemma in the target proof system. Special care is taken to resolve informal mathematical ambiguity: multi-choice questions are rephrased as assertions fixing the correct answer, "witness" problems provide an explicit solution, and all necessary side-conditions are formalized (e.g., nonnegativity, triangle inequalities). The Lean and Metamath encodings (along with partial Isabelle/HOL, HOL Light ports) serve as the reference implementations. Manual formalization and review require approximately 22 minutes per problem [2109.00110]. 

**FIMO** employs explicit Lean theorem declarations, bundling all hypotheses as conjunctions and specifying all domain, positivity, and type constraints. The LaTeX informal statement and solution, provided for every problem, facilitate hybrid reasoning settings and support evaluation of both closed-problem ATP and informal-to-formal proof translation. This structural pairing is crucial for new "Draft, Sketch, and Prove" (DSP) workflows that link human and machine representations [2309.04295].

## 3. Evaluation Protocols and Metrics

Both benchmarks primarily use the **pass@k** metric for ATP performance: the fraction of problems solved (i.e., proofs accepted by the Lean kernel with no `sorry`) within k attempts per problem. 

Formally, for a benchmark with $|P|$ problems and $k$ independent proof attempts per problem,
$$
\mathrm{pass@}k = \frac{1}{|P|} \sum_{p \in P} \mathbf{1}( \exists\, j \leq k : \texttt{proof}_{p,j} \text{ accepts} )
$$
"Pass@N" refers to N independent decodings per theorem; "cumulative accuracy" is pass@$k_{\max}$ for the largest attempted k (often in the thousands for RL-based methods) [2405.14333].

Additional end-to-end metrics, as used in recent miniF2F-v2 analysis, require semantic alignment between the informal source statement, the formalization, and the accepted proof:
$$
\mathrm{Acc}_{\mathrm{E2E}} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\bigl( F_i \text{ compiles, proved, and matches } s_i \bigr)
$$
This corrects for mismatches and over-simplifications found in earlier miniF2F versions [2511.03108].

## 4. Automation, LLM-Guided Proving, and System Integration

**miniF2F-Dafny** translated the benchmark into Dafny, moving away from purely interactive proofs to "auto-active" verification. Dafny's SMT automation (Z3 backend) handled approximately 40–45% of miniF2F test/validation problems with "empty" (no-step) proofs. Remaining problems are addressed using LLMs to generate high-level proof hints; the best model (Claude Sonnet 4.5) achieved $\mathrm{Pass@4} = 55.7\%$ using up to 4 error-corrected attempts per problem. The system leverages a division of labor: LLMs propose strategic assertions and lemma applications, while Dafny discharges low-level equational and quantifier reasoning, eliminating the need for LLMs to supply detailed stepwise algebra [2512.10187].

A direct comparison to FIMO demonstrates the impact of automation: FIMO’s interactive Lean tasks admit less than 10% auto-active (script-free) solutions, and even LLM-guided approaches (GPT-f, etc.) reach only 20–30% $\mathrm{pass@4}$. FIMO’s higher difficulty reflects both the intrinsic mathematical depth and the current limits of ATP and system automation [2512.10187], [2309.04295].

## 5. Baseline Results and System-Level Comparisons

| Benchmark         | Auto-active (%) | LLM-guided (%)      | System    | Proof Style        |
|-------------------|----------------|---------------------|-----------|--------------------|
| miniF2F-Dafny     | 40–45           | ≈55.7% ($\mathrm{pass@4}$) | Dafny     | auto-active        |
| FIMO              | <10%            | ~30% ($\mathrm{pass@4}$)   | Lean      | interactive        |

On the original miniF2F test set (Lean), state-of-the-art models such as DeepSeek-Prover (7B, trained on 8M synthetic theorem–proof pairs) reach 52.0% cumulative accuracy with large sample budgets (65536 attempts), compared to GPT-4's 23.0% at 64 attempts. Hypertree Proof Search (MCTS with 64 beams × 5000 rollouts) achieves 41.0%. On FIMO, DeepSeek-Prover successfully proved 5/148 problems (3.4%) at large sample budgets, while GPT-4 solved none [2405.14333]. End-to-end pipeline accuracy on the revised miniF2F-v2 dataset stands at ≈44.7%–70% depending on adherence to problem fidelity and proof alignment practices [2511.03108].

## 6. Curation Practices, Limitations, and Future Development

Audits of miniF2F’s original releases revealed that over half of pipeline failures arose from informal–formal mismatches: incomplete or oversimplified informal statements, unprovable formalizations, or extraneous assumptions. The construction of miniF2F-v2 entailed systematic manual correction, restoring missing hypotheses and alignment, and introducing two-tier scoring ("simplified" versus "competition" variants). This curation approach, emphasizing rigorous manual vetting and human-verified translation, establishes necessary standards for reliable benchmark advancement [2511.03108].

FIMO leverages paired LaTeX statements/proofs and formalizations to support hybrid evaluation protocols. This enables research into DSP workflows, premise-selection augmentation, and multi-stage LLM-guided search. Current limitations for both benchmarks include under-representation of geometry/combinatorics in FIMO, sensitivity to proof assistant/library evolution, and a persistent gap between autoformalization and theorem-prover accuracy versus end-to-end gold-standard performance. 

**Future directions** include expansion of domain coverage (notably combinatorics/geometry in FIMO), improved autoformalization cycles, tree-search/LLM hybrid ATP, and graduated curricula for progressive capability assessment [2309.04295]. Continuous benchmark maintenance and explicit handling of multiple-choice/problem selection remain best practices at the field's frontier [2511.03108].

## 7. Significance and Impact

miniF2F and FIMO define and measure core objectives in formal mathematical automation. miniF2F’s breadth and multi-system coverage facilitate generalization studies and neural ATP comparisons at moderate-to-high difficulty. FIMO isolates the "IMO Grand Challenge" at scale, establishing that even state-of-the-art LLMs (GPT-4, DeepSeek-Prover) are far from robust, human-level performance. The design principles, curation efforts, and paired informal–formal structure underpin emerging research in both whole-proof automation and informal-to-formal pipeline development, directly shaping next-generation methodologies in machine mathematics [2512.10187], [2309.04295], [2405.14333], [2109.00110], [2511.03108].

Source: https://www.emergentmind.com/topics/minif2f-and-fimo-benchmarks