---
title: 'ExamFormal-Bench: Formal Reasoning Benchmark'
url: https://www.emergentmind.com/topics/examformal-bench
type: topic
---

# ExamFormal-Bench: Formal Reasoning Benchmark

ExamFormal-Bench is a publicly released benchmark suite for evaluating the formal reasoning and autoformalization capabilities of large language models (LLMs) and formal theorem provers, with a particular focus on translation from informal (natural-language) mathematical statements to fully formal Lean 4 theorems, and on proof synthesis for competition-level mathematics. As of 2025, the term "ExamFormal-Bench" denotes both a set of formally aligned math problem datasets—most notably the 402-problem suite curated for Spark-Prover-X1—and a programmatic agenda for raising standards of completeness, transparency, and reproducibility in formal reasoning benchmarks [2511.13043][2406.06555][2507.04719][2109.00110].

## 1. Origins and Motivations

The design and deployment of ExamFormal-Bench responds to the critical need for rigorous, system-agnostic benchmarks in neural theorem proving and autoformalization. Earlier datasets such as miniF2F established the paradigm of cross-system alignment (e.g., Lean, Metamath, HOL Light, Isabelle) for 488 Olympiad and undergraduate-level problems, enabling apples-to-apples comparison across different proof assistants [2109.00110]. However, prior benchmarks suffered from limitations: incomplete pairing of informal and formal statements, lack of ground-truth proofs, small scale, and insufficient error-tracking. ExamFormal-Bench builds directly on these foundations by providing a single, human-vetted, multimodal test suite to evaluate real-world exam performance of LLM-based formal reasoning agents [2511.13043][2507.04719].

## 2. Dataset Structure and Construction

ExamFormal-Bench (in its canonical form, as released by Spark-Prover-X1) comprises 402 independently formalized Lean 4 theorem statements. Each problem is derived from real-world mathematical competitions and university qualifying exams, spanning three academic levels: middle school, high school, and undergraduate [2511.13043]. Problems were collected from official transcripts, processed via manual OCR and normalization, auto-formalized by an ensemble of LLM formalizers, and subjected to rigorous human review. Near-duplicate items (cosine similarity > 90%) were merged, resulting in a topic-balanced and non-redundant benchmark.

Problems are distributed evenly across six major domains:

- Analysis
- Geometry
- Algebra
- Probability & Statistics
- Computational Mathematics
- Discrete Mathematics (including combinatorics)

Each problem includes a Lean theorem statement (with any required imports and a proof skeleton), adhering to Mathlib4 compatibility and verified for successful Lean 4 compilation [2511.13043].

## 3. Evaluation Protocols and Metrics

Benchmarking on ExamFormal-Bench uses the pass@k metric for proof synthesis: if a prover emits $k$ independent proof candidates for a theorem, $\mathrm{pass@}k$ is the probability that at least one compiles to a complete Lean 4 proof:

\[
\mathrm{pass@}k = 1 - \prod_{i=1}^{k} (1 - n_\mathrm{correct}/n_\mathrm{total})
\]

The default evaluation regime fixes $k=32$ (pass@32) and all benchmarking is done in "whole-proof" mode under this computational budget. No train/val/test splits are introduced; the entire set is held out for evaluation [2511.13043].

Representative baseline results (pass@32):

| Prover                     | Pass@32 (%) |
|----------------------------|-------------|
| Spark-Prover-X1-7B         | 51.2        |
| DeepSeek-Prover-V2-7B      | 49.0        |
| Gödel-Prover-V2-8B        | 48.8        |
| Kimina-Prover-Distill-8B   | 45.3        |

For statement autoformalization, pass@8 is standard, and the strongest autoformalizers (Kimina-Formalizer-7B, Gödel-Formalizer-V2-8B, Spark-Formalizer-X1-7B) exceed 97% [2511.13043].

## 4. Problem Taxonomy and Examples

Uniquely, ExamFormal-Bench tags problems with lightweight ontologies (e.g., “inequalities,” “induction,” “counting,” “graph-theoretic,” “integral,” “matrix”). This enables stratified analysis and uniform sampling [2511.13043].

**Example 1: Algebra (Inequality)**
```lean
import Mathlib.Analysis.SpecialFunctions.Log
theorem examformal_algebra_ineq
  {a b c : ℝ} (ha : 0 < a) (hb : 0 < b) (hc : 0 < c) :
  (a + b + c) * (1 / a + 1 / b + 1 / c) ≥ 9 := by
  -- proof by AM–HM
  have h₁ : (a+b+c) / 3 ≥ 3 / (1/a + 1/b + 1/c) := by
    simpa only [div_eq_inv_mul] using (AM_le_HM (· / ·) 3 (fun i => ?_)).reverse
  simpa [mul_div_right_comm] using (mul_le_mul' h₁ (le_rfl : _ ≤ 1) (add_nonneg ha.lb hb.lb) (add_nonneg ha.lb hb.lb))
```
**Example 2: Discrete Mathematics (Combinatorics)**
```lean
import Mathlib.Combinatorics.SimpleGraph
import Mathlib.Data.Finset.Basic
theorem examformal_combi_count {n k : ℕ} (hk : k ≤ n) :
  (finset.range n).card.choose k = n.choose k := by
  -- proof by properties of finset.card and choose
  simp only [Finset.card_range, choose_eq_binomial]
```

## 5. Methodological Principles and Completeness

ExamFormal-Bench exemplifies benchmark completeness as defined in [2507.04719]: a complete formal reasoning benchmark contains four quadrants—(i) informal problem statements, (ii) formal problem statements, (iii) informal proofs, and (iv) formal proofs. While the initial dataset comprises only (i) and (ii), best practices outlined for future versions call for paired proofs in both natural language and Lean, with all statements and proofs verified for correctness and tracked for any error rate:

\[
\text{ErrorRate} = \frac{E}{N}
\]

where $E$ is the number of mistakes and $N$ the total number of theorems. Openness is enforced by releasing the dataset and code under OSI-approved licenses, providing all evaluation scripts and Docker-based reproducibility, and instituting CI pipelines for continuous verification [2507.04719].

## 6. Comparative Context and Impact

ExamFormal-Bench situates itself among a growing ecosystem of formal reasoning benchmarks:

- **miniF2F**: 488 problems, cross-system alignment, no paired informal proofs [2109.00110].
- **PutnamBench, ProofNet**: undergraduate/Putnam-level problems, typically only formal statements released.
- **FATE series**: targets graduate and research-level algebra, with FATE-H/X explicitly pushing beyond contest math [2511.02872].
- **FormalMATH**: large-scale (5560 problems), human-in-the-loop formalization pipeline, highlights autoformalization and scaling challenges [2505.02735].
- **VeriEquivBench/FormalSpecCpp**: focus on C++ and Dafny program verification, not theorem proving, but reflects similar completeness/openness trends [2510.06296][2502.15217].

ExamFormal-Bench is distinctive in its exclusive focus on real exam problems, fine-grained topic balancing, rigorous LLM + human curation, and in serving as a realistic test for models with practical exam-focused ambitions.

## 7. Future Directions and Best Practices

Key improvement paths identified [2507.04719][2406.06555]:

- Augmenting the benchmark with full informal and formal proofs for every problem to achieve strict completeness.
- Assigning detailed difficulty metadata and richer topic ontologies.
- Continuous community-driven addition of new problems and erratum-tracked patching.
- Locking verification to a stable Lean version and Docker image for environmental reproducibility.
- Publishing all model submissions, proofs, and metadata for direct ablation and secondary analysis.
- Addressing version drift and overfitting by maintaining a hidden subset reserved for final evaluation.

Adoption of these principles positions ExamFormal-Bench as a reference “exam bench” for formal mathematics reasoning, automated grading, and development of next-generation formal theorem provers.

---

**References:**  
- [2511.13043] Spark-Prover-X1: Formal Theorem Proving Through Diverse Data Training  
- [2406.06555] An Evaluation Benchmark for Autoformalization in Lean4  
- [2507.04719] Advocate for Complete Benchmarks for Formal Reasoning with Formal/Informal Statements and Formal/Informal Proofs  
- [2109.00110] MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics  
- [2505.02735] FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models

Source: https://www.emergentmind.com/topics/examformal-bench