---
title: 'FormalQualBench: Lean 4 Proof Benchmark'
url: https://www.emergentmind.com/topics/formalqualbench
type: topic
---

# FormalQualBench: Lean 4 Proof Benchmark

FormalQualBench is a benchmark of **23 PhD-qualifying-exam theorems formalized in Lean 4**, presented as a hard, graduate-level theorem-proving suite for evaluating end-to-end proof completion rather than local tactic search. In the available literature, it appears most concretely as the primary benchmark used to evaluate MerLean-Prover, where it is treated as a set of difficult, long-horizon formal proof problems whose solution requires sustaining a coherent proof plan over many dependent lemmas, with success determined by kernel-checkable correctness conditions rather than informal plausibility [2605.26959]. A separate position paper invokes “a benchmark like FormalQualBench” as the benchmark ideal for formal reasoning: complete, open, error-free, and matched across formal and informal artifacts [2507.04719].

## 1. Definition and provenance

FormalQualBench is described as a benchmark of **23 PhD-qualifying-exam theorems formalized in Lean 4**. In the MerLean-Prover study, it is characterized as a **hard, graduate-level Lean 4 theorem-proving suite** and used as the main testbed for end-to-end theorem proving under strict kernel-oriented evaluation [2605.26959].

The same source is explicit about provenance limits. It does **not** describe FormalQualBench as a newly constructed benchmark in that paper; rather, it cites FormalQualBench as an **external benchmark from Math, Inc.** The paper reports its size, theorem names, graduate-level character, and evaluation role, but does not present a separate benchmark-construction methodology there. This is important because some properties often expected of modern formal benchmarks—such as paired informal statements, reviewed informal proofs, or explicit data-release policy—are not stated in that source as benchmark facts.

As used in MerLean-Prover, FormalQualBench functions as a stress test for long-horizon proof synthesis. The intended difficulty is not merely finding the next valid tactic, but maintaining a globally coherent decomposition whose intermediate lemmas remain mathematically correct, Lean-elaborable, and faithful to the anchored target theorem.

## 2. Problem map and mathematical scope

The appendix problem map reported for FormalQualBench spans classic, nontrivial domains including **analysis, algebra, combinatorics, number theory, topology, and logic/quantifier elimination** [2605.26959]. The theorem set is as follows.

| ID | Theorem | Status in MerLean-Prover |
|---|---|---|
| 1 | BanachStoneTheorem | Solved |
| 2 | BurnsidePrimeDegreeTheorem | Solved |
| 3 | ColorfulCaratheodoryTheorem | Solved |
| 4 | DeBruijnErdos | Solved |
| 5 | DLOQuantifierElimination | Solved |
| 6 | GleasonKahaneZelazkoTheorem | Solved |
| 7 | JordanDerangementTheorem | Solved |
| 8 | ParisHarringtonPrinciple | Solved |
| 9 | RungeTheorem | Solved |
| 10 | VonNeumannDoubleCommutantTheorem | Solved |
| 11 | BorsukUlamTheorem | Remaining |
| 12 | CollatzMapAlmostBoundedValues | Remaining |
| 13 | ErdosDiscrepancyProblem | Remaining |
| 14 | GreenTaoTheorem | Remaining |
| 15 | Hilbert17thProblem | Remaining |
| 16 | JordanCycleTheorem | Remaining |
| 17 | KakeyaTheorem3D | Remaining |
| 18 | MaynardTaoBoundedPrimeGaps | Remaining |
| 19 | PontryaginDuality | Remaining |
| 20 | QuillenSuslinTheorem | Remaining |
| 21 | SchauderFixedPointTheorem | Remaining |
| 22 | SkolemMahlerLechTheorem | Remaining |
| 23 | TernaryGoldbachTheorem | Remaining |

This theorem list makes the benchmark unusual in two respects. First, the objects of evaluation are not short textbook lemmas but named theorems with substantial mathematical depth. Second, the distribution of topics implies heterogeneous Lean burdens: abstract algebraic structure, topological reasoning, measure- or analysis-adjacent argument forms, and logic-specific formalization patterns all appear within a single suite.

A plausible implication is that FormalQualBench is designed less as a narrow leaderboard set than as a probe of whether a system can sustain proof planning across very different mathematical regimes.

## 3. Evaluation semantics and correctness criteria

FormalQualBench is evaluated under a strict **kernel-oriented** regime. In the MerLean-Prover setup, a problem is counted as solved only if the produced Lean file **builds cleanly** with `lake build`, **preserves the original theorem signature**, and passes a **transitive `#print axioms` audit** [2605.26959].

The permitted axioms in that audit are exactly:
$$
\{ \texttt{propext},\ \texttt{Quot.sound},\ \texttt{Classical.choice} \}.
$$

The audit excludes `sorryAx`, and the final proof must contain **no forbidden `sorry`, `admit`, or user-introduced axiom**. The benchmark is therefore not scored by partial progress, informal readability, or proof plausibility. It is scored by end-to-end proof completion under Lean’s correctness mechanisms and an external axiom check.

The reported time budget is ordinarily **4 hours wall-clock per problem** unless otherwise stated. One FormalQualBench solve, identified as **ID 9**, finishes in **4h40m** and is still counted in the reported total [2605.26959]. This makes the evaluation neither purely interactive nor purely one-shot; it is long-horizon and operationally expensive.

The MerLean paper also gives a compact success characterization for runs on this benchmark: every statement in the current plan must be closed, the target theorem must still have the original Lean signature, and the final proof must contain no forbidden placeholder or user axiom. In effect, FormalQualBench is treated as a benchmark for **anchored theorem preservation plus proof completion**, not merely for generating compilable Lean fragments.

## 4. Reported benchmark results

The headline result reported on FormalQualBench is **MerLean-Prover: 10/23** versus **OpenGauss: 8/23**, with MerLean-Prover described as surpassing the strongest published open-source baseline on this benchmark [2605.26959].

The same evaluation reports that **9 of the 10 solved problems** close within the normal **4-hour budget**, while **1 problem** closes in an extended **4h40m** run. Across the ten solved problems, the total **LLM API spend** is **\$1,183.45**, the total **wall-clock time** is **19h54m**, and the averages per solved problem are **\$118.35** and **1h59m**.

The per-problem breakdown supplied for the ten solved theorems is notably uneven. At the low end, **ID 4** closes in **11m** at **\$10.05**, while **ID 7** closes in **16m** at **\$13.53**. At the high end, **ID 1** takes **3h41m** at **\$220.50**, and **ID 9** takes **4h40m** at **\$273.66**. This dispersion indicates that FormalQualBench is not uniform in difficulty even within the solved subset.

The benchmark also supports smaller controlled subsets. MerLean-Prover reports a **subset of four FormalQualBench problems**—**IDs 1, 3, 4, 7**—used for stability and model-variant experiments. On that subset, **Sonnet closes all four tested FormalQualBench problems**, and **Haiku closes the two short ones** [2605.26959]. This suggests that benchmark behavior can be meaningfully analyzed both at full-suite scale and on curated subsets.

## 5. BurnsidePrimeDegreeTheorem as a canonical benchmark instance

The worked example used to illustrate FormalQualBench in MerLean-Prover is **BurnsidePrimeDegreeTheorem**. It concerns transitive permutation groups of prime degree: for a transitive subgroup \(G \le \mathrm{Equiv.Perm}\,\alpha\) with \(\#\alpha\) prime, either \(G\) is **2-transitive**, or there exists a normal subgroup \(N \le G\) that is transitive and has trivial stabilizers, hence acts regularly [2605.26959].

The appendix includes the Lean shape of the main theorem:
```lean
theorem MainTheorem {α : Type*} [Fintype α] {G : Subgroup (Equiv.Perm α)}
  (htrans : IsPretransitive G α) (hp : (Fintype.card α).Prime) :
  IsMultiplyPretransitive G α 2 ∨
  ∃ N : Subgroup G, N.Normal ∧ IsPretransitive N α ∧
    ∀ a : α, MulAction.stabilizer N a = ⊥ := by
```

The reported proof sketch extracts an element of prime order, defines \(N := \mathrm{Subgroup.zpowers}\,\tau\), proves that \(N\) is pretransitive with trivial stabilizers, and establishes normality by a conjugation argument. The helper components explicitly named in the appendix are `exists_element_of_prime_order`, `Subgroup.zpowers τ`, `mueller_affine_conjugation`, `cyclicSubgroup_isPretransitive`, and `cyclicSubgroup_stabilizer_eq_bot`.

BurnsidePrimeDegreeTheorem is also used to expose the benchmark’s characteristic failure modes. Three are singled out. **Faithfulness failure** occurs when a clean-build Lean file is rejected because it weakens the original theorem signature. **Mathematical-correction failure** occurs when a proposed helper statement is mathematically wrong or omits a needed hypothesis. **Decomposition-driven replanning** occurs when a statement is sound but still too hard, requiring it to be split into smaller helpers. In the reported trace, the proof plan grows from **6 nodes → 12 nodes → 21 nodes → 32 nodes** before the dependency graph stabilizes [2605.26959].

This example clarifies why FormalQualBench is not well modeled as a benchmark for tactic prediction alone. The core difficulty is managing global proof structure under repeated revision while preserving the anchored target theorem.

## 6. Benchmark ideal, completeness debates, and relation to adjacent formal benchmarks

A position paper on benchmark quality in formal reasoning uses **FormalQualBench** as the name for the kind of benchmark that should contain all four components: **informal statements, formal statements, informal proofs, and formal proofs** [2507.04719]. In that paper’s formulation, such a benchmark should also be **complete**, **error-free**, and **open**, with transparent code, data, and evaluation procedures. The same paper argues that formal verification can certify formal proofs, but cannot by itself certify that a formalized statement is the correct translation of an informal one.

This creates an important distinction between the benchmark as **reported usage** and the benchmark as **methodological ideal**. The MerLean-Prover paper gives a concrete theorem suite and strict kernel-level solve criteria, but does not state that FormalQualBench itself was released with all four artifacts. The position paper, by contrast, uses “a benchmark like FormalQualBench” normatively, to argue that benchmark creators should release the full formal/informal stack rather than only formal statements [2507.04719].

FormalQualBench also belongs to a broader 2025–2026 shift toward deterministic evaluation in formal reasoning. Nearby examples include **FormalProofBench**, where each task pairs a natural-language mathematics problem with a Lean 4 formal statement and success requires a proof accepted by the Lean 4 kernel [2603.26996]; **FormalRewardBench**, which evaluates reward models on Lean 4 proof-preference pairs rather than theorem proving directly [2605.10141]; and **VeriEquivBench**, which replaces ground-truth specification matching with bidirectional code–spec equivalence checking in Dafny [2510.06296]. This suggests that FormalQualBench is best understood as part of a larger movement away from judge-based or weakly specified evaluation and toward verifiable, execution- or kernel-grounded correctness criteria.

Within that landscape, FormalQualBench’s distinctive role is its focus on **graduate-level, long-horizon Lean 4 theorem proving** over named theorems. It is therefore less a benchmark for isolated proof steps than a benchmark for sustained formal mathematical construction under strict signature preservation and axiom discipline.

Source: https://www.emergentmind.com/topics/formalqualbench