FormalQualBench: Lean 4 Proof Benchmark

• FormalQualBench is a benchmark of 23 PhD-qualifying exam theorems formalized in Lean 4, emphasizing long-horizon proof planning and kernel-verified correctness.
• It spans diverse mathematical domains such as algebra, topology, and combinatorics, challenging systems to maintain coherent, global proof decomposition.
• The benchmark enforces strict evaluation criteria, ensuring proofs preserve original theorem signatures without shortcuts like 'sorry' or user-introduced axioms.

FormalQualBench is a benchmark of 23 PhD-qualifying-exam theorems formalized in Lean 4, presented as a hard, graduate-level theorem-proving suite for evaluating end-to-end proof completion rather than local tactic search. In the available literature, it appears most concretely as the primary benchmark used to evaluate MerLean-Prover, where it is treated as a set of difficult, long-horizon formal proof problems whose solution requires sustaining a coherent proof plan over many dependent lemmas, with success determined by kernel-checkable correctness conditions rather than informal plausibility (Li et al., 26 May 2026). A separate position paper invokes “a benchmark like FormalQualBench” as the benchmark ideal for formal reasoning: complete, open, error-free, and matched across formal and informal artifacts (Yousefzadeh et al., 7 Jul 2025).

1. Definition and provenance

FormalQualBench is described as a benchmark of 23 PhD-qualifying-exam theorems formalized in Lean 4. In the MerLean-Prover study, it is characterized as a hard, graduate-level Lean 4 theorem-proving suite and used as the main testbed for end-to-end theorem proving under strict kernel-oriented evaluation (Li et al., 26 May 2026).

The same source is explicit about provenance limits. It does not describe FormalQualBench as a newly constructed benchmark in that paper; rather, it cites FormalQualBench as an external benchmark from Math, Inc. The paper reports its size, theorem names, graduate-level character, and evaluation role, but does not present a separate benchmark-construction methodology there. This is important because some properties often expected of modern formal benchmarks—such as paired informal statements, reviewed informal proofs, or explicit data-release policy—are not stated in that source as benchmark facts.

As used in MerLean-Prover, FormalQualBench functions as a stress test for long-horizon proof synthesis. The intended difficulty is not merely finding the next valid tactic, but maintaining a globally coherent decomposition whose intermediate lemmas remain mathematically correct, Lean-elaborable, and faithful to the anchored target theorem.

2. Problem map and mathematical scope

The appendix problem map reported for FormalQualBench spans classic, nontrivial domains including analysis, algebra, combinatorics, number theory, topology, and logic/quantifier elimination (Li et al., 26 May 2026). The theorem set is as follows.

ID	Theorem	Status in MerLean-Prover
1	BanachStoneTheorem	Solved
2	BurnsidePrimeDegreeTheorem	Solved
3	ColorfulCaratheodoryTheorem	Solved
4	DeBruijnErdos	Solved
5	DLOQuantifierElimination	Solved
6	GleasonKahaneZelazkoTheorem	Solved
7	JordanDerangementTheorem	Solved
8	ParisHarringtonPrinciple	Solved
9	RungeTheorem	Solved
10	VonNeumannDoubleCommutantTheorem	Solved
11	BorsukUlamTheorem	Remaining
12	CollatzMapAlmostBoundedValues	Remaining
13	ErdosDiscrepancyProblem	Remaining
14	GreenTaoTheorem	Remaining
15	Hilbert17thProblem	Remaining
16	JordanCycleTheorem	Remaining
17	KakeyaTheorem3D	Remaining
18	MaynardTaoBoundedPrimeGaps	Remaining
19	PontryaginDuality	Remaining
20	QuillenSuslinTheorem	Remaining
21	SchauderFixedPointTheorem	Remaining
22	SkolemMahlerLechTheorem	Remaining
23	TernaryGoldbachTheorem	Remaining

This theorem list makes the benchmark unusual in two respects. First, the objects of evaluation are not short textbook lemmas but named theorems with substantial mathematical depth. Second, the distribution of topics implies heterogeneous Lean burdens: abstract algebraic structure, topological reasoning, measure- or analysis-adjacent argument forms, and logic-specific formalization patterns all appear within a single suite.

A plausible implication is that FormalQualBench is designed less as a narrow leaderboard set than as a probe of whether a system can sustain proof planning across very different mathematical regimes.

3. Evaluation semantics and correctness criteria

FormalQualBench is evaluated under a strict kernel-oriented regime. In the MerLean-Prover setup, a problem is counted as solved only if the produced Lean file builds cleanly with lake build, preserves the original theorem signature, and passes a transitive #print axioms audit (Li et al., 26 May 2026).

The permitted axioms in that audit are exactly:

$$\{ \texttt{propext},\ \texttt{Quot.sound},\ \texttt{Classical.choice} \}.$$

The audit excludes sorryAx, and the final proof must contain no forbidden sorry, admit, or user-introduced axiom. The benchmark is therefore not scored by partial progress, informal readability, or proof plausibility. It is scored by end-to-end proof completion under Lean’s correctness mechanisms and an external axiom check.

The reported time budget is ordinarily 4 hours wall-clock per problem unless otherwise stated. One FormalQualBench solve, identified as ID 9, finishes in 4h40m and is still counted in the reported total (Li et al., 26 May 2026). This makes the evaluation neither purely interactive nor purely one-shot; it is long-horizon and operationally expensive.

The MerLean paper also gives a compact success characterization for runs on this benchmark: every statement in the current plan must be closed, the target theorem must still have the original Lean signature, and the final proof must contain no forbidden placeholder or user axiom. In effect, FormalQualBench is treated as a benchmark for anchored theorem preservation plus proof completion, not merely for generating compilable Lean fragments.

4. Reported benchmark results

The headline result reported on FormalQualBench is MerLean-Prover: 10/23 versus OpenGauss: 8/23, with MerLean-Prover described as surpassing the strongest published open-source baseline on this benchmark (Li et al., 26 May 2026).

The same evaluation reports that 9 of the 10 solved problems close within the normal 4-hour budget, while 1 problem closes in an extended 4h40m run. Across the ten solved problems, the total LLM API spend is \$1,183.45**, the total **wall-clock time** is **19h54m**, and the averages per solved problem are **\$118.35 and 1h59m.

The per-problem breakdown supplied for the ten solved theorems is notably uneven. At the low end, ID 4 closes in 11m at \$10.05**, while **ID 7** closes in **16m** at **\$13.53. At the high end, ID 1 takes 3h41m at \$220.50**, and **ID 9** takes **4h40m** at **\$273.66. This dispersion indicates that FormalQualBench is not uniform in difficulty even within the solved subset.

The benchmark also supports smaller controlled subsets. MerLean-Prover reports a subset of four FormalQualBench problems—IDs 1, 3, 4, 7—used for stability and model-variant experiments. On that subset, Sonnet closes all four tested FormalQualBench problems, and Haiku closes the two short ones (Li et al., 26 May 2026). This suggests that benchmark behavior can be meaningfully analyzed both at full-suite scale and on curated subsets.

5. BurnsidePrimeDegreeTheorem as a canonical benchmark instance

The worked example used to illustrate FormalQualBench in MerLean-Prover is BurnsidePrimeDegreeTheorem. It concerns transitive permutation groups of prime degree: for a transitive subgroup $G \le \mathrm{Equiv.Perm}\,\alpha$ with $\#\alpha$ prime, either $G$ is 2-transitive, or there exists a normal subgroup $N \le G$ that is transitive and has trivial stabilizers, hence acts regularly (Li et al., 26 May 2026).

The appendix includes the Lean shape of the main theorem: $1,183.45**, the total **wall-clock time** is **19h54m**, and the averages per solved problem are **\$0

The reported proof sketch extracts an element of prime order, defines $N := \mathrm{Subgroup.zpowers}\,\tau$, proves that $N$ is pretransitive with trivial stabilizers, and establishes normality by a conjugation argument. The helper components explicitly named in the appendix are exists_element_of_prime_order, Subgroup.zpowers τ, mueller_affine_conjugation, cyclicSubgroup_isPretransitive, and cyclicSubgroup_stabilizer_eq_bot.

BurnsidePrimeDegreeTheorem is also used to expose the benchmark’s characteristic failure modes. Three are singled out. Faithfulness failure occurs when a clean-build Lean file is rejected because it weakens the original theorem signature. Mathematical-correction failure occurs when a proposed helper statement is mathematically wrong or omits a needed hypothesis. Decomposition-driven replanning occurs when a statement is sound but still too hard, requiring it to be split into smaller helpers. In the reported trace, the proof plan grows from 6 nodes → 12 nodes → 21 nodes → 32 nodes before the dependency graph stabilizes (Li et al., 26 May 2026).

This example clarifies why FormalQualBench is not well modeled as a benchmark for tactic prediction alone. The core difficulty is managing global proof structure under repeated revision while preserving the anchored target theorem.

6. Benchmark ideal, completeness debates, and relation to adjacent formal benchmarks

A position paper on benchmark quality in formal reasoning uses FormalQualBench as the name for the kind of benchmark that should contain all four components: informal statements, formal statements, informal proofs, and formal proofs (Yousefzadeh et al., 7 Jul 2025). In that paper’s formulation, such a benchmark should also be complete, error-free, and open, with transparent code, data, and evaluation procedures. The same paper argues that formal verification can certify formal proofs, but cannot by itself certify that a formalized statement is the correct translation of an informal one.

This creates an important distinction between the benchmark as reported usage and the benchmark as methodological ideal. The MerLean-Prover paper gives a concrete theorem suite and strict kernel-level solve criteria, but does not state that FormalQualBench itself was released with all four artifacts. The position paper, by contrast, uses “a benchmark like FormalQualBench” normatively, to argue that benchmark creators should release the full formal/informal stack rather than only formal statements (Yousefzadeh et al., 7 Jul 2025).

FormalQualBench also belongs to a broader 2025–2026 shift toward deterministic evaluation in formal reasoning. Nearby examples include FormalProofBench, where each task pairs a natural-language mathematics problem with a Lean 4 formal statement and success requires a proof accepted by the Lean 4 kernel (Ravi et al., 27 Mar 2026); FormalRewardBench, which evaluates reward models on Lean 4 proof-preference pairs rather than theorem proving directly (Uluşan et al., 11 May 2026); and VeriEquivBench, which replaces ground-truth specification matching with bidirectional code–spec equivalence checking in Dafny (Zeng et al., 7 Oct 2025). This suggests that FormalQualBench is best understood as part of a larger movement away from judge-based or weakly specified evaluation and toward verifiable, execution- or kernel-grounded correctness criteria.

Within that landscape, FormalQualBench’s distinctive role is its focus on graduate-level, long-horizon Lean 4 theorem proving over named theorems. It is therefore less a benchmark for isolated proof steps than a benchmark for sustained formal mathematical construction under strict signature preservation and axiom discipline.