Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof. However, the kernel only checks that a proof establishes a \emph{formal} statement; it does not verify that the statement faithfully encodes the intended informal problem, nor that evaluation harnesses are robust to trivial or adversarial solutions. We audit five widely used Lean theorem-proving benchmarks and their forks, using corpus-scale static checkers to surface 4,833 findings, including 398 mechanically certified issues such as counterexamples, vacuous theorems, and unsound axioms. We also document semantic defects such as missing hypotheses, problem simplification, incomplete or incorrect translations, and Lean-specific specification hazards. Beyond dataset construction, we survey evaluation-time failure modes and show, on corrected subsets, that defects can both inflate and deflate reported prover scores. We propose a fault taxonomy, a suite of automated checkers and recall-oriented semantic audit prompts, and release standards to guide the creation of formal math datasets and to make evaluation more reproducible and trustworthy. Our checkers, audit prompts, and corrected dataset snapshots are available at https://github.com/Shashi456/atp-checkers.

• The paper reveals that pervasive defects in Lean benchmarking pipelines undermine metric reliability beyond kernel verification.
• The paper employs scalable detection methods, including static checkers and LLM-based prompts, to identify over 4,800 defects across multiple datasets.
• The paper offers actionable recommendations—such as banning unsafe tactics and version pinning—to enhance the trustworthiness of formal benchmarks.

Faults in Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

Overview

The paper "Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving" (2606.29493) systematically analyzes reliability issues in the current formal theorem proving benchmark ecosystem for Lean. Although the Lean kernel provides a high degree of certainty that a given proof artifact establishes a given formal statement, this does not imply that the formal statement is a correct or adequate translation of the original mathematical problem or that the evaluation harness is robust against loopholes and semantic drift. The authors perform a large-scale audit of five prominent Lean theorem proving datasets and their variants, introducing a structured taxonomy of defects, developing scalable detection methods—including static analyzers and LLM-based prompts—and highlighting their practical impact on reported metrics. Tools and recommendations for robust benchmark creation and maintenance are provided.

Formal Evaluation Pipeline and Error Taxonomy

The benchmarking pipeline is decomposed into five steps: from informal statement to formalization, prover, kernel verification, and the final reported metric. The only point of certified trust is kernel verification of formal statements. Crucially, errors can be introduced at every other stage: incorrect or ambiguous mathematical statements, faulty or incomplete formalization, exploitable evaluation harnesses, and maintenance-related drift.

Figure 1: The formal benchmarking pipeline, highlighting error injection points: fidelity issues in formalization, robustness loopholes at proof checking, and maintenance drift across the pipeline.

The paper develops a taxonomy dividing defects into three main categories:

• Fidelity Issues: Failures in the formalization step, including missing hypotheses, incorrect domains or types, incomplete translation, definition mismatches, vacuous hypotheses, and subtle encoding hazards (e.g., Lean's saturating natural number subtraction).
• Evaluation Loopholes: Vulnerabilities in the evaluation harness or Lean itself, such as improper axiom usage, reliance on unsound tactics (e.g., native_decide), acceptance of proofs via previously known Lean kernel bugs, and over-permissive automation.
• Maintenance Issues: Defects introduced or exacerbated over time due to software/library evolution, ill-posed or ambiguous original problems, and dataset fork proliferation with ambiguous provenance.

Systematic Audit and Detection Methods

The audit targets five major datasets: miniF2F, ProofNet, FormalMath, CombiBench, and ProverBench, including 13 variants and over 10,000 problems. To scale detection, the authors implement Lean 4 metaprograms as static checkers, capable of identifying mechanical defects such as division-by-zero, domain overflows, truncated subtraction, and vacuousness. These checkers (guarded by counterexample search and proof attempts for unsoundness) surfaced 4,833 findings, 398 with mechanically certified status (such as actual counterexamples or vacuous theorems), highlighting widespread issues not caught by kernel checking alone.

Complementing static analysis, LLM-based promps are used to screen for semantic mismatches in specification or translation—errors static checking cannot decide (e.g., omitted hypotheses, definition mismatches, quantifier scope violations). Empirically, LLM-based review achieves high recall but moderate precision, performing best for explicit specification or definition mismatches and less well for nuanced formalization errors.

Failure Modes and Case Analysis

The audit provides comprehensive evidence of both inflationary and deflationary impacts on reported prover scores. Flawed benchmarks can include unprovable statements that silently suppress pass rates, or weakened/formalizations that overstate model ability. Correcting such benchmark defects led to nontrivial changes (both increases and decreases) in measured solver success rates, illustrating the non-monotonic and confounding role of dataset quality on metric reliability.

Key illustrative examples include:

• Omission of critical hypotheses (e.g., finite-dimensionality in linear algebra problems) resulted in easier-than-intended proofs.
• Faulty domain translation (e.g., using natural instead of integer type) led to silent failure via truncation or saturation.
• Definition drift (e.g., encoding perfect sets differently across mathlib versions) obfuscated cross-benchmark comparability.
• Version/fork proliferation (especially for miniF2F), along with unrecorded corrections, make attribution and reproducibility problematic.

Recommendations and Release Standards

To address these systemic issues, the paper provides actionable standards for dataset release and evaluation:

• Use proof_wanted instead of sorry in Lean to prevent RL agents from exploiting environment axioms.
• Turn off Lean's auto_implicit to force explicit declaration of all type parameters.
• Ban axioms and unsafe tactics from dataset files.
• Record and pin relevant Lean/mathlib versions with every dataset release.
• Use static checkers and semantic audit pipelines as routine validation, with human verification prioritized for flagged items.
• Capture all problem requirements in the formalization, not just the main claim.

These simple, inexpensive interventions are shown to drastically reduce exploitability and raise the reliability of new benchmarks.

Implications and Future Work

Failings in formal benchmarking do not just undermine metric fidelity; they undermine the field's ability to accurately compare systems and make progress in automated theorem proving. The findings of this audit generalize to any environment where natural language mathematical statements are automatically formalized, proof assistants undergo rapid development, and benchmarks are repurposed and forked frequently. While the current implementation is Lean-specific, the taxonomy and methodologies readily transfer to other formal systems.

Future directions include refining LLM-aided triage for semantic review, extending audits to larger or alternative datasets, integrating semantic-fidelity evaluation in end-to-end benchmarks, and systematically hardening reward models in RL setups against evaluation artifacts.

Conclusion

The paper establishes that formal proof-checking, while necessary, is insufficient for trustworthy benchmarking of neural theorem provers. Systematic defects in formalization, harnesses, and dataset maintenance are prevalent and have a significant impact on evaluation. The authors' taxonomy, detection tooling, and practical standards offer a blueprint for more reliable and actionable benchmarking in formal mathematics and automated reasoning, fostering greater trust in reported advances and highlighting the importance of ongoing dataset stewardship and audit.