MSC-180 Benchmark for LLM-based ATP

• MSC-180 is a formally rigorous benchmark comprising 180 Lean 4 formalized problems across 60 MSC2020 subfields, ensuring both breadth and depth.
• It employs a systematic pipeline—combining OCR, automated formalization, and expert reconstruction—to generate semantically verified proposition-proof pairs.
• The evaluation framework utilizes multi-dimensional metrics (Pass@k, Domain@k, CV@k) to diagnose LLM-based ATPs' generalization abilities and domain biases.

MSC-180 is a domain-balanced, formally rigorous benchmark designed to evaluate and advance automated formal theorem proving (ATP) via LLMs. Covering the full breadth of the 2020 Mathematics Subject Classification (MSC2020), it comprises 180 formalized problems in Lean 4—three advanced, expert-selected theorems from each of 60 major mathematical subfields, ranging from logic and algebraic geometry to numerical analysis and information theory. Its construction and evaluation framework directly target key weaknesses observed in current LLM-based provers, quantifying limitations in systematic generalization, cross-domain reasoning, and domain bias (Li et al., 20 Dec 2025).

1. Dataset Scope and Construction

MSC-180 samples three formally stated verification problems per major MSC2020 branch (totaling 60 × 3 = 180), ensuring both disciplinary breadth and problem depth. The curation process stratifies problem difficulty between undergraduate-level (35 problems) and graduate-level (145 problems) mathematics. The selection draws from authoritative textbooks (post-2010), with undergraduate-level problems limited to content involving core analysis, algebra, or elementary number theory, and graduate-level problems encompassing advanced topics such as sheaf cohomology, Sobolev spaces, and stochastic calculus.

The dataset formalization proceeds through a structured pipeline:

• Text Sourcing and OCR: Post-2010 mathematical textbooks are digitized via optical character recognition. A pretrained LLM (DeepSeek) identifies candidate natural-language problem statements and informal proofs.
• Automated Formalization: Kimina-Autoformalizer-7B generates up to 20 Lean 4 code candidates per proposition. Syntactic filtering via the Lean 4 compiler (achieving an 81.7% pass rate) and semantic alignment filtering using DeepSeek-V2 (API with temperature=0.7, top-p=0.9) yield 6,071 semantically consistent proposition-proof pairs.
• Expert Reconstruction: From filtered candidates, domain experts manually select, rewrite, and cross-validate three problems per MSC field in Lean 4 through three verification rounds, explicitly enforcing logical equivalence, precise premises, and Lean 4 compilation.

The resulting Lean 4 library comprises 180 theorems, each independently compiling and representing its associated MSC domain.

2. Evaluation Protocol and Metrics

MSC-180 introduces a multi-faceted evaluation protocol to analyze prover performance across several axes. Its central accuracy metric, Pass@k, computes the probability that at least one proof attempt out of $n$ yields a correct result in $m$ draws without replacement:

$$\operatorname{pass}@m = 1 - \frac{\binom{n-c}{m}}{\binom{n}{m}}$$

where $n$ is total attempts, $c$ is successful attempts, and $m$ is the number of samples drawn ($n=m=k$ for end-to-end and lemma-based models; for tactic search, $k = N \times S \times T$).

To probe generalization and detect domain specialization, MSC-180 introduces two key cross-domain metrics:

• Domain@k: The fraction of MSC branches for which the prover solves at least one of the three problems under Pass@k, formally $|\{d: \text{success}_d \geq 1\}| / 60$.
• Coefficient of Variation@k (CV@k): Dispersion of per-domain Pass@k rates, computed as $CV@k = \sigma / \mu$, where $m$0 is the mean and $m$1 the standard deviation of Pass@k across domains.

A low CV@k reflects uniformly distributed performance; high values indicate strong domain bias.

3. Empirical Findings: Model Performance and Limitations

Three representative 7B-parameter theorem provers were evaluated on MSC-180 under Pass@32:

Prover	Pass@32	Domain@32	CV@32
DeepSeek-Prover-V2	0.189	41.7%	1.27
Kimina-Prover-Preview-Distill	0.167	38.3%	1.43
BFS-Prover 7B	0.044	14.6%	1.72

Key observations:

• Success rates are low: DeepSeek achieves 18.9%, Kimina 16.7%, BFS-Prover 4.4%.
• Domain coverage (Domain@32) is limited: Maximum coverage peaks at 41.7% (25/60 domains).
• High performance variance: CV@32 ranges from 1.27 (DeepSeek) to 1.72 (BFS), 4–6× above common "high variability" thresholds (≈0.2–0.3).
• Difficulty gap: DeepSeek solves 31.4% of undergraduate problems versus 15.9% of graduate problems; Kimina shows a similar drop (20.0% vs. 15.9%), and BFS is lowest (~8.6% vs. ~3.4%).

4. Analysis: Domain Bias, Difficulty Gap, and Reasoning Patterns

The benchmark highlights pronounced domain bias: models are far more successful in algorithmically structured areas (e.g., numerical analysis, computational algebra) and falter in abstract or less data-rich domains (e.g., commutative algebra, homotopy theory), as evidenced by outsized CV@k and modest Domain@k.

Undergraduate-level problems are 2–3× more tractable for current provers than graduate-level ones. Graduate-level theorems, requiring deeper proof search, handling of nested quantifiers, advanced library lookups, and complex type-theoretic constructs, remain especially challenging.

The steep drop in performance outside familiar domains and at higher difficulty levels suggests prevailing reliance on pattern matching and memorized proof fragments rather than transferable, domain-agnostic reasoning capabilities. Model successes tend to concentrate in training-data-rich or algorithmically homogeneous fields.

5. Multi-Dimensional Benchmarking Framework

MSC-180’s evaluation framework is explicitly multi-dimensional, combining:

• Accuracy (Pass@k): Absolute rate of successful solutions.
• Breadth (Domain@k): Prover’s coverage across diverse MSC domains.
• Uniformity (CV@k): Reflects whether performance is concentrated or broadly distributed.

This tri-axis design isolates "domain specialists" (high Pass@k, but low Domain@k and high CV@k) from "generalists" (robust, uniform performance), and supports systematic diagnosis of cross-domain deficiencies.

6. Guidance for ATP Research and System Development

MSC-180 provides actionable diagnostics:

• Zero-pass domains: Pinpoint where new tactics or additional domain-specific training data are needed.
• High-variance domains: Indicate inconsistent reasoning or inadequate model calibration.
• Graduate-level failures: Highlight the need for enhanced mathematical library retrieval, subgoal decomposition, and explicit premise management.

Recommended improvement strategies include:

• Diversifying pre-training datasets and retrieval mechanisms to encompass under-represented MSC domains.
• Integrating hierarchical proof planning and explicit lemma generation to improve long proof handling.
• Designing domain-agnostic reasoning modules, such as graph-based abstractions of proof states, to reduce overfitting and pattern-matching.
• Using CV@k as a regularizer in training, penalizing distributions with excessive performance concentration.

The dataset, Lean 4 statements, and evaluation tooling are available at https://github.com/Siri6504/MSC-180, enabling reproducible experimentation and systematic benchmarking.

7. Significance, Limitations, and Future Directions

MSC-180 exposes current LLM-based ATPs’ lack of systematic, cross-domain mathematical reasoning, quantitatively demonstrating the persistent gap between pattern-matching on familiar corpora and genuine mathematical intelligence. Its multi-dimensional framework provides a new standard for measuring the breadth, depth, and uniformity of formal reasoning in next-generation AI provers, serving both as a diagnostic platform and a development roadmap for the advancement of automated formal reasoning (Li et al., 20 Dec 2025).