uproof Dataset for Autoformalization

• uproof is a comprehensive dataset of undergraduate math proof problems that benchmark autoformalization and theorem-proving systems.
• It features standardized LaTeX statements paired with minimal Lean 4 formalizations, sourced from 14 classical textbooks across diverse math domains.
• The dataset enables robust evaluation through pass@k metrics and out-of-distribution testing, advancing research in formal verification and LLM theorem provers.

The uproof dataset is a large corpus of undergraduate-level mathematical proof problems designed to support the development and evaluation of autoformalization and advanced theorem-proving systems. Each problem is presented in standard LaTeX and paired, when possible, with a minimal Lean 4 formalization skeleton. uproof was introduced alongside the FormaRL framework for autoformalization, with the explicit aim of providing an out-of-distribution benchmark that reflects the diversity, breadth, and complexity of university-level mathematics as found in canonical textbook exercises (Huang et al., 26 Aug 2025).

1. Origins and Purpose

The central aim of uproof is to provide a benchmark suited for evaluating out-of-distribution autoformalization methods and catalyzing research in advanced mathematics theorem proving. Prior resources such as ProofNet and miniF2F were limited in scale (≤ 400 proofs) or overly focused on mathematical olympiad content. There existed no extensive resource for standard undergraduate mathematics—particularly in domains such as analysis, algebra, topology, and statistics—that captures the type of reasoning, abstraction, and notation encountered in classical university programs. uproof was curated from end-of-chapter exercises, lemmas, and theorems extracted from 14 classical undergraduate mathematics textbooks spanning a broad variety of sub-fields.

2. Data Collection and Curation

The assembly of uproof involved a multi-stage pipeline:

1. Textbook Selection and Digitization: Fourteen core undergraduate texts were selected, jointly covering foundational areas (see Section 3 for details). PDFs were converted to markdown and segmented into paragraphs.
2. Extraction and Cleaning: GPT-4o was used to extract and standardize complete mathematical statements, ensuring all embedded formulas conformed to standard LaTeX conventions.
3. Quality Control: The resulting statements were manually spot-checked for completeness, dropping partial/incomplete extractions and malformed logic.

All dataset instances correspond to exercises, lemmas, or theorems found in standard undergraduate course materials, with no additional difficulty annotation beyond the source location.

3. Dataset Structure and Statistics

uproof comprises 5,273 distinct problems distributed across major mathematical domains. The problems represent a diverse swath of undergraduate content that had previously been underrepresented in formalization benchmarks.

Domain	Textbook Authors	Problem Count
Analysis	Zorich I & II	873
Real & Complex Analysis	Stein & Shakarchi	504
Linear Algebra	Strang; Shafarevich & Remizov	312
Abstract/Commutative Algebra	Artin; Dummit & Foote; Atiyah & MacDonald	2,130
Topology	Armstrong; Hatcher	629
Functional Analysis	Bühler & Salamon	330
Probability & Statistics	Durrett; Casella & Berger	495

All problems are sourced from standard textbook exercises, and exhibit an average statement length of approximately 120 words (∼800 characters) with formulas. The validation set consists of 1,000 held-out problems suitable for prompt-tuning or early stopping. The remaining 4,273 constitute the primary out-of-distribution (OOD) evaluation split. No training split is supplied; uproof is principally intended for robust OOD benchmarking.

4. Data Format and Annotation Schema

Each uproof datum is encapsulated as a JSON object with the following structure:

{ "id": "Hatcher_2.2", "source": "Hatcher, Algebraic Topology §2.2", "category": "Topology", "latex": "Show that a subset %%%%0%%%% is connected iff there do not exist disjoint open %%%%1%%%% with %%%%2%%%%, %%%%3%%%%, and %%%%4%%%%.", "lean_skeleton": "theorem hatcher_2_2 {X : Type _} [TopologicalSpace X] (E : Set X) : Connected E ↔ ¬ ∃ U V : Set X, IsOpen U ∧ IsOpen V ∧ Disjoint U V ∧ E ⊆ U ∪ V ∧ U ∩ E ≠ ∅ ∧ V ∩ E ≠ ∅ := sorry" }

The "lean_skeleton" field provides a minimal Lean 4 declaration (with sorry as a placeholder) illustrating the core formal structure required for each proof. This enables direct integration into formal verification toolchains.

Annotated Examples

Topology: Connectedness Criterion

• LaTeX:

$$\begin{aligned} &\text{Show that in a topological space }(X,\tau)\text{, a subset }E\subset X\text{ is connected}\ &\text{if and only if there are no disjoint open (or closed) sets }G_1,G_2\subset X\ &\text{with }E\cap G_1\neq\emptyset,\;E\cap G_2\neq\emptyset,\;\text{and }E\subset G_1\cup G_2\,. \end{aligned}$$

• Lean 4 Skeleton:

import Mathlib.Topology.Connected theorem uplink_connected_iff_no_split {X : Type _} [TopologicalSpace X] (E : Set X) : Connected E ↔ ¬ ∃ (G1 G2 : Set X), IsOpen G1 ∧ IsOpen G2 ∧ Disjoint G1 G2 ∧ E ∩ G1 ≠ ∅ ∧ E ∩ G2 ≠ ∅ ∧ E ⊆ G1 ∪ G2 := by sorry

Analysis: Completeness of Function Spaces

• LaTeX:

“Prove that the metric space $R[a,b]$ of Riemann‐integrable functions on $[a,b]$ is not complete under

$d(f,g)=\int_a^b|f(x)-g(x)|\,dx\,.$

”

• Lean 4 Skeleton:

import Mathlib.Analysis.NormedSpace.Basic theorem not_complete_Riemann_integrable (a b : ℝ) : ¬ IsComplete (R[a,b] : MetricSpace) := by sorry

5. Access, Licensing, and Prerequisites

The uproof dataset is distributed under a CC-BY 4.0 license, requiring attribution for all extracted statements. It is publicly available with the FormaRL codebase at https://github.com/THUNLP-MT/FormaRL under the data/uproof/ directory. The recommended environment includes Lean 4 (≥ v4.15.0) and Mathlib4, together with community-standard REPL and CLI tools. Python 3.8+ is necessary for data handling and scripting.

6. Significance for Autoformalization Research

uproof was constructed to facilitate robust evaluation and development of autoformalization systems, especially in out-of-distribution and advanced-mathematics contexts. Its size—spanning nine domains and 14 textbooks—vastly exceeds previous advanced-math corpora such as ProofNet (371 problems). uproof problems typically require multi-step reasoning, manipulation of nested quantifiers, and sophisticated library features such as DifferentiableOn, Cardinality, or SylowSubgroup.

Autoformalization systems can be benchmarked using pass@k metrics (e.g., pass@1, pass@16), as well as fine-grained measures such as syntax-only (SC) versus syntax-plus-consistency (SC+CC) evaluation. For instance, the FormaRL method raised Qwen2.5-Coder-7B-Instruct’s pass@1 score on uproof from 2.4% to 9.6% after RL fine-tuning using only 859 unlabeled data points, and improved pass@16 from 24.4% to 33.6% (Huang et al., 26 Aug 2025). This suggests that the dataset is sufficiently challenging to reveal real improvements in generalization, and suitable for downstream adoption by modern LLM theorem-provers (e.g., DeepSeek-Prover, InternLM-StepProver).

7. Context and Impact

uproof establishes a new standard for breadth, diversity, and realism in undergraduate-level mathematics formalization benchmarks. Its inclusion of problem statements directly convertible to Lean 4 enables direct integration with formal verification environments. The absence of a dedicated training split and focus on OOD evaluation reflect the design goal of robust generalization assessment. By capturing the complexity and domain coverage of an undergraduate math curriculum, uproof directly supports the advancement and rigorous evaluation of next-generation autoformalizers and mathematical LLMs (Huang et al., 26 Aug 2025).