Goedel-Pset-V1 Dataset Overview

• Goedel-Pset-V1 is an extensive dataset of 1.64 million Lean 4 formalized problem statements, generated via dual autoformalization and rigorous filtering.
• The dataset's multi-stage pipeline uses correctness and faithfulness tests to ensure reliable, machine-checked formal mathematics statements.
• It underpins scalable supervised fine-tuning and iterative proof generation workflows, demonstrating significant empirical gains on formal math benchmarks.

Goedel-Pset-v1 is an open-source corpus of 1.64 million machine-checked Lean 4 formal problem statements generated through large-scale autoformalization and rigorous filtering of natural language mathematics, designed to address the scarcity of formalized mathematical statements necessary for training automated theorem provers. As the foundation of the Goedel-Prover research program, it enables scalable supervised fine-tuning and iterative proof generation workflows, delivering competitive empirical gains across standard formal mathematics benchmarks (Lin et al., 11 Feb 2025).

1. Dataset Construction Pipeline

Goedel-Pset-v1 integrates multiple sources and a multi-stage autoformalization process to yield a broad-coverage collection of formalized problems suitable for machine learning on formal mathematics. Its creation proceeds as follows:

• Source Datasets and Scale.
  • Numina (Li et al. 2024): Open-source high-school and contest mathematics, 860,000 problems.
  • Art of Problem Solving (AOPS, private collection): 68,000 problems.
  • Existing Lean Workbook & Workbook Plus (Ying et al. 2024; Wu et al. 2024): 140,000 formally stated (often proofless) problems.
  • Total informal inputs aggregate to 1,068,000 problems, with all Lean Workbook entries inherently formalized.
• Autoformalization Models.
  • Two "statement formalizers" (A and B), both based on Qwen2.5-Coder-32B, are trained on distinct Formal ↔ Informal (F-I) datasets:
  • Formalizer A: Lean Workbook ↔ natural language pairs.
  • Formalizer B: Claude-sonnet-3.5 ↔ filtered Numina pairs.
• Formalization and Filtering Workflow.
  1. Each informal statement is fed to both formalizers, yielding 8 Lean 4 drafts per model (16 in total).
  2. Compiling Correctness (CC) Test: The proof stub is replaced with by sorry; lean --make must succeed.
  3. Faithfulness & Completeness (FC) Test: Four independent Qwen2.5-72B-Instruct "judges" assess whether the Lean 4 statement faithfully captures the source natural language; only drafts with FC ≥ 0.5 are retained.
  4. For each problem, if both formalizers produce valid drafts, one per model is sampled for inclusion; if one formalizer fails all outputs, the other's valid output is used.

After this workflow, the dataset comprises 1.64 million unique, Lean 4-compile-checked, and semantically faithful formal statements.

2. Content, Format, and Schema

Goedel-Pset-v1 captures a diverse mix of mathematical problems, both in scope and thematic coverage.

• Source Allocation:

  • Numina-derived: 860,000 (760,000 with dual A/B versions, 100,000 with a single version)
  • AOPS: 68,000
  • Lean Workbook: 140,000
• Topic and Difficulty Coverage: The original sources span
  • High-school algebra, geometry, and number theory (Numina, AOPS)
  • Competition-style Olympiad problems (Numina, AOPS)
  • Undergraduate-style textbook theorems (Lean Workbook)
• File Format: Distributed as JSON Lines (JSON-L), one problem per line. Each entry includes:
  • "id": unique identifier (e.g., Numina_123456_A)
  • "imports": necessary Lean 4 library imports (e.g., ["Mathlib.Data.Real.Basic", "Mathlib.Tactic.Nlinarith"])
  • "context": Lean 4 context commands (e.g., open Real)
  • "statement": Lean 4 theorem header (e.g., theorem not_divisible_by_seven (n : ℕ) : ¬7 ∣ (2^n + 1) := by)
  • "proof": always by sorry for Goedel-Pset-v1

Sample Schema (editor’s term):

Field	Example	Description
id	Numina_123456_A	Unique per problem version
imports	["Mathlib.Tactic.Nlinarith"]	Required Lean 4 libraries
context	open Int	Local context commands
statement	theorem even_function_implies_a_zero...	Lean 4 theorem header
proof	by sorry	Proof placeholder

This ensures reproducibility and seamless integration with Lean-based workflows.

3. Dataset Statistics and Properties

The scale and content of Goedel-Pset-v1 are quantitatively characterized as follows:

• Total size and source breakdown:

$$N_{total} = 1.64 \times 10^6,\quad N_{Numina} = 0.86 \times 10^6,\quad N_{AOPS} = 0.068 \times 10^6,\quad N_{LWB} = 0.14 \times 10^6$$

• Statement Complexity:
  • Median token length: ≈45 tokens, interquartile range [30, 70]
  • Median line count: ≈4 lines (theorem header plus optional comment)
• Topic Distribution (approximate):
  • Algebra & inequalities: 35%
  • Combinatorics & discrete mathematics: 20%
  • Number theory: 15%
  • Geometry & measurement: 10%
  • Calculus & analysis: 10%
  • Other (including probability and set theory): 10%

These properties ensure that the dataset presents a realistic distribution of problem types encountered in formal mathematics curricula from secondary to undergraduate levels.

4. Downstream Usage and Empirical Gains

Goedel-Pset-v1 underpins the training of Goedel-Prover and subsequent LLMs for automated theorem proving:

• Supervised Fine-Tuning (SFT):
  • The dataset’s 1.64M statements (with by sorry) are combined with incrementally generated proofs.
  • Base model: DeepSeek-Prover-V1.5-Base.
  • Fine-tuning proceeds in 8 expert-iteration rounds; by iteration 8, approximately 915,000 (statement, proof) pairs are collected.
• Expert Iteration Loop:

1. For each round $k$, up to 16 proof candidates per statement are generated by the iter–$k$ prover.
2. Proofs are verified for correctness via the Lean 4 compiler; one successful proof per statement is retained.
3. The supervised fine-tuning dataset is augmented with new (statement, proof) pairs, and retraining yields the next-round prover (iter–$k$+1).

• Benchmark Performance:
  • On miniF2F (488 test problems, whole-proof generation): Pass@32 rises from 50.0% (DeepSeek-RL) to 57.6% using Goedel-Prover-SFT.
  • On PutnamBench (644 problems): Goedel-Prover-SFT solves 7 (Pass@512), leading the leaderboard.
  • On Lean Workbook (140,000 problems): cumulative proofs found increase from 15,700 (prior work) to 29,700.

Further reinforcement learning steps (including DPO) push miniF2F success rates above 60% (Pass@32).

5. Design Rationale and Methodological Choices

• Selection of Lean 4 Platform: Lean 4’s modern, dependent-type theory, tactic-driven proof infrastructure, and rapid compilation (via lean --make)—combined with a robust community library (mathlib4)—enable large-scale autoformalization and proof search. The by sorry placeholder allows statements to be added and checked without explicit proofs, facilitating expert-iteration at scale.
• Dual Formalizers for Diversity: Incorporating formalizations from both A (direct style, e.g., $f(-x)=f(x)$) and B (predicate-naming, e.g., IsEven f) exposes theorem provers to a spectrum of lemma invocation patterns. Empirical ablations demonstrate that blending these styles yields +1–1.5% absolute gain on miniF2F and NuminaTest benchmarks compared to using either formalizer alone.
• Scope and Balance: Sourcing from both high-school/contest (Numina, AOPS) and undergraduate (Lean Workbook) domains permits coverage from “hard but elementary” reasoning to “abstract but lemma-rich” theorems, supporting the objective of a generalist, robust prover.

A plausible implication is that the structural diversity and rigorous multi-stage quality filters of Goedel-Pset-v1 amplify the capacity and generalization of formal proof-generating LLMs.

6. Open-source Distribution and Impact

Goedel-Pset-v1, alongside all associated models, code, and the Goedel-Pset-v1-solved corpus of collected proofs, is fully open-sourced to support reproducible research and further advances in formal mathematics automation. In particular, release of formal proofs for 29,700 Lean Workbook problems nearly doubles the 15,700 achieved by previous models, establishing a new benchmark for large-scale, open formal mathematics resources (Lin et al., 11 Feb 2025).

The dataset’s rigorous design and broad coverage position it as a central resource for advancing LLM-driven formal mathematics, enabling downstream work in benchmark evaluation, supervised and reinforcement learning, and curriculum-formalized automated theorem proving.