MLFMF Dataset Overview

• MLFMF Dataset is a comprehensive collection of formalized mathematics data featuring both heterogeneous network and s-expression AST representations for rigorous ML benchmarking.
• It supports key tasks like premise selection and link prediction using graph neural networks and NLP techniques to enhance proof automation.
• The dataset integrates over 250,000 entries from diverse libraries such as Lean 4 and Agda, providing detailed statistical insights and structured benchmarks.

MLFMF is a collection of datasets designed for benchmarking machine learning methods in the formalization of mathematics, targeting recommendation systems for proof assistants. These datasets support rigorous evaluation of premise selection, reference prediction, and related learning tasks based on structured libraries of formalized mathematics from major proof assistant ecosystems. MLFMF leverages both syntactic and graph-theoretic representations—incorporating heterogeneous dependency graphs and full abstract syntax trees—to enable graph neural models, structure-aware embeddings, and conventional NLP approaches. It is currently the largest corpus of machine-learnable formalized mathematical knowledge, encompassing over $250{,}000$ entries extracted from Agda and Lean 4 libraries (Bauer et al., 2023).

1. Scope and Library Composition

MLFMF integrates four principal mathematical libraries reflecting diverse proof assistant paradigms, formalization styles, and mathematical subdomains:

• Lean 4 Mathlib: The primary, rapidly evolving Lean 4 mathematics library, noted for extensive tactic use, a broad subject spectrum, and an active premise-selection research community.
• Agda Standard Library: The core Agda library, characterized by tactic-free, “full-term” proofs and coverage of algebra, combinatorics, and order theory.
• Agda-unimath: A large Agda corpus formalizing univalent mathematics and Homotopy Type Theory, with emphasis on higher-dimensional algebraic structures.
• TypeTopology (Agda): A mid-sized Agda sublibrary focusing on formal topology, featuring technical constructions and intricate dependency chains.

Each library is parsed and represented in two modalities: (1) as a directed heterogeneous network encoding the modular structure and cross-references among entries, and (2) as a per-entry s-expression dump of its elaborated abstract syntax tree (AST).

2. Data Extraction and Representation

Heterogeneous Network

For each library, MLFMF constructs a directed graph $G=(V,E)$, where nodes $V$ include:

• Library node (one per corpus)
• Module/namespace nodes (file-based for Agda, declared for Lean)
• Entry nodes (such as functions, data types, records, axioms, etc.), each labeled with entry kind (e.g., :data, :function, :axiom)

Edges $E$ encode:

• CONTAINS: library $\rightarrow$ module, module $\rightarrow$ submodule
• DEFINES: module $\rightarrow$ entry
• REFERENCE_FROM_DECLARATION: entry $u$ references entry $v$ in its type signature
• REFERENCE_FROM_BODY: entry $u$ references entry $G=(V,E)$0 in its implementation or proof term

This network provides a granular map of the formal library's modular and referential architecture, enabling graph-based relational learning.

S-expression AST Dump

For each entry, the fully elaborated AST is exported as a Lisp-style s-expression, structured as $G=(V,E)$1 where:

• name: fully-qualified identifier
• decl: s-expression encoding the type, with implicitness and binding details
• body: s-expression for the proof term or definition

In Lean, shared subterms lead to a DAG structure; in Agda, trees are true syntactic trees. Each entry line is independent and fully self-contained, permitting direct batch parsing and analysis.

3. Statistical Overview

Dataset statistics across all four libraries are as follows:

Library	Entries	Nodes	Edges
Agda Stdlib	$G=(V,E)$2	$G=(V,E)$3	$G=(V,E)$4
Agda-unimath	$G=(V,E)$5	$G=(V,E)$6	$G=(V,E)$7
TypeTopology	$G=(V,E)$8	$G=(V,E)$9	$V$0
Lean Mathlib4	$V$1	$V$2	$V$3

Across all corpora, the total number of entries is $V$4. Mean per-entry (tree or DAG) node count is approximately 420, with standard deviation 200; average AST depth is $V$5 (std $V$6); and the vocabulary size for s-expression tokens is approximately 10,000.

4. File Formats and Organization

Each library’s data is distributed in a canonical folder structure:

$E$6

network.graphml files encode the node and edge types for $V$7. entries.sexpr are plain UTF-8 text files, one s-expression per line, directly parseable with standard Lisp parsers. Typical loading code involves common Python libraries:

$E$7

This structure supports direct integration with graph analytics, NLP pipelines, and structure-aware ML models.

5. Benchmarking Protocols and Baseline Results

Tasks

Two primary tasks are benchmarked:

• Link prediction: Predicting presence of missing REFERENCE edges in $V$8—paradigmatic for premise selection.
• Recommendation: Top-$V$9 prediction of missing dependencies in the context of an incomplete proof construction.

Train/test protocol: $E$0 of function nodes are “masked” as incomplete; within each, $E$1 of reference edges are removed (as positives), with negatives sampled from non-edge pairs.

Baseline Methods

1. Dummy: Ranks entries by static in-degree.
2. BoW Jaccard: Token set overlap for entry s-exprs, $E$2.
3. TF-IDF: Similarity by cosine and Manhattan distances in TF-IDF-weighted token space.
4. fastText: Pretrained embedding averages (Common Crawl), weighted by TF-IDF.
5. Analogy-based: fastText vector offsets.
6. node2vec + tree bagging: node2vec embeddings on $E$3 followed by a “tree-bagging” (ensemble of 100 trees) classifier.

Summary of Results

Method	Agda stdlib acc/minRank	Agda unimath acc/minRank	TypeTopology acc/minRank	Lean Mathlib4 acc/minRank
Dummy	0.51 / 218	0.53 / 2134	0.50 / 4556	0.51 / 26065
BoW	0.50 / 1608	0.50 / 1571	0.50 / 4496	0.50 / 15458
TF-IDF	0.51 / 144	0.52 / 112	0.51 / 552	0.51 / 443
fastText	0.51 / 132	0.52 / 394	0.50 / 1292	NA
Analogies	0.52 / 37	0.51 / 158	NA	NA
node2vec	0.96 / 4.4	0.96 / 3.2	0.98 / 5.8	0.95 / 195

Interpretation: node2vec-based approaches substantially outperform token-based or in-degree baselines (mean minimal rank $E$4 for Agda), whereas text-centric models yield marginal improvements over the dummy. Lean Mathlib4 remains challenging for minimal-rank recommendation due to tactic-heavy idioms and sparser explicit dependency tagging.

6. Use Cases and Extensibility

MLFMF supports a range of tasks in mathematical machine learning for formalization, including:

• Premise selection via link-prediction and recommendation.
• Graph neural network training over $E$5 with per-node and per-edge features from s-exprs.
• Joint text-structure models that integrate AST sequence data and dependency graph architecture.
• Entry classification (e.g., distinguishing functions from axioms).
• Transfer learning across proof assistant frameworks and formalization styles.

The extraction pipeline generalizes: for a new proof assistant, a “lib2sexp” extension or plugin can generate s-expr dumps; a “sexp2graph” parser constructs the network. The modular pipeline is agnostic to the underlying logic, allowing extension to Coq, Isabelle/HOL, HOL Light, Metamath, or similar systems with modest implementation effort.

7. Significance in Formalized Mathematics and Machine Learning

MLFMF provides the largest machine-learnable corpus of formalized mathematics to date, spanning over a quarter-million entries with semantic and syntactic structure. Its dual representations—heterogeneous networks and AST s-expressions—render it uniquely suitable for evaluating premise selection, reference prediction, and proof automation techniques that require deep structural and linguistic understanding. The dataset has become a standard for empirical validation in machine learning for mathematical formalization and is poised for further expansion into new proof assistant ecosystems (Bauer et al., 2023).