LeanCat: A Lean Category Theory Benchmark

• LeanCat is a benchmark suite for Lean formalization of category theory, focusing on abstraction and library-mediated reasoning.
• It comprises 100 standalone theorems organized into eight topic families and three difficulty tiers for comprehensive testing.
• It challenges both human and LLM-based proof automation to validate modular, long-horizon proofs in contemporary mathematical practice.

LeanCat is a compact, open-source benchmark suite for category-theoretic formalization in Lean, specifically designed to stress-test and expose the abstraction and interface-layer reasoning capabilities of LLM-based proof automation. It comprises 100 formalized theorems focused on 1-categories, and is positioned as a reproducible checkpoint to measure both human and AI progress toward research-level formalization in Lean. As a response to the limitations of existing theorem-proving benchmarks that focus on olympiad-style or computation-heavy problems, LeanCat emphasizes modularity, abstraction, and library-mediated reasoning in contemporary mathematical practice (Xu et al., 31 Dec 2025).

1. Motivation and Scope

Most formal theorem-proving benchmarks (such as MiniF2F, ProofNet, PutnamBench, FIMO, and FATE) are constructed around concrete or "olympiad-style" problems, often reducible to a single algebraic trick or computational step. This style is inadequate for capturing the kinds of abstraction and library-based reasoning found in modern research mathematics, particularly as practiced in category theory. Category theory, with its emphasis on reusable structural definitions (categories, functors, natural transformations, limits, adjunctions, monads, etc.), is a natural setting for benchmarking abstraction-aware planning, library retrieval, and long-horizon proof search, all of which are essential for advanced proof engineering.

2. Benchmark Design and Structure

LeanCat-1 (Part I: 1-Categories) contains 100 standalone Lean 4 theorems, each formulated as a single Lean file with one theorem … := by sorry, explicit universe annotations, and a brief LaTeX natural-language statement as a comment. Each theorem is classified into one of eight topic families and sorted into three difficulty tiers based on a hybrid LLM + expert rating pipeline.

Topic Families:

Family	Number of Problems	Typical Content
Basic Category Properties	18	Monomorphisms, epimorphisms, idempotents, initial/terminal objects
Adjunctions	11	Universal property constructions, comma-category criteria, examples
Reflective/Coreflective Subcategories	4	Classification in Set, Top$^{\mathbf{CH}}$, etc.
Concrete Categories	8	Forgetful functors, examples from topology and order theory
Limits and Colimits	32	Existence/preservation, diagrams, new definitions
Cocompletions	5	Presheaf categories, universal property proofs
Abelian Categories	12	Kernels, cokernels, exactness, images
Monads	10	Kleisli/Eilenberg–Moore constructions, adjunctions of monads

Difficulty Tiers:

• Easy (20 tasks): Trivial library invocations or one-step universal property unfolds.
• Medium (42 tasks): Requires composing two or more [mathlib](https://www.emergentmind.com/topics/mathlib) lemmas, mild diagrammatic reasoning.
• High (38 tasks): Demands introduction of auxiliary lemmas/definitions (e.g., Eilenberg–Moore object construction), chaining several abstraction layers, often not directly covered by [mathlib](https://www.emergentmind.com/topics/lean-mathematical-library-mathlib-ecd4e1c5-b152-44c7-9708-2ddb79890dde).

Each task consists solely of imports, namespace, definitions, and the formal Lean statement; no informal hints or intermediate lemmas are provided. The proof must compile under Lean 4.19.0 and Mathlib 4.19.0.

3. Core Formal Content

LeanCat’s core formalism systematically encodes central notions of category theory, both in LaTeX and Lean, reflecting contemporary proof engineering practices.

Definition of Category:

• LaTeX: A category $\mathcal C$ consists of objects $\mathrm{Ob}(\mathcal C)$, morphisms $\mathrm{Hom}_{\mathcal C}(X,Y)$, identities $\mathrm{id}_X \in \mathrm{Hom}(X,X)$, and composition $\circ$, subject to associativity and unitality.
• Lean:

  universe u v structure Category := (obj : Type u) (hom : obj → obj → Type v) (id : Π X, hom X X) (comp : Π {X Y Z}, hom Y Z → hom X Y → hom X Z) (id_comp' : ∀ {X Y} (f : hom X Y), comp (id Y) f = f . obviously) (comp_id' : ∀ {X Y} (f : hom X Y), comp f (id X) = f . obviously) (assoc' : ∀ {W X Y Z} (h : hom Y Z) (g : hom X Y) (f : hom W X), comp h (comp g f) = comp (comp h g) f . obviously)

Definition of Functor:

structure Functor (C : Category) (D : Category) := (obj : C.obj → D.obj) (map : Π {X Y}, C.hom X Y → D.hom (obj X) (obj Y)) (map_id' : ∀ X, map (Category.id C X) = Category.id D (obj X) . obviously) (map_comp' : ∀ {X Y Z} (f : C.hom X Y) (g : C.hom Y Z), map (Category.comp g f) = Category.comp (map g) (map f) . obviously)

Definition of Natural Transformation:

structure NatTrans {C D : Category} (F G : Functor C D) := (app : ∀ X, D.hom (F.obj X) (G.obj X)) (naturality' : ∀ {X Y} (f : C.hom X Y), Category.comp (app Y) (F.map f) = Category.comp (G.map f) (app X) . obviously)

4. Curation and Difficulty Annotation Pipeline

The task set was curated as follows:

• Problem Selection: Three category theory experts selected candidate exercises from canonical sources (Mac Lane 1998, Riehl 2017, AHS 1990) and academic lecture notes, covering both abstract and concrete facets.
• LLM-Assisted Formalization: Multiple LLMs (GPT-5.1/5.2, Claude 4.5, Gemini 3 Pro, DeepSeek, Kimi) drafted Lean statements. Domain experts reviewed/corrected outputs, and created missing formalizations during a three-day Lean expert workshop.
• Difficulty Annotation: Each problem received a numeric difficulty score based on:
  • LLM “proof points” for successful attempts
  • LLM “statement points” for correct statement generations
  • Human expert ratings (1–10 scale) assessing proof length, abstraction depth, and lemma inventiveness required
  • Scores were aggregated with equal weight (50% LLM, 50% human), yielding: ≤6 “Easy,” ≥8.5 “High,” intermediate “Medium.”

5. Experimental Evaluation and Results

Evaluation employed a standardized protocol under Lean 4.19.0 + Mathlib 4.19.0, with 5-minute timeouts and a 50,000-token limit per proof. The context was limited strictly to the Lean statement and necessary imports; no informal guidance or auxiliary lemmas were provided.

Summary of Model Results:

Model	Easy (pass@1/pass@4)	Medium	High	Overall
Claude Opus 4.5	32.50% / 50.00%	4.17% / 4.76%	0.00% / 0.00%	8.25% / 12.00%
GPT-5.2	27.50% / 30.00%	0.00% / 0.00%	0.00% / 0.00%	5.50% / 7.00%
Gemini 3 Pro, DeepSeek, Kimi K2	2–11% pass@1 (all tiers), 0% on High

Tool-augmented approaches using the “LeanBridge” protocol (involving LeanExplore to retrieve Mathlib lemmas plus compiler feedback) consistently improved performance by 2–5% over single-shot LLM baselines but did not yield solutions on High-difficulty tasks.

Error Analysis:

• Library knowledge gaps—frequent failure to recall or correctly apply precise lemma names.
• Abstraction mismatch—models often generated element-wise proofs, neglecting universal property arguments.
• Breakdown in multi-step planning—partial subgoal progress, but inability to compose or chain abstraction layers to conclusion.

6. Significance and Prospective Developments

LeanCat establishes a reproducible checkpoint for assessing progress in formalization—both human-led and LLM-driven—on abstraction-intensive, library-grounded mathematical problems. As verified AI and human solutions accrue, these are intended to be merged back into Mathlib, bootstrapping the library and elevating the automation frontier. Planned extensions include Part II (addressing monoidal, enriched, and 2-categories), higher categorical structures, and cross-system ports (Coq, Isabelle) for ecosystem comparison.

This focus on structured organization, explicit difficulty stratification, and rigorous evaluation reveals persistent weaknesses in current LLM-based provers—especially for Medium/High-tier tasks—and concretely delineates future targets: improved library retrieval, explicit abstraction management, and robust long-horizon proof planning (Xu et al., 31 Dec 2025).

7. Contextual Impact Within Theorem Proving and AI

By emphasizing the interface-level compositionality and proof abstraction characteristic of modern category theory, LeanCat complements existing benchmarks while filling an explicit gap: it directly targets sustained, modular reasoning within formal mathematics libraries as encountered in current research. The benchmark’s architecture, content, and evaluation protocol serve as a template for future work on AI-driven formalization toolkits and systematic progress tracking in formal mathematical research environments.