---
title: 'LeanCat: A Lean Category Theory Benchmark'
url: https://www.emergentmind.com/topics/leancat
type: topic
---

# LeanCat: A Lean Category Theory Benchmark

LeanCat is a compact, open-source benchmark suite for category-theoretic formalization in Lean, specifically designed to stress-test and expose the abstraction and interface-layer reasoning capabilities of large language model (LLM)-based proof automation. It comprises 100 formalized theorems focused on 1-categories, and is positioned as a reproducible checkpoint to measure both human and AI progress toward research-level formalization in Lean. As a response to the limitations of existing theorem-proving benchmarks that focus on olympiad-style or computation-heavy problems, LeanCat emphasizes modularity, abstraction, and library-mediated reasoning in contemporary mathematical practice [2512.24796].

## 1. Motivation and Scope

Most formal theorem-proving benchmarks (such as MiniF2F, ProofNet, PutnamBench, FIMO, and FATE) are constructed around concrete or "olympiad-style" problems, often reducible to a single algebraic trick or computational step. This style is inadequate for capturing the kinds of abstraction and library-based reasoning found in modern research mathematics, particularly as practiced in category theory. Category theory, with its emphasis on reusable structural definitions (categories, functors, natural transformations, limits, adjunctions, monads, etc.), is a natural setting for benchmarking abstraction-aware planning, library retrieval, and long-horizon proof search, all of which are essential for advanced proof engineering.

## 2. Benchmark Design and Structure

LeanCat-1 (Part I: 1-Categories) contains 100 standalone Lean 4 theorems, each formulated as a single Lean file with one `theorem … := by sorry`, explicit universe annotations, and a brief LaTeX natural-language statement as a comment. Each theorem is classified into one of eight topic families and sorted into three difficulty tiers based on a hybrid LLM + expert rating pipeline.

**Topic Families**:

| Family                            | Number of Problems | Typical Content                          |
|------------------------------------|--------------------|------------------------------------------|
| Basic Category Properties          | 18                 | Monomorphisms, epimorphisms, idempotents, initial/terminal objects |
| Adjunctions                        | 11                 | Universal property constructions, comma-category criteria, examples |
| Reflective/Coreflective Subcategories | 4                | Classification in Set, Top$^{\mathbf{CH}}$, etc. |
| Concrete Categories                | 8                  | Forgetful functors, examples from topology and order theory |
| Limits and Colimits                | 32                 | Existence/preservation, diagrams, new definitions |
| Cocompletions                      | 5                  | Presheaf categories, universal property proofs |
| Abelian Categories                 | 12                 | Kernels, cokernels, exactness, images |
| Monads                             | 10                 | Kleisli/Eilenberg–Moore constructions, adjunctions of monads |

**Difficulty Tiers**:
- Easy (20 tasks): Trivial library invocations or one-step universal property unfolds.
- Medium (42 tasks): Requires composing two or more `mathlib` lemmas, mild diagrammatic reasoning.
- High (38 tasks): Demands introduction of auxiliary lemmas/definitions (e.g., Eilenberg–Moore object construction), chaining several abstraction layers, often not directly covered by `mathlib`.

Each task consists solely of imports, namespace, definitions, and the formal Lean statement; no informal hints or intermediate lemmas are provided. The proof must compile under Lean 4.19.0 and Mathlib 4.19.0.

## 3. Core Formal Content

LeanCat’s core formalism systematically encodes central notions of category theory, both in LaTeX and Lean, reflecting contemporary proof engineering practices.

**Definition of Category**:
- LaTeX: A category $\mathcal C$ consists of objects $\mathrm{Ob}(\mathcal C)$, morphisms $\mathrm{Hom}_{\mathcal C}(X,Y)$, identities $\mathrm{id}_X \in \mathrm{Hom}(X,X)$, and composition $\circ$, subject to associativity and unitality.
- Lean:
  ```lean
  universe u v
  structure Category :=
    (obj : Type u)
    (hom : obj → obj → Type v)
    (id   : Π X, hom X X)
    (comp : Π {X Y Z}, hom Y Z → hom X Y → hom X Z)
    (id_comp' : ∀ {X Y} (f : hom X Y), comp (id Y) f = f . obviously)
    (comp_id' : ∀ {X Y} (f : hom X Y), comp f (id X) = f . obviously)
    (assoc'   : ∀ {W X Y Z} (h : hom Y Z) (g : hom X Y) (f : hom W X),
                 comp h (comp g f) = comp (comp h g) f . obviously)
  ```

**Definition of Functor**:
  ```lean
  structure Functor (C : Category) (D : Category) :=
    (obj        : C.obj → D.obj)
    (map        : Π {X Y}, C.hom X Y → D.hom (obj X) (obj Y))
    (map_id'    : ∀ X, map (Category.id C X) = Category.id D (obj X) . obviously)
    (map_comp'  : ∀ {X Y Z} (f : C.hom X Y) (g : C.hom Y Z),
                  map (Category.comp g f) = Category.comp (map g) (map f) . obviously)
  ```
  
**Definition of Natural Transformation**:
  ```lean
  structure NatTrans {C D : Category} (F G : Functor C D) :=
    (app         : ∀ X, D.hom (F.obj X) (G.obj X))
    (naturality' : ∀ {X Y} (f : C.hom X Y),
      Category.comp (app Y) (F.map f)
      = Category.comp (G.map f) (app X) . obviously)
  ```

## 4. Curation and Difficulty Annotation Pipeline

The task set was curated as follows:
- **Problem Selection**: Three category theory experts selected candidate exercises from canonical sources (Mac Lane 1998, Riehl 2017, AHS 1990) and academic lecture notes, covering both abstract and concrete facets.
- **LLM-Assisted Formalization**: Multiple LLMs (GPT-5.1/5.2, Claude 4.5, Gemini 3 Pro, DeepSeek, Kimi) drafted Lean statements. Domain experts reviewed/corrected outputs, and created missing formalizations during a three-day Lean expert workshop.
- **Difficulty Annotation**: Each problem received a numeric difficulty score based on:
  - LLM “proof points” for successful attempts
  - LLM “statement points” for correct statement generations
  - Human expert ratings (1–10 scale) assessing proof length, abstraction depth, and lemma inventiveness required
  - Scores were aggregated with equal weight (50% LLM, 50% human), yielding: ≤6 “Easy,” ≥8.5 “High,” intermediate “Medium.”

## 5. Experimental Evaluation and Results

Evaluation employed a standardized protocol under Lean 4.19.0 + Mathlib 4.19.0, with 5-minute timeouts and a 50,000-token limit per proof. The context was limited strictly to the Lean statement and necessary imports; no informal guidance or auxiliary lemmas were provided.

**Summary of Model Results**:

| Model          | Easy (pass@1/pass@4) | Medium | High | Overall |
|----------------|----------------------|--------|------|---------|
| Claude Opus 4.5| 32.50% / 50.00%      | 4.17% / 4.76% | 0.00% / 0.00% | 8.25% / 12.00% |
| GPT-5.2        | 27.50% / 30.00%      | 0.00% / 0.00% | 0.00% / 0.00% | 5.50% / 7.00%  |
| Gemini 3 Pro, DeepSeek, Kimi K2 | 2–11% pass@1 (all tiers), 0% on High |

Tool-augmented approaches using the “LeanBridge” protocol (involving LeanExplore to retrieve Mathlib lemmas plus compiler feedback) consistently improved performance by 2–5% over single-shot LLM baselines but did not yield solutions on High-difficulty tasks.

**Error Analysis**:
- Library knowledge gaps—frequent failure to recall or correctly apply precise lemma names.
- Abstraction mismatch—models often generated element-wise proofs, neglecting universal property arguments.
- Breakdown in multi-step planning—partial subgoal progress, but inability to compose or chain abstraction layers to conclusion.

## 6. Significance and Prospective Developments

LeanCat establishes a reproducible checkpoint for assessing progress in formalization—both human-led and LLM-driven—on abstraction-intensive, library-grounded mathematical problems. As verified AI and human solutions accrue, these are intended to be merged back into Mathlib, bootstrapping the library and elevating the automation frontier. Planned extensions include Part II (addressing monoidal, enriched, and 2-categories), higher categorical structures, and cross-system ports (Coq, Isabelle) for ecosystem comparison.

This focus on structured organization, explicit difficulty stratification, and rigorous evaluation reveals persistent weaknesses in current LLM-based provers—especially for Medium/High-tier tasks—and concretely delineates future targets: improved library retrieval, explicit abstraction management, and robust long-horizon proof planning [2512.24796].

## 7. Contextual Impact Within Theorem Proving and AI

By emphasizing the interface-level compositionality and proof abstraction characteristic of modern category theory, LeanCat complements existing benchmarks while filling an explicit gap: it directly targets sustained, modular reasoning within formal mathematics libraries as encountered in current research. The benchmark’s architecture, content, and evaluation protocol serve as a template for future work on AI-driven formalization toolkits and systematic progress tracking in formal mathematical research environments.

Source: https://www.emergentmind.com/topics/leancat