---
title: Putnam-like Benchmark Overview
url: https://www.emergentmind.com/topics/putnam-like-benchmark
type: topic
---

# Putnam-like Benchmark Overview

Searching arXiv for recent papers using “Putnam-like benchmark” and closely related benchmark formulations.
Search query: Putnam-like benchmark mathematics competition theorem proving dataset
In the cited literature, “Putnam-like benchmark” does not denote a single canonical artifact. It names several related but non-identical benchmark constructions: proof-style undergraduate mathematics datasets modeled on the William Lowell Putnam Mathematical Competition; formal theorem-proving corpora built from Putnam problems; small, fixed evaluation sets such as the 2025 Putnam exam in Rocq; and, in other technical literatures, benchmark formulations modeled on Fuglede–Putnam- or Kreisel–Putnam-type principles. This suggests that the phrase functions less as a unique proper name than as a family of reference problems used to test the boundary between routine derivation and genuinely nontrivial reasoning [2509.24827], [2407.11214], [2603.20405], [2003.00339], [2503.19930], [1106.0218].

## 1. Terminological scope

The dominant contemporary use of the term is competition-inspired mathematical evaluation. One line of work treats large language models as contestants on proof-style undergraduate problems, with natural-language solutions graded by human-style rubrics. Another line formalizes Putnam problems in proof assistants and measures pass@\(n\) for neural and symbolic provers. A third line uses a single Putnam exam as a sharp, closed benchmark for an agentic prover in a fixed formal environment. Alongside these, operator theory and proof-theoretic semantics use “Putnam-like” to denote benchmark theorems or benchmark rules whose role is to mark the sharp frontier of a framework [2509.24827], [2407.11214], [2603.20405], [2201.10604], [2101.06725], [1112.1302], [2503.19930].

| Context | Benchmark object | Core criterion |
|---|---|---|
| Natural-language competition evaluation | 96 original problems and 576 LLM solutions | Human-style 0–10 grading with rubrics |
| Formal theorem proving | PutnamBench formalizations in Lean 4, Isabelle, and Coq | Machine-checked proof success, typically pass@\(n\) |
| Agentic proof assistants | Putnam 2025 problems in Rocq | Compiles and verifies without `Admitted` or non-whitelisted axioms |
| Operator theory | Fuglede–Putnam-type benchmark statements | Whether intertwining/commutation passes to adjoints |
| Proof-theoretic semantics | Atomic Kreisel–Putnam rule | Logical validity via uniform, constructive reductions |
| Model counting | Davis–Putnam-style benchmark families | Runtime, recursive calls, memory, and hard-instance regimes |

Not every usage refers to the Putnam competition itself. In the model-counting literature, the lineage is the Davis–Putnam procedure rather than the competition; in operator theory, the reference point is the Fuglede–Putnam theorem; and in proof-theoretic semantics, the benchmark principle is the atomic Kreisel–Putnam rule. The shared feature is the use of a sharply defined, difficult reference object against which proposed methods or theories are measured [1106.0218], [2003.00339], [2503.19930].

## 2. Competition-style natural-language benchmarks

A recent competition-style benchmark is a curated dataset of **96 original, proof-style mathematics problems** together with **576 large language model solutions**, designed to imitate a university-level math competition in the style of the William Lowell Putnam Mathematical Competition and the International Mathematics Competition for University Students. The problems are organized into **8 “Putnam-style” sets**, each with Part A and Part B, six problems per part, and an internal difficulty level from 1 to 6. Every problem was solved once by each of **six models**, producing the 576 solutions. Grading is **0–10 points per problem**, using detailed rubrics that decompose an official solution into steps with assigned point values summing to 10; grading was performed by human experts, and the dataset also includes grades from **gemini-2.5-pro\_20250718** acting as an auto-grader [2509.24827].

The benchmark is explicitly proof-oriented rather than final-answer oriented. Models were prompted only with the problem statement, without additional requirements on style or rigor, while the grading adhered to contest standards: logical steps had to be written, nonstandard theorems had to be stated correctly, and unjustified numerical approximations were penalized. The dataset is heavily skewed toward analysis: the category counts are **Linear algebra 11, Abstract algebra 4, Analysis 57, Discrete mathematics 4, Probability 8, Number theory 6, Polynomials 6**. The source states that there are **no geometry or combinatorics problems** because each problem is expected to provide a Lean formalization, and geometry or contest combinatorics is significantly harder to formalize [2509.24827].

The reported grade distribution is highly bimodal. Across all 576 solutions, the relative frequencies of integer grades 0 through 10 are reported as **0.15, 0.04, 0.09, 0.07, 0.01, 0.02, 0.02, 0.01, 0.07, 0.05, 0.46** respectively, so **46%** of all solutions received **10/10** and around **15%** received **0**. Average overall scores are **8.68** for **gemini-2.5-pro-03-25**, **7.59** for **gemini-2.5-flash-04-17**, **6.00** for **sonnet-3.7**, **5.95** for **o4-mini-high**, **5.59** for **o3-mini-high**, and **4.52** for **r1**. The same six models were also evaluated on Putnam 2024 problems, where the average scores were lower, and the source concludes that the Putnam-like dataset is **slightly easier** than the actual 2024 Putnam competition [2509.24827].

The benchmark is therefore designed to expose proof-generation behavior that final-answer benchmarks hide. The source highlights model-specific patterns: **gemini-2.5-pro-03-25** tends to produce either fully correct or fully incorrect solutions; **gemini-2.5-flash-04-17** is often wordy and mixes correct arguments with dead ends; **o3-mini-high** frequently earns partial credit; **r1** often produces sketches without rigorous details; and **sonnet-3.7** is comparatively well edited. Common failure modes include unjustified assumptions, overreliance on numerical or approximate reasoning, incomplete proofs, and misinterpretation of the problem scope [2509.24827].

## 3. Formal theorem-proving benchmarks

PutnamBench is a multi-language formal benchmark built from problems of the William Lowell Putnam Mathematical Competition. Its abstract describes **1692 hand-constructed formalizations of 640 theorems**, while the detailed breakdown lists **640 Lean 4 theorems, 640 Isabelle theorems, and 417 Coq theorems**. Each of the 640 problems has a Lean 4 and an Isabelle version, and **417** also have a Coq version. The benchmark is explicitly competition-style and undergraduate-level rather than textbook-exercise style, and it is intended to test substantial problem-solving ability across a broad range of topics taught in undergraduate mathematics courses [2407.11214].

A central design choice is the preservation of the “search” component of Putnam problems. For many problems that ask to find an object rather than merely verify one, PutnamBench uses a **factored-solution** design: the candidate solution is defined outside the theorem statement, and the theorem expresses an equivalence between satisfying the original conditions and belonging to the solution set. The source distinguishes **Task 1**, in which a system must discover a candidate and prove the theorem, from **Task 2**, in which the solution object is already supplied and only verification remains. This is intended to keep the benchmark “Putnam-hard” rather than reducing it to routine verification [2407.11214].

The baseline performance reported on PutnamBench is extremely low. On the **640 Lean problems**, **GPT-4** solves **1/640**, **COPRA** solves **1/640**, and **ReProver** with or without retrieval solves **0/640**. On the **640 Isabelle problems**, **GPT-4** solves **1/640**, **DSP** solves **4/640**, and **Sledgehammer** solves **3/640**. On the **417 Coq problems**, **GPT-4** solves **1/417**, **COPRA** solves **1/417**, and both **Tactician** and **CoqHammer** solve **0/417**. Across all systems and languages, only **6 distinct problems** are solved in total, and the paper presents PutnamBench as a difficult open challenge for research on neural theorem proving [2407.11214].

PutnamBench is positioned against several other datasets. Unlike **miniF2F**, which is mostly high-school level, and unlike **ProofNet**, which contains undergraduate textbook exercises, PutnamBench targets **competition** mathematics at the **undergraduate** level. It is also multilingual, aligned across **Lean 4**, **Isabelle**, and **Coq**, and explicitly contamination-resistant in the sense that the formal statements and proofs did not previously exist in public repositories. This makes it both an evaluation set for theorem provers and an autoformalization target, since the English problem statements are available alongside the formal theorem files [2407.11214].

## 4. Agentic proof-assistant benchmark instances

A more sharply delimited Putnam-like benchmark is the use of the **2025 Putnam Mathematical Competition** itself as a closed evaluation set in Rocq. One study reports an experiment in which **Claude Opus 4.6**, equipped with a suite of **Model Context Protocol** tools for Rocq, autonomously proved **10 of 12 problems** from the 2025 Putnam competition. The run took **17.7 hours of active compute**, **51.6h wall-clock**, used **141 subagents**, and consumed approximately **1.9 billion tokens**. The environment was an isolated virtual machine with **no internet access**, and success was defined operationally: a `.v` file had to compile under `rocq_compile` and pass `rocq_verify` without `Admitted` or non-whitelisted axioms [2603.20405].

The tool design is summarized by a **“compile-first, interactive-fallback”** strategy. The Rocq-MCP server exposes eight tools, with `rocq_compile`, `rocq_verify`, and `rocq_auto_solve` in a compilation tier, and `rocq_query`, `rocq_step`, `rocq_step_multi`, `rocq_toc`, and `rocq_notations` in an interactive tier. The reported pattern is that the agent writes a full proof file, compiles it, iterates on errors, and only falls back to interactive stepping when compiler feedback is insufficient. The source states that every successful Putnam proof was ultimately verified via `rocq_compile` or direct `coqc` invocation, and that `rocq_step` was used almost exclusively on the three hardest problems [2603.20405].

The 12 problems are tagged by domain and outcome. The study classifies **A1, A2, A3, A4, A6, B1, B2, B3, B4, B5** as solved, and **A5, B6** as unsolved. It also reports proof lengths and solution times, such as **305 lines, 51 min** for A1, **531 lines, 4h58** for A4, **897 lines, 19h56** for A6, and **1455 lines, 46h01** for B5. A difficulty scaling table groups **A1, A2, A3, B4** as “easy,” **A4, B1, B2, B3** as “medium,” **A6, B5** as “hard,” and the unsolved **A5, B6** as consuming roughly **800M tokens**. This gives the benchmark an explicit resource-sensitive dimension, not merely a solved/unsolved label [2603.20405].

This Rocq study effectively defines a benchmark protocol: fixed problems, fixed tools, a strict verification criterion, and measurable budgets in time, tokens, tool calls, and subagent allocations. It therefore differs from broader datasets such as PutnamBench by treating a single Putnam exam as a high-resolution stress test for an agentic prover rather than as a large formal corpus [2603.20405].

## 5. Benchmark formulations beyond competition mathematics

In operator theory, “Putnam-like benchmark” refers not to a contest dataset but to a reference theorem family modeled on the Fuglede–Putnam theorem. One note presents what it calls the main **“all-unbounded” Fuglede–Putnam-type theorem**: if \(T\) is closed, \(M\) and \(N\) are normal, \(D(N)\subset D(T)\), and \(TN\subset MT\), then \(TN^*\subset M^*T\). The paper argues that this is the **“simplest possible (and most probably minimal in terms of hypotheses)”** all-unbounded generalization, highlights the minimal domain hypothesis \(D(N)\subset D(T)\), and supplies a counterexample showing that this condition cannot simply be omitted. The source explicitly says that the theorems can be viewed as **benchmark formulations** for unbounded Fuglede–Putnam results [2003.00339].

A later paper systematizes this benchmark language. It proves positive results, such as Theorem 3.4 for an unbounded intertwiner \(A\) under the spectral condition \(\sigma(p(A))\neq\mathbb C\), and also provides counterexamples showing that naïve all-unbounded versions fail. It then presents these results as **“Putnam-like benchmarks”**: minimal conditions for unbounded Fuglede-type results, anti-commuting benchmarks, limits of generalization, and benchmarks for commutativity of bounded self-adjoint with unbounded symmetric or self-adjoint operators [2201.10604].

Related work extends the benchmark idea to other operator classes. For **EP operators**, the classical Fuglede–Putnam theorem fails in general, but a family of corrected **Putnam-like** statements is obtained, especially when the adjoint is replaced by the **Moore–Penrose inverse**. The strongest clean results are \(AT=TA\Rightarrow AT^\dagger=T^\dagger A\) for one EP operator and \(AT=SA\Rightarrow AT^\dagger=S^\dagger A\) for two EP operators. The paper explicitly describes these theorems as a coherent **“Putnam-like benchmark”** for commutativity phenomena beyond normal operators [2101.06725].

The benchmark idea also appears in proof-theoretic semantics. One paper identifies the **atomic Kreisel–Putnam rule**
\[
\infer[\;aKP\;]{(p \rightarrow q) \vee (p \rightarrow r)}{p \rightarrow q \vee r}
\]
as a minimal “Putnam-like” benchmark for monotonic, introduction-based proof-theoretic validity. Its main result is that \(aKP\) is logically valid relative to a fixed set of reductions that are **constructive and uniform in Prawitz’s sense**, while the rule is not derivable in intuitionistic logic. The paper uses this to refute completeness of intuitionistic logic over the corresponding proof-theoretic semantics [2503.19930].

In model counting, the benchmark lineage is the **Davis–Putnam** procedure rather than the Putnam competition. The **CDP** algorithm counts exact models of CNF formulas and has average running time
\[
T(m,n)=O(m^d n), \qquad d=-\frac{1}{\log_2(1-p)}.
\]
The paper and its structured summary describe how to build benchmark families by varying the number of variables \(n\), the number of clauses \(m\), the literal probabilities \(p_1,p_2\), and, for random 3-CNF, the clause-density ratio \(m/n\). The reported hard region for counting random 3-CNF occurs around \(m/n\approx 1.2\), distinct from the classic SAT-decision hard region near \(m/n\approx 4.3\) [1106.0218].

Putnam-inspired problems can themselves be reworked into benchmark suites. A study of sequences inspired by **2017 Putnam A1** turns the closure rules \(n^2\mapsto n\) and \(n\mapsto (n+5)^2\) into an infinite directed graph \(\Gamma_5\), proves that
\[
S=\{n\in\mathbb N:n\ge 2,\ n\not\equiv 0 \pmod 5\},
\]
and proposes layered benchmark tasks ranging from the characterization of \(S\) to generalized maximality theorems for \(A1(d,e,r)\). The paper explicitly lists candidate benchmark subtasks such as proving connectivity of \(\Gamma_5\), the chutes-and-ladders decomposition, first-appearance bounds, and the generalized maximality criterion [2405.16068].

## 6. Methodological significance and limitations

Across these literatures, Putnam-like benchmarks are defined by their insistence on structure rather than mere endpoint correctness. In the natural-language competition setting, the decisive move is from boxed answers to graded proofs. In formal theorem proving, the decisive move is from informal success claims to machine-checked proof objects. In the Rocq case study, the decisive move is further sharpened into a tool-mediated protocol in which a problem is solved only when the generated file compiles and verifies. This suggests that “Putnam-like” increasingly denotes evaluations that require multi-step reasoning, explicit justification, and a clear correctness oracle [2509.24827], [2407.11214], [2603.20405].

The recent competition-style dataset also makes the limitations of such benchmarks explicit. It contains only **96 problems**; it is **slightly easier** than the actual Putnam 2024 competition; and its subject distribution is heavily skewed toward **analysis**, with **no geometry or combinatorics**. PutnamBench, by contrast, has much broader topic coverage and formal rigor, but current systems solve only a handful of problems. The Rocq study achieves much higher absolute success on a 12-problem set, but at the cost of very large resource usage and with only a single year of Putnam problems [2509.24827], [2407.11214], [2603.20405].

In the non-competition usages, the benchmark function is different but structurally analogous. In operator theory, a benchmark theorem fixes the strongest known conclusion under the weakest known hypotheses and then serves as the reference point for future extensions or counterexamples. In proof-theoretic semantics, a benchmark rule such as atomic Kreisel–Putnam marks the point at which a semantics validates more than intuitionistic derivability. In model counting, a benchmark family isolates the parameter regimes that stress a Davis–Putnam-style counting algorithm. These usages indicate that a Putnam-like benchmark need not be a dataset; it may also be a sharp formal statement that defines the frontier of a theory [2003.00339], [2201.10604], [2503.19930], [1106.0218].

Taken together, these developments suggest two dominant meanings. In artificial intelligence and theorem proving, a Putnam-like benchmark is a proof-centered evaluation regime modeled on the Putnam competition, whether in natural language or in formal proof assistants. In mathematical logic and operator theory, it is a benchmark principle whose validity or failure sharply discriminates between competing semantic or analytic frameworks. Both senses retain the same core function: to provide a compact but demanding reference object against which claims of reasoning power, structural adequacy, or theoretical sharpness can be judged.

Source: https://www.emergentmind.com/topics/putnam-like-benchmark