Putnam-like Benchmark Overview

• Putnam-like benchmarks are evaluation problems inspired by the Putnam Competition, designed to test multi-step reasoning and rigorous proof construction.
• They appear in various contexts such as natural language competition problems, formal theorem proving datasets, and operator theory benchmarks.
• Key insights include graded human-style assessments, low baseline success rates in formal settings, and resource-intensive verification protocols.

Searching arXiv for papers using “Putnam-like benchmark” and closely related benchmark formulations. Search query: Putnam-like benchmark mathematics competition theorem proving dataset In the cited literature, “Putnam-like benchmark” does not denote a single canonical artifact. It names several related but non-identical benchmark constructions: proof-style undergraduate mathematics datasets modeled on the William Lowell Putnam Mathematical Competition; formal theorem-proving corpora built from Putnam problems; small, fixed evaluation sets such as the 2025 Putnam exam in Rocq; and, in other technical literatures, benchmark formulations modeled on Fuglede–Putnam- or Kreisel–Putnam-type principles. This suggests that the phrase functions less as a unique proper name than as a family of reference problems used to test the boundary between routine derivation and genuinely nontrivial reasoning (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026, Mortad, 2020, d'Aragona, 22 Mar 2025, Birnbaum et al., 2011).

1. Terminological scope

The dominant contemporary use of the term is competition-inspired mathematical evaluation. One line of work treats LLMs as contestants on proof-style undergraduate problems, with natural-language solutions graded by human-style rubrics. Another line formalizes Putnam problems in proof assistants and measures pass@$n$ for neural and symbolic provers. A third line uses a single Putnam exam as a sharp, closed benchmark for an agentic prover in a fixed formal environment. Alongside these, operator theory and proof-theoretic semantics use “Putnam-like” to denote benchmark theorems or benchmark rules whose role is to mark the sharp frontier of a framework (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026, Dehimi et al., 2022, Johnson et al., 2021, Moslehian et al., 2011, d'Aragona, 22 Mar 2025).

Context	Benchmark object	Core criterion
Natural-language competition evaluation	96 original problems and 576 LLM solutions	Human-style 0–10 grading with rubrics
Formal theorem proving	PutnamBench formalizations in Lean 4, Isabelle, and Coq	Machine-checked proof success, typically pass@$n$
Agentic proof assistants	Putnam 2025 problems in Rocq	Compiles and verifies without Admitted or non-whitelisted axioms
Operator theory	Fuglede–Putnam-type benchmark statements	Whether intertwining/commutation passes to adjoints
Proof-theoretic semantics	Atomic Kreisel–Putnam rule	Logical validity via uniform, constructive reductions
Model counting	Davis–Putnam-style benchmark families	Runtime, recursive calls, memory, and hard-instance regimes

Not every usage refers to the Putnam competition itself. In the model-counting literature, the lineage is the Davis–Putnam procedure rather than the competition; in operator theory, the reference point is the Fuglede–Putnam theorem; and in proof-theoretic semantics, the benchmark principle is the atomic Kreisel–Putnam rule. The shared feature is the use of a sharply defined, difficult reference object against which proposed methods or theories are measured (Birnbaum et al., 2011, Mortad, 2020, d'Aragona, 22 Mar 2025).

2. Competition-style natural-language benchmarks

A recent competition-style benchmark is a curated dataset of 96 original, proof-style mathematics problems together with 576 LLM solutions, designed to imitate a university-level math competition in the style of the William Lowell Putnam Mathematical Competition and the International Mathematics Competition for University Students. The problems are organized into 8 “Putnam-style” sets, each with Part A and Part B, six problems per part, and an internal difficulty level from 1 to 6. Every problem was solved once by each of six models, producing the 576 solutions. Grading is 0–10 points per problem, using detailed rubrics that decompose an official solution into steps with assigned point values summing to 10; grading was performed by human experts, and the dataset also includes grades from gemini-2.5-pro_20250718 acting as an auto-grader (Bieganowski et al., 29 Sep 2025).

The benchmark is explicitly proof-oriented rather than final-answer oriented. Models were prompted only with the problem statement, without additional requirements on style or rigor, while the grading adhered to contest standards: logical steps had to be written, nonstandard theorems had to be stated correctly, and unjustified numerical approximations were penalized. The dataset is heavily skewed toward analysis: the category counts are Linear algebra 11, Abstract algebra 4, Analysis 57, Discrete mathematics 4, Probability 8, Number theory 6, Polynomials 6. The source states that there are no geometry or combinatorics problems because each problem is expected to provide a Lean formalization, and geometry or contest combinatorics is significantly harder to formalize (Bieganowski et al., 29 Sep 2025).

The reported grade distribution is highly bimodal. Across all 576 solutions, the relative frequencies of integer grades 0 through 10 are reported as 0.15, 0.04, 0.09, 0.07, 0.01, 0.02, 0.02, 0.01, 0.07, 0.05, 0.46 respectively, so 46% of all solutions received 10/10 and around 15% received 0. Average overall scores are 8.68 for gemini-2.5-pro-03-25, 7.59 for gemini-2.5-flash-04-17, 6.00 for sonnet-3.7, 5.95 for o4-mini-high, 5.59 for o3-mini-high, and 4.52 for r1. The same six models were also evaluated on Putnam 2024 problems, where the average scores were lower, and the source concludes that the Putnam-like dataset is slightly easier than the actual 2024 Putnam competition (Bieganowski et al., 29 Sep 2025).

The benchmark is therefore designed to expose proof-generation behavior that final-answer benchmarks hide. The source highlights model-specific patterns: gemini-2.5-pro-03-25 tends to produce either fully correct or fully incorrect solutions; gemini-2.5-flash-04-17 is often wordy and mixes correct arguments with dead ends; o3-mini-high frequently earns partial credit; r1 often produces sketches without rigorous details; and sonnet-3.7 is comparatively well edited. Common failure modes include unjustified assumptions, overreliance on numerical or approximate reasoning, incomplete proofs, and misinterpretation of the problem scope (Bieganowski et al., 29 Sep 2025).

3. Formal theorem-proving benchmarks

PutnamBench is a multi-language formal benchmark built from problems of the William Lowell Putnam Mathematical Competition. Its abstract describes 1692 hand-constructed formalizations of 640 theorems, while the detailed breakdown lists 640 Lean 4 theorems, 640 Isabelle theorems, and 417 Coq theorems. Each of the 640 problems has a Lean 4 and an Isabelle version, and 417 also have a Coq version. The benchmark is explicitly competition-style and undergraduate-level rather than textbook-exercise style, and it is intended to test substantial problem-solving ability across a broad range of topics taught in undergraduate mathematics courses (Tsoukalas et al., 2024).

A central design choice is the preservation of the “search” component of Putnam problems. For many problems that ask to find an object rather than merely verify one, PutnamBench uses a factored-solution design: the candidate solution is defined outside the theorem statement, and the theorem expresses an equivalence between satisfying the original conditions and belonging to the solution set. The source distinguishes Task 1, in which a system must discover a candidate and prove the theorem, from Task 2, in which the solution object is already supplied and only verification remains. This is intended to keep the benchmark “Putnam-hard” rather than reducing it to routine verification (Tsoukalas et al., 2024).

The baseline performance reported on PutnamBench is extremely low. On the 640 Lean problems, GPT-4 solves 1/640, COPRA solves 1/640, and ReProver with or without retrieval solves 0/640. On the 640 Isabelle problems, GPT-4 solves 1/640, DSP solves 4/640, and Sledgehammer solves 3/640. On the 417 Coq problems, GPT-4 solves 1/417, COPRA solves 1/417, and both Tactician and CoqHammer solve 0/417. Across all systems and languages, only 6 distinct problems are solved in total, and the paper presents PutnamBench as a difficult open challenge for research on neural theorem proving (Tsoukalas et al., 2024).

PutnamBench is positioned against several other datasets. Unlike miniF2F, which is mostly high-school level, and unlike ProofNet, which contains undergraduate textbook exercises, PutnamBench targets competition mathematics at the undergraduate level. It is also multilingual, aligned across Lean 4, Isabelle, and Coq, and explicitly contamination-resistant in the sense that the formal statements and proofs did not previously exist in public repositories. This makes it both an evaluation set for theorem provers and an autoformalization target, since the English problem statements are available alongside the formal theorem files (Tsoukalas et al., 2024).

4. Agentic proof-assistant benchmark instances

A more sharply delimited Putnam-like benchmark is the use of the 2025 Putnam Mathematical Competition itself as a closed evaluation set in Rocq. One study reports an experiment in which Claude Opus 4.6, equipped with a suite of Model Context Protocol tools for Rocq, autonomously proved 10 of 12 problems from the 2025 Putnam competition. The run took 17.7 hours of active compute, 51.6h wall-clock, used 141 subagents, and consumed approximately 1.9 billion tokens. The environment was an isolated virtual machine with no internet access, and success was defined operationally: a .v file had to compile under rocq_compile and pass rocq_verify without Admitted or non-whitelisted axioms (Baudart et al., 20 Mar 2026).

The tool design is summarized by a “compile-first, interactive-fallback” strategy. The Rocq-MCP server exposes eight tools, with rocq_compile, rocq_verify, and rocq_auto_solve in a compilation tier, and rocq_query, rocq_step, rocq_step_multi, rocq_toc, and rocq_notations in an interactive tier. The reported pattern is that the agent writes a full proof file, compiles it, iterates on errors, and only falls back to interactive stepping when compiler feedback is insufficient. The source states that every successful Putnam proof was ultimately verified via rocq_compile or direct coqc invocation, and that rocq_step was used almost exclusively on the three hardest problems (Baudart et al., 20 Mar 2026).

The 12 problems are tagged by domain and outcome. The study classifies A1, A2, A3, A4, A6, B1, B2, B3, B4, B5 as solved, and A5, B6 as unsolved. It also reports proof lengths and solution times, such as 305 lines, 51 min for A1, 531 lines, 4h58 for A4, 897 lines, 19h56 for A6, and 1455 lines, 46h01 for B5. A difficulty scaling table groups A1, A2, A3, B4 as “easy,” A4, B1, B2, B3 as “medium,” A6, B5 as “hard,” and the unsolved A5, B6 as consuming roughly 800M tokens. This gives the benchmark an explicit resource-sensitive dimension, not merely a solved/unsolved label (Baudart et al., 20 Mar 2026).

This Rocq study effectively defines a benchmark protocol: fixed problems, fixed tools, a strict verification criterion, and measurable budgets in time, tokens, tool calls, and subagent allocations. It therefore differs from broader datasets such as PutnamBench by treating a single Putnam exam as a high-resolution stress test for an agentic prover rather than as a large formal corpus (Baudart et al., 20 Mar 2026).

5. Benchmark formulations beyond competition mathematics

In operator theory, “Putnam-like benchmark” refers not to a contest dataset but to a reference theorem family modeled on the Fuglede–Putnam theorem. One note presents what it calls the main “all-unbounded” Fuglede–Putnam-type theorem: if $T$ is closed, $M$ and $N$ are normal, $D(N)\subset D(T)$, and $TN\subset MT$, then $TN^*\subset M^*T$. The paper argues that this is the “simplest possible (and most probably minimal in terms of hypotheses)” all-unbounded generalization, highlights the minimal domain hypothesis $D(N)\subset D(T)$, and supplies a counterexample showing that this condition cannot simply be omitted. The source explicitly says that the theorems can be viewed as benchmark formulations for unbounded Fuglede–Putnam results (Mortad, 2020).

A later paper systematizes this benchmark language. It proves positive results, such as Theorem 3.4 for an unbounded intertwiner $A$ under the spectral condition $n$0, and also provides counterexamples showing that naïve all-unbounded versions fail. It then presents these results as “Putnam-like benchmarks”: minimal conditions for unbounded Fuglede-type results, anti-commuting benchmarks, limits of generalization, and benchmarks for commutativity of bounded self-adjoint with unbounded symmetric or self-adjoint operators (Dehimi et al., 2022).

Related work extends the benchmark idea to other operator classes. For EP operators, the classical Fuglede–Putnam theorem fails in general, but a family of corrected Putnam-like statements is obtained, especially when the adjoint is replaced by the Moore–Penrose inverse. The strongest clean results are $n$1 for one EP operator and $n$2 for two EP operators. The paper explicitly describes these theorems as a coherent “Putnam-like benchmark” for commutativity phenomena beyond normal operators (Johnson et al., 2021).

The benchmark idea also appears in proof-theoretic semantics. One paper identifies the atomic Kreisel–Putnam rule

$n$3

as a minimal “Putnam-like” benchmark for monotonic, introduction-based proof-theoretic validity. Its main result is that $n$4 is logically valid relative to a fixed set of reductions that are constructive and uniform in Prawitz’s sense, while the rule is not derivable in intuitionistic logic. The paper uses this to refute completeness of intuitionistic logic over the corresponding proof-theoretic semantics (d'Aragona, 22 Mar 2025).

In model counting, the benchmark lineage is the Davis–Putnam procedure rather than the Putnam competition. The CDP algorithm counts exact models of CNF formulas and has average running time

$n$5

The paper and its structured summary describe how to build benchmark families by varying the number of variables $n$6, the number of clauses $n$7, the literal probabilities $n$8, and, for random 3-CNF, the clause-density ratio $n$9. The reported hard region for counting random 3-CNF occurs around $T$0, distinct from the classic SAT-decision hard region near $T$1 (Birnbaum et al., 2011).

Putnam-inspired problems can themselves be reworked into benchmark suites. A study of sequences inspired by 2017 Putnam A1 turns the closure rules $T$2 and $T$3 into an infinite directed graph $T$4, proves that

$T$5

and proposes layered benchmark tasks ranging from the characterization of $T$6 to generalized maximality theorems for $T$7. The paper explicitly lists candidate benchmark subtasks such as proving connectivity of $T$8, the chutes-and-ladders decomposition, first-appearance bounds, and the generalized maximality criterion (Alm et al., 2024).

6. Methodological significance and limitations

Across these literatures, Putnam-like benchmarks are defined by their insistence on structure rather than mere endpoint correctness. In the natural-language competition setting, the decisive move is from boxed answers to graded proofs. In formal theorem proving, the decisive move is from informal success claims to machine-checked proof objects. In the Rocq case study, the decisive move is further sharpened into a tool-mediated protocol in which a problem is solved only when the generated file compiles and verifies. This suggests that “Putnam-like” increasingly denotes evaluations that require multi-step reasoning, explicit justification, and a clear correctness oracle (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026).

The recent competition-style dataset also makes the limitations of such benchmarks explicit. It contains only 96 problems; it is slightly easier than the actual Putnam 2024 competition; and its subject distribution is heavily skewed toward analysis, with no geometry or combinatorics. PutnamBench, by contrast, has much broader topic coverage and formal rigor, but current systems solve only a handful of problems. The Rocq study achieves much higher absolute success on a 12-problem set, but at the cost of very large resource usage and with only a single year of Putnam problems (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026).

In the non-competition usages, the benchmark function is different but structurally analogous. In operator theory, a benchmark theorem fixes the strongest known conclusion under the weakest known hypotheses and then serves as the reference point for future extensions or counterexamples. In proof-theoretic semantics, a benchmark rule such as atomic Kreisel–Putnam marks the point at which a semantics validates more than intuitionistic derivability. In model counting, a benchmark family isolates the parameter regimes that stress a Davis–Putnam-style counting algorithm. These usages indicate that a Putnam-like benchmark need not be a dataset; it may also be a sharp formal statement that defines the frontier of a theory (Mortad, 2020, Dehimi et al., 2022, d'Aragona, 22 Mar 2025, Birnbaum et al., 2011).

Taken together, these developments suggest two dominant meanings. In artificial intelligence and theorem proving, a Putnam-like benchmark is a proof-centered evaluation regime modeled on the Putnam competition, whether in natural language or in formal proof assistants. In mathematical logic and operator theory, it is a benchmark principle whose validity or failure sharply discriminates between competing semantic or analytic frameworks. Both senses retain the same core function: to provide a compact but demanding reference object against which claims of reasoning power, structural adequacy, or theoretical sharpness can be judged.