---
title: 'Putnam-AXIOM: Axiomatic Evaluation in Contest Math'
url: https://www.emergentmind.com/topics/putnam-axiom
type: topic
---

# Putnam-AXIOM: Axiomatic Evaluation in Contest Math

Searching arXiv for the specific term to ground the article in the relevant benchmark/formalization literature.
Putnam-AXIOM denotes an axiomatic and rigor-oriented conception of Putnam-level mathematical evaluation that sits at the intersection of benchmark design, proof-based grading, and proof-assistant formalization. In the comparative literature, it appears both as a benchmark on natural-language Putnam problems with binary correctness on boxed final answers and as a broader methodological target: contest mathematics should be evaluated not only by final answers, but also by explicit proof obligations, theorem-bank constraints, and machine-checked formal proofs when possible [2509.24827; 2407.11214; 2603.20405].

## 1. Conceptual scope

Putnam-AXIOM is best understood as a response to a specific evaluation gap in mathematical reasoning systems. Final-answer benchmarks can verify whether a model reached the correct output, but they do not necessarily establish whether the derivation was mathematically valid. The Putnam-like dataset summary makes this contrast explicit by stating that Putnam-AXIOM evaluates LLMs on natural-language Putnam problems with binary correctness on boxed final answers, whereas Putnam-like emphasizes proof-based grading with partial credit via rubrics [2509.24827].

This distinction matters because Putnam-style problems are often proof-centric rather than answer-centric. Many require existence proofs, counterexamples, structural arguments, or delicate use of hypotheses. A system can therefore obtain the correct final expression while omitting a decisive lemma, importing an unjustified theorem, or silently assuming a false auxiliary condition. The AXIOM perspective treats those omissions as primary evaluation targets rather than secondary presentation flaws.

The literature around formal theorem proving extends this viewpoint further. PutnamBench formalizes Putnam problems across Lean 4, Isabelle/HOL, and Coq, while the Rocq experiment on Putnam 2025 frames AXIOM in explicitly formal-methods terms: theorem statements are encoded in a proof assistant, proofs are accepted only if they compile end-to-end, and axiom use is tracked rather than hidden behind informal prose [2407.11214; 2603.20405].

## 2. Position within the benchmark landscape

The current ecosystem contains several closely related but non-identical resources. Their relationship clarifies what Putnam-AXIOM contributes.

| Resource | Core unit | Reported emphasis |
|---|---|---|
| Putnam-AXIOM | Natural-language Putnam problems | Binary correctness on boxed final answers |
| Putnam-like benchmark | 96 original problems and 576 model-generated solutions | Proof-based grading with partial credit via rubrics |
| PutnamBench | 1697 hand-constructed formalizations of 640 Putnam theorems | Multilingual formal theorem proving in Lean 4, Isabelle, and Coq |
| Putnam 2025 in Rocq | Formal proofs of 12 contest problems | End-to-end proof-assistant verification with Rocq-MCP tools |

The Putnam-like benchmark was released by Google DeepMind on September 23, 2025, built as curated modifications of past problems from the Putnam Competition and the International Mathematics Competition for University Students. It is organized into 8 Putnam-style papers, each with parts A and B and six problems of increasing difficulty, for a total of 96 original problems and 576 model-generated solutions. Difficulty is bucketed into levels 1–6, with levels 5–6 reported as “significantly harder” for LLMs than levels 1–4, and with a statistically significant non-monotonicity in which level 4 problems were easier than level 3 under a one-sided Welch t-test with $p = 0.02$ [2509.24827].

Its subject distribution is also informative for AXIOM-style design: analysis including inequalities (57), linear algebra (11), probability (8), number theory (6), polynomials (6), abstract algebra (4), and discrete mathematics (4), with geometry and combinatorics omitted to facilitate forthcoming Lean formalizations. This omission already indicates one of the tensions in an axiomatic benchmark: formalizability can shape topic coverage [2509.24827].

PutnamBench broadens the picture from evaluation-by-grading to evaluation-by-formalization. It contains 1697 hand-constructed formalizations of 640 Putnam theorems: 640 in Lean 4, 640 in Isabelle/HOL, and 417 in Coq. Its topical breadth is wider than the Putnam-like dataset, spanning Algebra (253), Analysis (226), Number Theory (107), Geometry (68), Linear Algebra (51), Abstract Algebra (28), Combinatorics (26), Probability (9), and Set Theory (8) [2407.11214].

## 3. Evaluation semantics: answers, proofs, and formal derivations

The central methodological question behind Putnam-AXIOM is what counts as a correct solution. The surrounding literature presents three distinct semantics.

The first is **binary final-answer verification**. In the comparison reported by the Putnam-like dataset summary, Putnam-AXIOM is characterized as evaluating natural-language Putnam problems with binary correctness on boxed final answers, with the best reported model achieving 41.94% correct (o1-preview). This regime is operationally simple and scalable, but it sharply compresses the distinction between a fully justified proof and a lucky or heuristic answer [2509.24827].

The second is **proof-oriented rubric grading**. In Putnam-like, every problem is accompanied by a solution rubric decomposed into steps with point values summing to 10. Solutions are graded on a 0–10 scale, partial credit is awarded for correct sub-steps, and grading standards explicitly disallow numerical computation or unjustified approximation when a proof is required. Use of non-standard theorems or external references requires explicit justification, and sketches without details receive partial or low credit. No formal proof assistant verification is used for these natural-language solutions, although a companion Lean repository is maintained separately [2509.24827].

The third is **machine-checked formal verification**. PutnamBench does not include proofs, only theorem statements, so it evaluates proof synthesis under formal semantics. The Rocq experiment pushes this further: a proof is accepted only if the Rocq file compiles, passes sandboxed verification, respects an axiom whitelist, and avoids statement redefinition or `Admitted`. This is the strongest interpretation of AXIOM among the resources cited here, because every accepted output is a formal derivation in the assistant’s logic rather than an informally persuasive proof sketch [2603.20405].

A plausible implication is that Putnam-AXIOM is less a single benchmark format than a methodological gradient: final-answer exactness, rubric-level proof accounting, and full proof-assistant verification form increasingly strict realizations of the same aspiration.

## 4. Formal-methods realization in proof assistants

The most explicit formal realization of the AXIOM idea in the provided literature is the Rocq experiment on Putnam 2025. Claude Opus 4.6, equipped with Rocq-MCP tools, autonomously proved 10 of the 12 Putnam 2025 problems in Rocq, using an isolated VM with no internet access. The run deployed 141 subagents over 17.7 hours of active compute, 51.6 hours of wall-clock time, and approximately 1.9 billion tokens. All proofs are publicly available [2603.20405].

Its methodology is described as **“compile-first, interactive-fallback.”** The agent writes a complete Rocq file, compiles it, reads structured errors, and iterates; interactive stepping is reserved for debugging specific subgoals. This protocol is significant in AXIOM terms because it prioritizes global soundness, uses `rocq_verify` to sandbox the theorem and enforce an axiom whitelist, and thereby blocks common failure modes such as silently changing the statement or relying on `Admitted` [2603.20405].

The experiment solved A1, A2, A3, A4, A6, B1, B2, B3, B4, and B5, leaving A5 and B6 unsolved. Five proofs—A6, B3, B4, B5, and the corrected A3—were fully constructive. The A3 case is especially instructive: an initial formalization loophole modeled strategies as partial functions, allowing “never move” to satisfy the specification vacuously; the corrected encoding required the current player to exhibit a valid move. This episode shows that in an AXIOM framework, the precision of the formal statement is itself part of evaluation, not merely a preprocessing detail [2603.20405].

PutnamBench supplies the broader multilingual substrate for this style of work. It is designed as a contamination-resistant benchmark without formal proofs included, and it exposes substantial differences among assistant foundations: Lean 4 uses dependent type theory with frequent classical reasoning in practice, Isabelle/HOL is classically grounded, and Coq is constructive by default but often relies on classical real-number libraries such as `Coq.Reals.Raxioms` and Coquelicot for analysis. The benchmark therefore makes assistant-level logical commitments visible rather than tacit [2407.11214].

## 5. Empirical results and characteristic failure modes

The empirical literature around Putnam-AXIOM and adjacent benchmarks shows both rapid progress and strong residual difficulty.

On the Putnam-like benchmark, average model scores across all 96 problems were 8.68 for `gemini-2.5-pro-03-25`, 7.59 for `gemini-2.5-flash-04-17`, 6.00 for `sonnet-3.7`, 5.95 for `o4-mini-high`, 5.59 for `o3-mini-high`, and 4.52 for `r1`. The grade distribution over 576 solutions placed 46% at grade 10 and 15% at grade 0, yielding an overall fully-correct proportion of approximately 0.46 if one defines accuracy as the fraction of 10-point solutions. Levels 1–2 had more than 75% of solutions graded at least 7, whereas levels 5–6 were significantly harder; only `gemini-2.5-pro-03-25` achieved a near-perfect median at the top levels, with median 10.0 at level 5 and 9.5 at level 6 [2509.24827].

The same paper reports that Putnam-like is “rather easier” than the actual Putnam 2024. On 2024 Putnam problems with double review, the average scores were 7.2 vs 8.7 for `gemini-2.5-pro-03-25`, 4.3 vs 7.6 for `gemini-2.5-flash-04-17`, 3.6 vs 5.6 for `o3-mini-high`, 3.7 vs 6.0 for `o4-mini-high`, 1.1 vs 4.5 for `r1`, and 3.8 vs 6.0 for `sonnet-3.7`. This implies that even strong performance on a proof-oriented surrogate can overestimate authentic contest performance [2509.24827].

PutnamBench remains much harder for formal theorem provers. Across all assistants and baselines evaluated in the paper, only 6 unique problems were solved. In Lean 4, GPT-4 solved 1 of 640 problems and ReProver solved 0. In Isabelle/HOL, GPT-4 solved 1 of 640, Draft-Sketch-Prove solved 4, and Sledgehammer solved 3. In Coq, GPT-4 and COPRA each solved 1 of 417, while Tactician and CoqHammer solved 0. This indicates that the transition from natural-language grading to formal proof synthesis is still severe even when the underlying mathematics is the same [2407.11214].

The recurrent error modes are also stable across settings. Putnam-like identifies misapplied assumptions, unjustified approximations in proof problems, missing steps, algebraic slips, and invalid inference chains that mix correct fragments with dead ends. The sample problem on vanishing ball integrals in $\mathbb{R}^2$ is emblematic: a failed response asserted $f \equiv 0$ by implicitly assuming $f \ge 0$, whereas a correct approach constructs a counterexample using plane waves and the Bessel identity
$$
\int_{B(0,r)} e^{i \xi \cdot y} \, dy = 2\pi \int_0^r \rho J_0(|\xi|\rho)\, d\rho = \frac{2\pi r J_1(|\xi|r)}{|\xi|}.
$$
If $|\xi|r$ is a zero of $J_1$, then $f(x)=\cos(\xi\cdot x)$ satisfies the hypothesis without being identically zero [2509.24827].

## 6. Gaps, design pressures, and future directions

The literature identifies several limitations that any mature Putnam-AXIOM program would need to address. The first is **coverage bias**. The Putnam-like dataset is heavily skewed toward analysis and excludes geometry and combinatorics, while PutnamBench includes those areas but still reflects assistant-specific library constraints, especially in Coq geometry and probability [2509.24827; 2407.11214].

The second is **evaluation noise and protocol under-specification**. Putnam-like uses human experts and an LLM grader for triangulation, but no formal proof assistant verification for natural-language submissions. It also does not specify tool-use prompting, self-consistency sampling, or external solver access. The Rocq experiment is much stricter, yet it demonstrates that performance can depend strongly on formalization quality, theorem retrieval, library coverage, and early detection of repeated proof-search bottlenecks [2509.24827; 2603.20405].

The third is **metric design**. The Putnam-like paper explicitly recommends moving beyond coarse grades toward “full-proof rate, lemma success rate, formal check pass rate, and binary final-answer accuracy.” It also proposes step-level verification with formal witnesses, graded tiers from sketch to semiformal to fully formal proof, and curated theorem banks for contest-appropriate lemmas and identities. These recommendations give AXIOM a more precise methodological content: the benchmark should specify not only questions and answers, but also admissible theorem dependencies, proof granularity, and verification obligations [2509.24827].

A fourth pressure is **cross-assistant portability**. The Rocq experiment suggests that canonical encodings of common contest motifs could be shared across assistants, with verified translation templates and explicit axiom dashboards distinguishing classical from constructive proofs. The A3 correction, which conceptually imported a more robust game formalization from AxiomProver’s Lean encoding into Rocq, is a concrete instance of this portability agenda [2603.20405].

The available evidence therefore suggests that Putnam-AXIOM is converging toward a layered research program rather than a single static dataset. At one end lie natural-language Putnam problems with binary correctness on boxed final answers; at the other lies machine-checked contest mathematics with explicit axioms, theorem-bank governance, and reproducible proof artifacts. The intermediate systems—rubric-driven proof grading, multilingual formal theorem statements, and compile-first proof-assistant agents—supply the present operational bridge [2509.24827; 2407.11214; 2603.20405].

Source: https://www.emergentmind.com/topics/putnam-axiom