Putnam-AXIOM: Axiomatic Evaluation in Contest Math

• Putnam-AXIOM is an axiomatic framework for evaluating Putnam-level problems by assessing both natural-language final answers and machine-checked proofs.
• It differentiates evaluation methods by comparing binary final-answer verification, detailed rubric grading, and formal proof assistant validation.
• Empirical results reveal challenges with binary accuracies around 41.94% and minimal formal theorem proving success, highlighting current performance gaps.

Searching arXiv for the specific term to ground the article in the relevant benchmark/formalization literature. Putnam-AXIOM denotes an axiomatic and rigor-oriented conception of Putnam-level mathematical evaluation that sits at the intersection of benchmark design, proof-based grading, and proof-assistant formalization. In the comparative literature, it appears both as a benchmark on natural-language Putnam problems with binary correctness on boxed final answers and as a broader methodological target: contest mathematics should be evaluated not only by final answers, but also by explicit proof obligations, theorem-bank constraints, and machine-checked formal proofs when possible (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026).

1. Conceptual scope

Putnam-AXIOM is best understood as a response to a specific evaluation gap in mathematical reasoning systems. Final-answer benchmarks can verify whether a model reached the correct output, but they do not necessarily establish whether the derivation was mathematically valid. The Putnam-like dataset summary makes this contrast explicit by stating that Putnam-AXIOM evaluates LLMs on natural-language Putnam problems with binary correctness on boxed final answers, whereas Putnam-like emphasizes proof-based grading with partial credit via rubrics (Bieganowski et al., 29 Sep 2025).

This distinction matters because Putnam-style problems are often proof-centric rather than answer-centric. Many require existence proofs, counterexamples, structural arguments, or delicate use of hypotheses. A system can therefore obtain the correct final expression while omitting a decisive lemma, importing an unjustified theorem, or silently assuming a false auxiliary condition. The AXIOM perspective treats those omissions as primary evaluation targets rather than secondary presentation flaws.

The literature around formal theorem proving extends this viewpoint further. PutnamBench formalizes Putnam problems across Lean 4, Isabelle/HOL, and Coq, while the Rocq experiment on Putnam 2025 frames AXIOM in explicitly formal-methods terms: theorem statements are encoded in a proof assistant, proofs are accepted only if they compile end-to-end, and axiom use is tracked rather than hidden behind informal prose (Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026).

2. Position within the benchmark landscape

The current ecosystem contains several closely related but non-identical resources. Their relationship clarifies what Putnam-AXIOM contributes.

Resource	Core unit	Reported emphasis
Putnam-AXIOM	Natural-language Putnam problems	Binary correctness on boxed final answers
Putnam-like benchmark	96 original problems and 576 model-generated solutions	Proof-based grading with partial credit via rubrics
PutnamBench	1697 hand-constructed formalizations of 640 Putnam theorems	Multilingual formal theorem proving in Lean 4, Isabelle, and Coq
Putnam 2025 in Rocq	Formal proofs of 12 contest problems	End-to-end proof-assistant verification with Rocq-MCP tools

The Putnam-like benchmark was released by Google DeepMind on September 23, 2025, built as curated modifications of past problems from the Putnam Competition and the International Mathematics Competition for University Students. It is organized into 8 Putnam-style papers, each with parts A and B and six problems of increasing difficulty, for a total of 96 original problems and 576 model-generated solutions. Difficulty is bucketed into levels 1–6, with levels 5–6 reported as “significantly harder” for LLMs than levels 1–4, and with a statistically significant non-monotonicity in which level 4 problems were easier than level 3 under a one-sided Welch t-test with $p = 0.02$ (Bieganowski et al., 29 Sep 2025).

Its subject distribution is also informative for AXIOM-style design: analysis including inequalities (57), linear algebra (11), probability (8), number theory (6), polynomials (6), abstract algebra (4), and discrete mathematics (4), with geometry and combinatorics omitted to facilitate forthcoming Lean formalizations. This omission already indicates one of the tensions in an axiomatic benchmark: formalizability can shape topic coverage (Bieganowski et al., 29 Sep 2025).

PutnamBench broadens the picture from evaluation-by-grading to evaluation-by-formalization. It contains 1697 hand-constructed formalizations of 640 Putnam theorems: 640 in Lean 4, 640 in Isabelle/HOL, and 417 in Coq. Its topical breadth is wider than the Putnam-like dataset, spanning Algebra (253), Analysis (226), Number Theory (107), Geometry (68), Linear Algebra (51), Abstract Algebra (28), Combinatorics (26), Probability (9), and Set Theory (8) (Tsoukalas et al., 2024).

3. Evaluation semantics: answers, proofs, and formal derivations

The central methodological question behind Putnam-AXIOM is what counts as a correct solution. The surrounding literature presents three distinct semantics.

The first is binary final-answer verification. In the comparison reported by the Putnam-like dataset summary, Putnam-AXIOM is characterized as evaluating natural-language Putnam problems with binary correctness on boxed final answers, with the best reported model achieving 41.94% correct (o1-preview). This regime is operationally simple and scalable, but it sharply compresses the distinction between a fully justified proof and a lucky or heuristic answer (Bieganowski et al., 29 Sep 2025).

The second is proof-oriented rubric grading. In Putnam-like, every problem is accompanied by a solution rubric decomposed into steps with point values summing to 10. Solutions are graded on a 0–10 scale, partial credit is awarded for correct sub-steps, and grading standards explicitly disallow numerical computation or unjustified approximation when a proof is required. Use of non-standard theorems or external references requires explicit justification, and sketches without details receive partial or low credit. No formal proof assistant verification is used for these natural-language solutions, although a companion Lean repository is maintained separately (Bieganowski et al., 29 Sep 2025).

The third is machine-checked formal verification. PutnamBench does not include proofs, only theorem statements, so it evaluates proof synthesis under formal semantics. The Rocq experiment pushes this further: a proof is accepted only if the Rocq file compiles, passes sandboxed verification, respects an axiom whitelist, and avoids statement redefinition or Admitted. This is the strongest interpretation of AXIOM among the resources cited here, because every accepted output is a formal derivation in the assistant’s logic rather than an informally persuasive proof sketch (Baudart et al., 20 Mar 2026).

A plausible implication is that Putnam-AXIOM is less a single benchmark format than a methodological gradient: final-answer exactness, rubric-level proof accounting, and full proof-assistant verification form increasingly strict realizations of the same aspiration.

4. Formal-methods realization in proof assistants

The most explicit formal realization of the AXIOM idea in the provided literature is the Rocq experiment on Putnam 2025. Claude Opus 4.6, equipped with Rocq-MCP tools, autonomously proved 10 of the 12 Putnam 2025 problems in Rocq, using an isolated VM with no internet access. The run deployed 141 subagents over 17.7 hours of active compute, 51.6 hours of wall-clock time, and approximately 1.9 billion tokens. All proofs are publicly available (Baudart et al., 20 Mar 2026).

Its methodology is described as “compile-first, interactive-fallback.” The agent writes a complete Rocq file, compiles it, reads structured errors, and iterates; interactive stepping is reserved for debugging specific subgoals. This protocol is significant in AXIOM terms because it prioritizes global soundness, uses rocq_verify to sandbox the theorem and enforce an axiom whitelist, and thereby blocks common failure modes such as silently changing the statement or relying on Admitted (Baudart et al., 20 Mar 2026).

The experiment solved A1, A2, A3, A4, A6, B1, B2, B3, B4, and B5, leaving A5 and B6 unsolved. Five proofs—A6, B3, B4, B5, and the corrected A3—were fully constructive. The A3 case is especially instructive: an initial formalization loophole modeled strategies as partial functions, allowing “never move” to satisfy the specification vacuously; the corrected encoding required the current player to exhibit a valid move. This episode shows that in an AXIOM framework, the precision of the formal statement is itself part of evaluation, not merely a preprocessing detail (Baudart et al., 20 Mar 2026).

PutnamBench supplies the broader multilingual substrate for this style of work. It is designed as a contamination-resistant benchmark without formal proofs included, and it exposes substantial differences among assistant foundations: Lean 4 uses dependent type theory with frequent classical reasoning in practice, Isabelle/HOL is classically grounded, and Coq is constructive by default but often relies on classical real-number libraries such as Coq.Reals.Raxioms and Coquelicot for analysis. The benchmark therefore makes assistant-level logical commitments visible rather than tacit (Tsoukalas et al., 2024).

5. Empirical results and characteristic failure modes

The empirical literature around Putnam-AXIOM and adjacent benchmarks shows both rapid progress and strong residual difficulty.

On the Putnam-like benchmark, average model scores across all 96 problems were 8.68 for gemini-2.5-pro-03-25, 7.59 for gemini-2.5-flash-04-17, 6.00 for sonnet-3.7, 5.95 for o4-mini-high, 5.59 for o3-mini-high, and 4.52 for r1. The grade distribution over 576 solutions placed 46% at grade 10 and 15% at grade 0, yielding an overall fully-correct proportion of approximately 0.46 if one defines accuracy as the fraction of 10-point solutions. Levels 1–2 had more than 75% of solutions graded at least 7, whereas levels 5–6 were significantly harder; only gemini-2.5-pro-03-25 achieved a near-perfect median at the top levels, with median 10.0 at level 5 and 9.5 at level 6 (Bieganowski et al., 29 Sep 2025).

The same paper reports that Putnam-like is “rather easier” than the actual Putnam 2024. On 2024 Putnam problems with double review, the average scores were 7.2 vs 8.7 for gemini-2.5-pro-03-25, 4.3 vs 7.6 for gemini-2.5-flash-04-17, 3.6 vs 5.6 for o3-mini-high, 3.7 vs 6.0 for o4-mini-high, 1.1 vs 4.5 for r1, and 3.8 vs 6.0 for sonnet-3.7. This implies that even strong performance on a proof-oriented surrogate can overestimate authentic contest performance (Bieganowski et al., 29 Sep 2025).

PutnamBench remains much harder for formal theorem provers. Across all assistants and baselines evaluated in the paper, only 6 unique problems were solved. In Lean 4, GPT-4 solved 1 of 640 problems and ReProver solved 0. In Isabelle/HOL, GPT-4 solved 1 of 640, Draft-Sketch-Prove solved 4, and Sledgehammer solved 3. In Coq, GPT-4 and COPRA each solved 1 of 417, while Tactician and CoqHammer solved 0. This indicates that the transition from natural-language grading to formal proof synthesis is still severe even when the underlying mathematics is the same (Tsoukalas et al., 2024).

The recurrent error modes are also stable across settings. Putnam-like identifies misapplied assumptions, unjustified approximations in proof problems, missing steps, algebraic slips, and invalid inference chains that mix correct fragments with dead ends. The sample problem on vanishing ball integrals in $\mathbb{R}^2$ is emblematic: a failed response asserted $f \equiv 0$ by implicitly assuming $f \ge 0$, whereas a correct approach constructs a counterexample using plane waves and the Bessel identity

$$\int_{B(0,r)} e^{i \xi \cdot y} \, dy = 2\pi \int_0^r \rho J_0(|\xi|\rho)\, d\rho = \frac{2\pi r J_1(|\xi|r)}{|\xi|}.$$

If $|\xi|r$ is a zero of $J_1$, then $f(x)=\cos(\xi\cdot x)$ satisfies the hypothesis without being identically zero (Bieganowski et al., 29 Sep 2025).

6. Gaps, design pressures, and future directions

The literature identifies several limitations that any mature Putnam-AXIOM program would need to address. The first is coverage bias. The Putnam-like dataset is heavily skewed toward analysis and excludes geometry and combinatorics, while PutnamBench includes those areas but still reflects assistant-specific library constraints, especially in Coq geometry and probability (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024).

The second is evaluation noise and protocol under-specification. Putnam-like uses human experts and an LLM grader for triangulation, but no formal proof assistant verification for natural-language submissions. It also does not specify tool-use prompting, self-consistency sampling, or external solver access. The Rocq experiment is much stricter, yet it demonstrates that performance can depend strongly on formalization quality, theorem retrieval, library coverage, and early detection of repeated proof-search bottlenecks (Bieganowski et al., 29 Sep 2025, Baudart et al., 20 Mar 2026).

The third is metric design. The Putnam-like paper explicitly recommends moving beyond coarse grades toward “full-proof rate, lemma success rate, formal check pass rate, and binary final-answer accuracy.” It also proposes step-level verification with formal witnesses, graded tiers from sketch to semiformal to fully formal proof, and curated theorem banks for contest-appropriate lemmas and identities. These recommendations give AXIOM a more precise methodological content: the benchmark should specify not only questions and answers, but also admissible theorem dependencies, proof granularity, and verification obligations (Bieganowski et al., 29 Sep 2025).

A fourth pressure is cross-assistant portability. The Rocq experiment suggests that canonical encodings of common contest motifs could be shared across assistants, with verified translation templates and explicit axiom dashboards distinguishing classical from constructive proofs. The A3 correction, which conceptually imported a more robust game formalization from AxiomProver’s Lean encoding into Rocq, is a concrete instance of this portability agenda (Baudart et al., 20 Mar 2026).

The available evidence therefore suggests that Putnam-AXIOM is converging toward a layered research program rather than a single static dataset. At one end lie natural-language Putnam problems with binary correctness on boxed final answers; at the other lies machine-checked contest mathematics with explicit axioms, theorem-bank governance, and reproducible proof artifacts. The intermediate systems—rubric-driven proof grading, multilingual formal theorem statements, and compile-first proof-assistant agents—supply the present operational bridge (Bieganowski et al., 29 Sep 2025, Tsoukalas et al., 2024, Baudart et al., 20 Mar 2026).