ProofBench: Benchmark for Evaluating LLM Math Proofs

Updated 18 August 2026

ProofBench is an expert-annotated benchmark of 435 natural-language proofs for 145 competition problems, using a unified 0–7 scale to measure validity and partial progress rather than final-answer correctness alone.
ProofGrader combines o3, reference solutions, problem-specific marking schemes, flexible instructions, and median aggregation, achieving 0.926 MAE and 77.7% within-one-point accuracy against expert scores.
ProofBench supports practical research on proof evaluation and selection, with ProofGrader reaching an average selected-proof score of 4.14/7 and closing about 78% of the gap between a binary evaluator and a human oracle.

ProofBench is an expert-annotated benchmark for fine-grained evaluation of LLM–generated natural-language mathematical proofs. Introduced in “Reliable Fine-Grained Evaluation of Natural Language Math Proofs” (Ma et al., 14 Oct 2025), it contains 145 problems from six major mathematics competitions and 435 generated solutions, each assigned an expert score on a unified 0–7 scale. The benchmark is designed to distinguish final-answer correctness from proof validity, quantify partial mathematical progress, and support the development of automated proof evaluators such as ProofGrader. ProofBench is not a formal theorem-proving benchmark: its solutions are natural-language proofs, and correctness is assessed by expert annotation rather than Lean, Coq, Isabelle, or another proof-assistant kernel.

1. Purpose and position in mathematical evaluation

ProofBench addresses the limitations of answer-oriented mathematical benchmarks. A model can produce the correct numerical or symbolic answer while relying on an invalid inference, omitting an essential case, invoking an inappropriate theorem, or leaving a central claim unjustified. ProofBench therefore treats proof evaluation as a distinct task from answer evaluation.

The benchmark also addresses the coarseness of binary proof judgments. A binary label cannot distinguish a proof establishing two of seven essential steps from one establishing six, nor can it represent useful but incomplete progress. ProofBench instead assigns integer scores from 0 through 7, preserving a grading structure analogous to major mathematical competitions.

Its scope is complementary to formal theorem-proving datasets. “ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics” (Azerbayev et al., 2023) evaluates translation between natural-language and Lean 3 theorem statements, while leaving formal proof generation for future work. miniCodeProps evaluates Lean 4 proof generation for properties of functional programs (Lohn et al., 2024). FormalProofBench evaluates machine-checked Lean 4 proofs of advanced undergraduate and graduate mathematics (Ravi et al., 27 Mar 2026). ProofBench occupies a different position: it evaluates informal proof construction and fine-grained proof-quality assessment in mathematical natural language.

ProofBench also differs from research-level natural-language proof benchmarks such as IMProofBench, which contains 39 peer-reviewed research-level problems and uses expert grading of detailed proofs (Schmitt et al., 30 Sep 2025). ProofBench is centered on competition mathematics, whereas IMProofBench emphasizes research-like problem solving, tool use, and human evaluation of proof progress.

2. Dataset composition

ProofBench contains 145 problems drawn from six competitions over the years 2022–2025:

Competition	Problems
APMO	20
EGMO	23
IMO	24
Putnam	36
USA TST	18
USAMO	24
Total	145

Problem 4 from EGMO 2023 was intentionally excluded because its statement contains a figure. The source problems were collected from official competition materials, parsed and normalized, and paired with available human reference solutions.

The contests cover different mathematical settings. APMO, EGMO, IMO, USA TST, and USAMO represent high-school and Olympiad mathematics, while Putnam contributes undergraduate competition problems. The resulting collection spans algebra, combinatorics, geometry, number theory, and related areas represented by the six competitions.

Each problem has a main proof-writing task. Where possible, it also has two or three follow-up subquestions with unique short answers, such as numerical values, ranks, special cases, or other quantities. The main proof is evaluated by experts, while the follow-up answers support automatic grading.

A representative problem concerns the number $N_g$ of isomorphism classes of stable graphs of genus $g$ with exactly three edges. Its main task asks for a closed formula for all $g\geq 2$ , while subquestions request values including $N_3$ , $N_8$ , and $N_{10000}$ . The reference solution is a cubic quasi-polynomial of period 6, with values

$N_3=9,\qquad N_8=114,\qquad N_{10000}=111198615276.$

This example demonstrates the distinction between answer and proof performance: a model may compute the requested special values while failing to derive or justify the general quasi-polynomial.

3. Generation of model solutions

For each of the 145 problems, the authors generated one solution from each of three reasoning models:

OpenAI o3;
Gemini-2.5-Pro;
DeepSeek-R1-0528.

The resulting dataset contains

$145\times 3=435$

model-generated solutions.

The generation prompt required complete, self-contained proofs rather than outlines. It emphasized explaining important steps, avoiding unsupported advanced results, not skipping computations, and explicitly stating uncertainty instead of inventing claims.

The generated solutions were generally weak relative to competition standards. Their overall expert-score distribution had a mean of 2.41 and a median of 1.00 on the 0–7 scale. o3 was the strongest generator, with a mean score of 2.87, followed closely by Gemini-2.5-Pro. Fewer than 30% of problems received a score of 6 or 7 even from the strongest models.

Performance differed by competition. Putnam was the easiest source in the dataset, with a mean score of 3.11, whereas USA TST was the hardest, with a mean score of 1.28. These values characterize this benchmark collection and are not presented as universal rankings of mathematical capability.

4. Expert annotation and the seven-point scale

Marking schemes

Annotation has two stages. First, an LLM generates a problem-specific marking scheme from the problem and reference solution. The scheme specifies:

checkpoints that award points;
zero-credit conditions;
deductions and score caps.

Experts then evaluate the marking schemes independently on a 0–3 quality scale:

0: invalid;
3: high-fidelity.

Approximately 85% of the final generated marking schemes were judged reasonable and high quality. The marking scheme is advisory rather than a rigid checklist: valid alternative proofs receive credit when they establish equivalent mathematical content by a different route.

Human grading

Five experts with Putnam-level or national Mathematical Olympiad experience graded the generated proofs through a web interface. The annotation procedure included pilot calibration, discussion of disagreements, refinement of edge-case rules, periodic drift checks, double annotation of approximately 40% of the solutions, and adjudication of flagged disagreements.

The scale is normalized to 0–7. The paper does not provide a single universal verbal definition for every integer, because marking schemes are problem-specific, but its examples support the following interpretation:

0: irrelevant, nonsensical, or entirely incorrect;
1–2: fragments of relevant reasoning with major failures;
3–4: meaningful partial progress with substantial omissions or errors;
5–6: largely correct work with limited defects;
7: complete, valid, and sufficiently justified proof.

One example marking scheme assigns one point for reducing the problem and setting up the argument, two points for a pigeonhole step, two points for deducing two consecutive zero coefficients, and two points for proving a crucial lemma. It also caps scores when a reduction is used without justification or when a central lemma is flawed.

The rubric penalizes unjustified “without loss of generality” claims, unsupported theorem invocation, failure to use necessary assumptions, incomplete lemmas, missing logical links, examples substituted for universal proof, and omitted distinctness or nonzero conditions.

5. ProofGrader and automatic evaluation

ProofGrader is the best-performing evaluator configuration identified in the study. It combines:

o3 as the evaluation backbone;
the problem statement;
the generated proof;
the reference solution or solutions;
the original problem-specific marking scheme;
the “Norm” instruction, which permits valid alternative approaches;
five independent evaluation runs;
median aggregation of the five scores.

If the five evaluator calls return scores $\hat s^{(1)},\ldots,\hat s^{(5)}$ , ProofGrader returns

$\hat s=\operatorname{median}\left(\hat s^{(1)},\ldots,\hat s^{(5)}\right).$

ProofGrader is a prompting and inference configuration rather than a separately trained reward model.

Evaluator design space

The study varies four major dimensions.

Backbone model: o3, GPT-5-Thinking, Gemini-2.5-Pro, o4-mini, DeepSeek-R1-0528, and GPT-4o.

Input context:

reference solution and marking scheme;
marking scheme only;
reference solution only;
no problem-specific context.

Instructions: flexible “Norm,” literal “Strict,” and minimal “Basic” instructions.

Workflow: single-pass evaluation, repeated evaluation with aggregation, and staged workflows. The staged methods include binary correctness plus error analysis followed by fine-grained scoring, and evaluation followed by reflection and a final verdict.

The study also compares independent scoring with pairwise selection methods. Tournament selection requires $g$ 0 comparisons, while knockout selection uses $g$ 1 comparisons per bracket.

Metrics

The primary evaluator metric is mean absolute error:

$g$ 2

where $g$ 3 is the expert score and $g$ 4 is the evaluator prediction.

The study also reports root mean squared error, bias, within-one accuracy, and ties-adjusted Kendall correlation. Within-one accuracy is the proportion of predictions satisfying

$g$ 5

Main results

The strongest single-pass result is obtained by o3 with reference solutions and marking schemes:

Backbone	Context	RMSE	MAE	Within one point	Kendall correlation	Bias
o3	Reference + marking scheme	1.273	0.964	76.5%	0.502	−0.008
GPT-5	Reference + marking scheme	1.353	1.055	73.2%	0.532	0.295
Gemini-2.5-Pro	Reference + marking scheme	1.696	1.342	62.7%	0.529	0.626
o4-mini	Reference + marking scheme	1.816	1.367	67.6%	0.476	0.762
DeepSeek-R1	Reference + marking scheme	1.735	1.357	66.4%	0.429	0.732
GPT-4o	Reference + marking scheme	2.599	2.197	39.7%	0.479	1.824

For o3, removing context worsens MAE from 0.964 with reference solution plus marking scheme to 1.680 with no problem-specific context. The marking scheme alone performs better than the reference solution alone, with MAE values of 1.069 and 1.330, respectively.

With five independent o3 evaluations under reference-plus-marking-scheme conditions, median aggregation achieves MAE 0.926 and within-one accuracy of 77.7%. This is the ProofGrader configuration.

The study finds that the evaluator backbone has the largest effect, followed by context and then instruction style. Stronger evaluators benefit from flexible interpretation of the rubric, whereas weaker evaluators can benefit from stricter instructions. Ensembling reduces variance, but staged workflows are not universally beneficial: binary-plus-error analysis improves o4-mini while worsening o3.

6. Best-of- $g$ 6 selection and limitations

ProofGrader was evaluated as a proof-selection mechanism using 29 problems, 16 candidate proofs per problem, and 464 unique generated proofs. For each candidate subset, the evaluator selected the proof with the highest predicted score, and the selected proof was assessed using its human score.

At $g$ 7, the average selected-proof scores were:

Selector	Average selected score
Naive binary evaluator	2.48/7
ProofGrader	4.14/7
Human oracle	4.62/7

ProofGrader therefore closes approximately 78% of the gap between the binary evaluator and the human oracle:

$g$ 8

Fine-grained scoring is important because binary labels cannot distinguish a proof scored 5 from one scored 7. A binary evaluator consequently has limited ability to rank several acceptable candidates, whereas a 0–7 evaluator preserves information about relative proof quality.

ProofBench has several limitations. Its problems are primarily competition-level rather than research-level, and the benchmark does not cover formal proof languages. It does not explicitly score readability, elegance, clarity, originality, or pedagogical usefulness. Its labels depend on human judgments, problem-specific marking schemes, and the quality of generated reference materials. The study reports agreement operationally but does not provide a conventional inter-rater reliability coefficient such as Cohen’s $g$ 9 or Krippendorff’s $g\geq 2$ 0.

The benchmark is also vulnerable to contamination because its problems and solutions are publicly available. Public competition problems may have appeared in model-training data, and memorized reference solutions may produce an asymmetry between original proofs and generated distractors. The benchmark therefore measures proof evaluation under its particular data and annotation conditions rather than providing a contamination-free estimate of general mathematical verification.

ProofBench’s principal contribution is methodological: it establishes a fine-grained, expert-annotated resource for studying natural-language proof quality and demonstrates that evaluator performance depends strongly on reasoning capacity, reference context, marking schemes, prompt design, and aggregation. Its most direct successors and complements extend these principles in different directions. IMO-ProofBench provides 60 basic and advanced IMO-level proof problems with a ProofAutoGrader and a 1,000-example grading benchmark (Luong et al., 3 Nov 2025). Proof2Hybrid constructs proof-centric tasks automatically through multiple-judge formats and distractor generation (Peng et al., 4 Aug 2025). AdvancedMathBench separates proof generation from proof verification and annotates fatal and recoverable errors (Kong et al., 13 Jul 2026). Formal benchmarks such as MA-ProofBench and FormalProofBench replace model- or human-graded natural-language validity with Lean kernel acceptance (Pu et al., 11 Jun 2026, Ravi et al., 27 Mar 2026).

Together, these lines of work distinguish at least three capabilities: generating a mathematically valid argument, evaluating the validity and completeness of an informal proof, and producing a formally checked proof artifact. ProofBench is principally concerned with the second capability while using generated proofs as its evaluation objects.