ProofRank: LLM Proof Quality Benchmark

• ProofRank is a benchmark that evaluates LLM mathematical proofs using proxies like conciseness, computational ease, cognitive simplicity, diversity, and adaptivity.
• It employs pairwise comparisons on correct proofs to ensure that evaluations reflect clarity, compactness, and methodological variety in competition-style problems.
• The framework bridges correctness and qualitative proof attributes, offering nuanced insights into LLM proof generation beyond simple answer verification.

ProofRank is a benchmark and methodology for evaluating how good LLMs’ mathematical proofs are, beyond just whether they are correct (Petrov et al., 11 May 2026). It evaluates several scalable proxies of proof quality—conciseness, computational ease, cognitive simplicity, diversity, and adaptivity—on difficult, high-school competition “final-answer” problems treated as proof-writing tasks, and it compares models only on proofs that pass a correctness prerequisite (Petrov et al., 11 May 2026).

1. Definition and scope

ProofRank is motivated by the claim that correctness alone is not sufficient for mathematical proof quality. In the benchmark formulation, mathematical proofs should also be clear, concise, insightful, and transferable to other problems, even though proof quality remains subjective and context-dependent (Petrov et al., 11 May 2026). Rather than collapsing everything into a single absolute notion of elegance, ProofRank defines several concrete, broadly valued proxies and measures them separately.

A central design choice is to separate proof-level metrics from problem-level metrics. Proof-level metrics compare two individual proofs of the same problem: conciseness, computational ease, and cognitive simplicity. Problem-level metrics assess a model’s behavior across multiple attempts on the same problem: diversity and adaptivity (Petrov et al., 11 May 2026). This decomposition avoids forcing heterogeneous proof qualities into one undifferentiated score.

Another central choice is to use pairwise comparison rather than absolute scoring for most quality dimensions. What counts as concise or computationally light depends strongly on the problem, so ProofRank asks which of two correct proofs is preferable on a given dimension, then aggregates those preferences into model-level ratings (Petrov et al., 11 May 2026). This suggests a ranking framework in which proof quality is treated as relational and context-sensitive rather than globally scalar.

Correctness is treated as a prerequisite. ProofRank only compares quality metrics between correct proofs, because otherwise weaker models can appear better by producing short, high-level, but wrong proofs (Petrov et al., 11 May 2026). The benchmark therefore distinguishes sharply between being correct and being useful.

2. Benchmark construction

ProofRank uses 382 final-answer problems drawn from MathArena and IMO-AnswerBench (Petrov et al., 11 May 2026). The MathArena portion includes AIME 2025 and 2026, HMMT February 2025 and 2026, Apex 2025, and Apex Shortlist 2025. The IMO-AnswerBench portion contributes 241 problems, originally proof problems rephrased to final-answer format (Petrov et al., 11 May 2026). Topics are classified into Algebra, Geometry, Combinatorics, Number Theory, and Calculus.

Human-written solutions are collected from official competition materials and public Art of Problem Solving posts (Petrov et al., 11 May 2026). These solutions serve as references for technique clustering and for constructing adaptivity tasks. For diversity, a problem is retained only if human solutions can be clustered into at least 3 distinct technique groups, yielding 119 problems for that metric (Petrov et al., 11 May 2026). For adaptivity, problem–technique pairs are formed from human technique clusters, then filtered to remove techniques that GPT-OSS-120B already uses naturally, resulting in 445 technique instances across 237 problems (Petrov et al., 11 May 2026).

ProofRank also constructs a correctness filter before any quality comparison is performed. First, it checks final-answer correctness using deterministic numeric-equivalence judgment. Second, it checks completeness with a separate prompt asking whether the proof is self-contained, whether every step follows from prior steps or standard facts, and whether no major computations or pattern generalizations are skipped (Petrov et al., 11 May 2026). A solution is treated as correct only if it passes both checks.

The correctness-and-completeness filter is validated against GPT-5.4 judging 200 solutions, with 93.5% agreement after manual adjudication (Petrov et al., 11 May 2026). Most residual errors come from slight over-permissiveness on completeness, which is consequential because ProofRank’s downstream rankings depend on filtering out incomplete but superficially attractive proofs.

3. Metric suite and ranking formalism

ProofRank evaluates five proxies of proof quality (Petrov et al., 11 May 2026).

Proxy	Object evaluated	Operationalization
Conciseness	Single proof	Compressibility after LLM shortening
Computational ease	Pair of proofs	LLM judge compares computation burden
Cognitive simplicity	Pair of proofs	LLM judge compares conceptual heaviness
Diversity	Set of proofs for one problem	Entropy over technique clusters
Adaptivity	Problem–technique pair	Success under a requested method

Conciseness is measured by compressibility rather than raw length. A strong LLM rewrites a proof $p$ to be as concise as possible while preserving correctness and completeness, producing $\tilde p$. The benchmark then computes

$C(p) = \frac{|p|}{|\tilde p|}.$

If two proofs differ in $C(p)$ by less than $10\%$ relative to the larger value, they are treated as equally concise; otherwise the proof with smaller $C(p)$ is preferred (Petrov et al., 11 May 2026). This definition is designed to discount harmless variation in exposition length while penalizing redundancy.

Computational ease and cognitive simplicity are implemented as pairwise LLM-as-judge comparisons. For computational ease, the judge is instructed to focus on low-level computation burden, such as symbol tracking, routine algebra, coordinate bash, or lengthy case analysis. For cognitive simplicity, the judge focuses on the conceptual accessibility of the method, such as whether the proof relies on straightforward contest tools or on layered, specialized techniques (Petrov et al., 11 May 2026).

Diversity is defined at the level of a model’s distribution over solution-method clusters for a fixed problem. After generating $k=4$ solutions per model, GPT-OSS-120B produces short “method fingerprints” and clusters all summaries across models into technique groups. For model $m_i$, if $P(C_j \mid m_i)$ is the empirical frequency of cluster $C_j$, its diversity score is the entropy

$\tilde p$0

Higher entropy means that the model spreads its correct solutions across more distinct techniques (Petrov et al., 11 May 2026).

Adaptivity evaluates whether a model can solve a problem while following a specified proof technique extracted from human solution clusters. For a model $\tilde p$1, the raw adaptivity score is the fraction of problem–technique pairs for which its solution is both correct and judged to use the specified technique: $\tilde p$2 The benchmark then also compares adaptivity outcomes pairwise across models (Petrov et al., 11 May 2026).

To aggregate pairwise preferences, ProofRank fits a Bradley–Terry model with a position-bias term: $\tilde p$3 Here $\tilde p$4 and $\tilde p$5 are latent model strengths on a given metric, and $\tilde p$6 captures first-position bias in LLM judging. The resulting strengths are mapped to an Elo-style scale

$\tilde p$7

This makes different models comparable even when they solve different subsets of problems (Petrov et al., 11 May 2026).

4. Empirical profile of models

ProofRank evaluates 10 frontier LLMs: GPT-5.4, GPT-OSS-120B, Gemini-3.1-Pro, Gemini-3-Flash, Qwen3.5-397B-A17B, GLM-5, Kimi-K2.5-Think, Grok-4.1-Fast, DeepSeek-V3.2, and Step-3.5-Flash (Petrov et al., 11 May 2026). On the correctness prerequisite, GPT-5.4 is the clear leader with 86.4% accuracy; Gemini-3.1-Pro is second with 60.7%, and several other models fall between roughly 29% and 54% (Petrov et al., 11 May 2026).

On conciseness, the best model is Kimi-K2.5-Think, with a rating around 1535 and average compressibility 1.92. GPT-5.4 is also strong, with compressibility 1.93, whereas Gemini-3.1-Pro is markedly verbose, with average compressibility about 3.51 (Petrov et al., 11 May 2026). ProofRank therefore separates models that are highly accurate but inflated in presentation from models that deliver compact arguments.

On computational ease, Qwen3.5-397B and GPT-5.4 are strongest (Petrov et al., 11 May 2026). This is notable because Qwen3.5-397B has relatively low correctness yet often attempts less brute-force methods. On cognitive simplicity, the spread is narrower, but GPT-5.4 remains the strongest, followed closely by DeepSeek-V3.2 and Gemini-3-Flash (Petrov et al., 11 May 2026). This indicates that some models can produce computationally light proofs that are still conceptually heavy, while others maintain both low computational burden and accessible methods.

On diversity, the leading models are Gemini-3.1-Pro, Kimi-K2.5-Think, and DeepSeek-V3.2, with Kimi having the highest raw entropy score of about 0.40 (Petrov et al., 11 May 2026). Diversity is therefore not tightly coupled to correctness: a model can be only moderately accurate yet varied in successful proof strategies.

On adaptivity, GPT-5.4 is dominant, with Elo 1558 and raw success 77.7% on technique-constrained tasks. Gemini-3.1-Pro reaches 52.1%, Step-3.5-Flash 40.9%, and GLM-5 39.6% (Petrov et al., 11 May 2026). Adaptivity is the quality metric most strongly correlated with correctness; the reported Spearman correlation between accuracy and adaptivity is about 0.84, while the correlations with conciseness and computational ease are near zero (Petrov et al., 11 May 2026).

A key empirical result is that removing the correctness prerequisite distorts rankings. If all proofs are compared, including incorrect ones, weaker models can appear superior because short but wrong solutions score well on conciseness and simplicity. ProofRank therefore uses correctness filtering to prevent “simple but wrong” proofs from dominating the quality rankings (Petrov et al., 11 May 2026).

5. Relation to adjacent proof-evaluation frameworks

ProofRank belongs to a broader landscape of proof-evaluation systems, but it occupies a distinct niche. A close comparison point is ProofGrader, which scores natural-language proofs on a 0–7 scale using problem-specific reference solutions and marking schemes; its final evaluator achieves MAE 0.926 against expert scores and, in best-of-$\tilde p$8 selection, reaches an average score of 4.14/7, closing 78% of the gap between a naïve binary evaluator and the human oracle (Ma et al., 14 Oct 2025). ProofRank differs in emphasizing pairwise quality comparison across correct proofs rather than direct fine-grained grading against human annotations.

ProofFlow addresses a different axis: structural fidelity in autoformalization. It represents proofs as DAGs of dependencies and evaluates systems with ProofScore, a composite metric over syntactic correctness, semantic faithfulness, and structural fidelity. In its reported experiments, ProofFlow DAG reaches 0.545, compared with 0.417 for noDAG, 0.123 for full-proof formalization, and 0.072 for step-proof formalization (Cabral et al., 13 Oct 2025). This suggests that a broader notion of “proof ranking” can include structural alignment, not only stylistic or pedagogical quality.

Pseudo-Formalization and Block Verification push in yet another direction: modular verification of natural-language proofs through self-contained proof blocks, dependency DAGs, and block-wise LLM checking. On both Hard2Verify and ArxivMathGradingBench, PF+BV is reported to Pareto-dominate direct LLM-as-judge baselines on precision–recall trade-offs for error finding (Barkallah et al., 19 May 2026). This indicates that ranking proofs by quality can be augmented with ranking them by local verifiability and error localization.

QEDBench provides a cautionary complement. It evaluates evaluators rather than solvers and shows that standard LLM-as-a-Judge protocols exhibit an alignment gap on university-level mathematical proofs. Certain judges show substantial positive bias, with mean score inflation up to +0.18, +0.20, +0.30, and +0.36, and Llama 4 Maverick reaches 74.8% leniency under a pass/fail threshold (Gonzalez et al., 24 Feb 2026). This suggests that any ProofRank-style benchmark relying on LLM judges must explicitly model evaluator bias, rubric sensitivity, and domain dependence.

Taken together, these systems imply a broader ecosystem of proof ranking: ProofRank foregrounds proof usefulness beyond correctness, ProofGrader foregrounds calibrated fine-grained scoring, ProofFlow foregrounds structural fidelity, PF+BV foregrounds modular verifiability, and QEDBench foregrounds judge alignment.

6. Limitations and significance

ProofRank explicitly treats proof quality as subjective and context-dependent, so its five metrics are proxies rather than a universal definition of good proof writing (Petrov et al., 11 May 2026). The benchmark relies on LLM judges for most comparisons, which introduces potential biases such as position bias and model-specific stylistic preferences, although the Bradley–Terry model includes a position-bias term and the paper reports human-validation studies for several metrics (Petrov et al., 11 May 2026).

Its domain is also specific: difficult, high-school competition final-answer problems rendered as proof-writing tasks. The framework therefore does not directly measure research-level exposition, long-form theorem proving, or formal proof artifacts (Petrov et al., 11 May 2026). This suggests that ProofRank should be interpreted as a benchmark of proof style and controllability in Olympiad-style reasoning, not as a complete theory of mathematical proof value.

Nonetheless, the benchmark establishes several substantive points. First, correctness and proof quality are only partially aligned: some models are accurate but verbose or brute-force, whereas others are less accurate yet produce cleaner or more varied methods (Petrov et al., 11 May 2026). Second, adaptivity and diversity expose capabilities that correctness-only leaderboards miss. Third, correctness-conditioned pairwise ranking is crucial: without it, elegantly wrong proofs can dominate quality comparisons (Petrov et al., 11 May 2026).

A plausible implication is that ProofRank reframes evaluation from “Can the model solve the problem?” to “What kind of proof does the model produce when it solves the problem?” In that sense, it turns proof generation into a multi-objective evaluation problem spanning validity, style, pedagogical usability, methodological breadth, and controllability.