Proof2Hybrid: Automated Proof Benchmarking

• Proof2Hybrid is a framework that automatically synthesizes proof-centric mathematical benchmarks from natural-language texts, enhancing the evaluation of proof understanding.
• It introduces the Proof2X roadmap to convert proofs into verifiable tasks using an innovative m-out-of-n multiple judge question format.
• The hybrid evaluation strategy, exemplified by AlgGeoTest, addresses limitations of traditional benchmarks by preserving the substantive complexity of mathematical proofs.

Proof2Hybrid is a framework for automatically synthesizing proof-centric mathematical benchmarks from natural-language mathematical texts. Introduced in “Proof2Hybrid: Automatic Mathematical Benchmark Synthesis for Proof-Centric Problems” (Peng et al., 4 Aug 2025), it is designed to evaluate whether LLMs genuinely understand mathematical statements and proofs rather than merely producing plausible answer strings. Its defining ideas are the Proof2X roadmap for transforming proof-bearing source material into automatically verifiable tasks, and a hybrid evaluation format called “$m$-out-of-$n$ multiple judge questions,” which is intended to preserve proof substance while remaining scalable and machine-gradable.

1. Problem setting and conceptual scope

Proof2Hybrid is motivated by a structural gap in mathematical evaluation. The paper distinguishes between number-centric problems, whose answers are concrete values or formulas and are therefore easy to verify automatically, and proof-centric problems, whose correctness depends on chains of reasoning, subtle use of definitions, and the validity of proofs, often without a single short canonical answer. Advanced mathematics is dominated by the second regime, but most widely used benchmarks concentrate on the first (Peng et al., 4 Aug 2025).

The framework is positioned against three existing benchmarking traditions. Benchmarks such as GSM8K, MATH, MMLU-Pro (Math), and AIME-style datasets mostly rely on problems with definite short answers. Manually constructed frontier benchmarks such as FrontierMath and HLE-Math can probe deeper reasoning, but they depend on scarce expert labor and are difficult to scale. Formal-proof benchmarks such as miniF2F and PutnamBench provide verifiable proof tasks, but formalization itself is labor-intensive. Proof2Hybrid aims at a different point in this design space: proof-centric evaluation based on natural-language mathematics, but with automated synthesis and automated verification.

This scope matters because the framework is not primarily a theorem-proving system and not primarily a formalization project. Its target is benchmark construction. A plausible implication is that Proof2Hybrid addresses the evaluation bottleneck for proof understanding rather than the proof-generation bottleneck itself.

2. Proof2X and the hybrid question format

The conceptual core of the framework is Proof2X, described as a roadmap for converting mathematical proofs into automatically verifiable question formats. The “X” denotes the target format: multiple-choice, true-or-false, blank-filling, or other verifiable tasks. In operational terms, Proof2X keeps an original mathematically correct item as a positive example and generates nearby but incorrect variants by modifying keywords, conditions, formulas, or, for proposition-proof pairs, the proof while keeping the proposition fixed (Peng et al., 4 Aug 2025).

The task format chosen by Proof2Hybrid is the $m$-out-of-$n$ multiple judge question. Each question contains $n$ items, each of which is either a definition or a proposition together with a proof. Exactly $m$ of the $n$ items are mathematically correct, and the model must identify the correct subset under the constraint that exactly $m$ are correct. In AlgGeoTest, the concrete configuration is

$m=2,\qquad n=6.$

This format is presented as “hybrid” because it combines properties usually associated with true/false and multiple-choice tasks. Like true/false, each item is judged for correctness. Like multiple choice, the response has a combinatorial structure. The random-guess baseline is therefore much lower than in standard true/false: $\frac{1}{\binom{n}{m}}$ which the paper writes as $n$0. For $n$1 and $n$2,

$n$3

whereas ordinary true/false has expected accuracy $n$4.

A further design rule is that all $n$5 items in one question must come from distinct seed items. This prevents the model from solving the question by comparing multiple edited versions of the same source proof and exploiting stylistic or local textual artifacts. The paper’s methodological claim is that the task thereby becomes a relative proof-judgment problem: even if models differ in strictness, the original seed item should remain “more correct” than a generated distractor.

3. Automated synthesis pipeline

Proof2Hybrid is organized as a multi-stage pipeline from natural-language mathematical corpora to final benchmark questions. The source corpus supplies seed items, specifically definitions and proposition-proof pairs. These are extracted, filtered, perturbed into distractors, filtered again, and finally aggregated into hybrid questions (Peng et al., 4 Aug 2025).

For seed item filtration, the framework uses $n$6 LLMs, each making $n$7 judgments per item, and retains an item if it is judged mathematically correct at least $n$8 times, with

$n$9

For AlgGeoTest, the values are

$m$0

The judge models are o3, Gemini-2.5-Pro, DeepSeek-R1, and Qwen3-235B-A22B.

For distractor generation, $m$1 LLMs each generate $m$2 candidate distractors, and $m$3 are randomly selected per model, with $m$4. For AlgGeoTest, the configuration is

$m$5

The generator models are DeepSeek-V3, Qwen2.5-72B-Instruct, GPT-4.1, Claude-4-Sonnet, and Gemini-2.5-Flash. After generation, distractors are deduplicated by removing spaces and line breaks and retaining one representative from each resulting equivalence class.

For distractor filtration, the framework keeps distractors only if they are judged incorrect a number of times within a target interval $m$6, with

$m$7

This lower bound excludes distractors that are too often judged correct or ambiguous; the upper bound excludes distractors that are too easy. For AlgGeoTest,

$m$8

A compact summary of the instantiated pipeline is:

Stage	Parameters	AlgGeoTest values
Seed filtration	$m$9	$n$0
Distractor generation	$n$1	$n$2
Distractor filtration	$n$3	$n$4
Question assembly	$n$5	$n$6

Question aggregation repeatedly samples $n$7 seed items and $n$8 distractors, with all $n$9 items required to come from distinct seed items, until the remaining pool is insufficient to form another full question.

4. AlgGeoTest

The main benchmark instance produced by Proof2Hybrid is AlgGeoTest, a benchmark in algebraic geometry. Its source corpus is The Stacks Project, chosen because it provides a large open-source reference text with content organized into tagged units such as definitions, lemmas, propositions, theorems, and proofs. The implementation reported in the paper begins from 1,100 seed items and produces a final benchmark containing 456 items (Peng et al., 4 Aug 2025).

AlgGeoTest uses the $n$0 hybrid format, so each benchmark item consists of six mathematical sub-items, exactly two of which are correct. The source items are of two types: definitions and proposition-proof pairs. This allows the benchmark to probe both definitional precision and proof validation. The algebraic-geometry setting is significant because the paper treats it as a frontier domain of modern mathematics, far removed from short-answer contest mathematics and structurally dominated by proof-centric reasoning.

The paper also reports an expert audit of the generated benchmark. According to that audit, over 98.75% of generated distractors are mathematically incorrect yet deceptively plausible, and more than 95% of benchmark questions meet the same standard. These figures are used as external evidence that the automated filtration pipeline produces mathematically meaningful negative examples rather than trivial corruptions.

5. Evaluation methodology and empirical results

Proof2Hybrid supports two evaluation modes. The first is generation-based evaluation, intended for instruction-tuned models. Here the model receives a six-item hybrid question and must identify the two mathematically correct items. The paper defines two scoring rules. Under the Loose metric, a prediction receives score 1 if both chosen items are correct, 0.5 if exactly one is correct, and 0 otherwise. Under the Tight metric, a prediction receives score 1 only if both are correct, and 0 otherwise (Peng et al., 4 Aug 2025).

The second mode is perplexity-based evaluation, intended particularly for base models. In this protocol, the benchmark is reformulated as standard multiple-choice questions, and the model chooses the option with the lowest perplexity. Because questions may have different numbers of options, weighted scores are used so that total maximum score remains 100 and the expected score under random guessing is equalized across questions.

The reported topline result is that contemporary state-of-the-art models perform poorly to moderately on AlgGeoTest relative to their performance on standard benchmarks. Selected generation-based results are:

Model	AlgGeoTest score	Rank
Gemini 2.5 Pro	61.4	1
Grok-4	59.0	2
o3	45.6	3
o4-mini	30.9	4
Qwen3-235B	23.9	5
Claude 4 Sonnet	23.7	6

Additional reported scores include Claude-4-Opus at 23.0, DeepSeek-R1 at 18.6, GPT-4.1 at 11.7, Kimi-K2 at 11.4, and DeepSeek-V3 at 7.7. The paper emphasizes that all of these models score in the high 80s to high 90s on MATH-500, whereas the best AlgGeoTest score is only around 60.

The benchmark also induces substantial ranking shifts relative to mainstream math benchmarks. Gemini 2.5 Pro rises from rank 3 on MATH-500 to rank 1 on AlgGeoTest; o3 drops from rank 1 to rank 3; Claude-4-Opus falls from rank 4 to rank 7; and o4-mini rises from rank 8 to rank 4. Correlation with standard benchmarks is only moderate: $n$1

$n$2

The paper interprets this as evidence that AlgGeoTest measures something related to, but distinct from, standard answer-centric mathematical ability.

Perplexity-based experiments on Qwen2.5, Qwen3, and Llama3.1 families show monotone scaling with model size. The paper also reports that reasoning models generally outperform non-reasoning counterparts. In the benchmark’s failure analysis, two patterns dominate: failure to detect subtle flaws in distractors, and hallucinating flaws in valid items.

6. Significance, related directions, and limitations

Proof2Hybrid contributes three linked ideas: an automated benchmark-synthesis pipeline from natural-language mathematics, the Proof2X conversion roadmap, and the $n$3-out-of-$n$4 multiple judge question format. Taken together, these constitute a method for making proof-centric benchmarking scalable in settings where short canonical answers are unavailable (Peng et al., 4 Aug 2025).

Its significance lies in changing the evaluation target. The benchmark measures proof discrimination rather than full proof generation. Success requires recognizing whether a definition or proof is mathematically valid after a controlled perturbation. This suggests complementarity with formal theorem-proving systems. For example, “HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement” (Hu et al., 21 May 2025) combines whole-proof synthesis and tactic-based generation for Isabelle and reaches 59.4% on the Isabelle miniF2F test set. A plausible implication is that Proof2Hybrid supplies an evaluation paradigm for proof understanding in natural language, whereas HybridProver targets proof construction in a formal assistant.

The framework also has clear limitations. It depends on a rich, structured, high-quality mathematical corpus; The Stacks Project is unusually suitable in this respect. The filtration stages rely on strong LLM judges, so benchmark quality partly inherits the competence and biases of present-day models. Residual synthesis errors remain possible, even if the audit suggests they are infrequent. The benchmark probes proof judgment more than original proof invention. Finally, the robustness arguments are plausible and partly quantified, but they do not amount to formal guarantees of validity or adversarial robustness.

Within those limits, Proof2Hybrid marks a shift from manually curated, small-scale proof evaluation toward automated synthesis from existing mathematical literature. Its central methodological claim is that proof-centric assessment need not require full formalization in a proof assistant and need not be restricted to expert-authored tasks. Instead, existing natural-language mathematical corpora can be transformed into robustly scorable benchmark items that expose genuine weaknesses in proof understanding, especially in advanced domains.