QEDBench: Benchmarking Mathematical Proof Evaluation
- QEDBench is a benchmark framework that quantifies the alignment gap between automated grading and expert human evaluation using a dual-rubric approach.
- It decouples solver performance from grading bias by contrasting course-specific and expert common-knowledge standards to expose systematic discrepancies.
- The framework includes 272 advanced proof problems, extensive human annotations, and a detailed evaluator–solver matrix across diverse mathematical domains.
QEDBench is a benchmark and empirical study for quantifying the “Alignment Gap” in automated evaluation of university-level mathematical proofs. It targets upper-undergraduate to early-graduate mathematics, where standard “LLM-as-a-Judge” protocols are reported to correlate poorly with PhD-level human graders and to systematically reward stylistic or surface plausibility while missing subtle logical gaps or illicit use of advanced theorems (Gonzalez et al., 24 Feb 2026). The benchmark is designed to decouple judge bias from solver skill, to contrast pedagogical and research-oriented grading norms through a dual-rubric framework, and to provide a reproducible, human-verified ground truth for auditing frontier evaluators on proof assessment.
1. Research problem and scope
QEDBench is motivated by a shift in emphasis from proof generation to proof evaluation. The central claim is that, as LLMs saturate elementary benchmarks, reliability of automated evaluation becomes the critical bottleneck for advanced mathematical reasoning tasks (Gonzalez et al., 24 Feb 2026). In this setting, the benchmark focuses specifically on proof-based mathematics at the university level rather than on elementary contest-style or short-answer tasks.
The study identifies a systematic “Alignment Gap” in LLM-based grading. In the benchmark’s terminology, this gap measures divergence between model-assigned scores and expert human judgments. The reported failure mode is not merely random noise: automated judges are said to exhibit systematic leniency or harshness, and these biases vary by mathematical domain. This framing places QEDBench at the intersection of benchmark design, mathematical reasoning evaluation, and model calibration.
A further distinguishing feature is that QEDBench is organized around proof evaluation rather than proof synthesis alone. This matters because the benchmark treats grading as a structured epistemic task: evaluators must determine whether an argument is logically valid, whether omitted steps are acceptable, and whether a proof violates course-level method constraints. This suggests that the benchmark is intended to probe not just output fluency, but rubric adherence and domain-sensitive judgment.
2. Dual-rubric framework
A defining component of QEDBench is its dual-rubric design, which contrasts two grading standards for every proof (Gonzalez et al., 24 Feb 2026). The first is the Course-Specific Rubric (Pedagogical Standard), which enforces only definitions and methods covered in a standard undergraduate course and penalizes any advanced machinery used without derivation. The benchmark gives examples such as invoking the Residue Theorem or Hall’s Marriage Theorem without derivation. The second is the Expert Common-Knowledge Rubric (Research Standard), which assumes a domain specialist’s implicit background, allows omission of trivial steps, and penalizes hidden circularities or structural logical errors.
This duality is methodologically important because it separates two notions of correctness that are often conflated in mathematical grading. Under a course-specific standard, a proof may be mathematically sound yet pedagogically unacceptable if it uses machinery outside the intended curriculum. Under an expert common-knowledge standard, the same proof may receive a higher score if its omitted steps would normally be taken as routine by specialists. QEDBench operationalizes this distinction rather than treating “correctness” as a single scalar notion.
The rubric engineering process is also specified. Initial drafts were synthesized by GPT-5.2 Pro, verified by Gemini 3.0 Pro, and then iteratively refined by PhD-level experts to match their grading criteria. This is a notable design choice because the final rubrics are explicitly expert-validated, while model assistance is confined to drafting and verification.
A plausible implication is that QEDBench is not only measuring whether judges agree with humans in aggregate, but also whether they remain stable under shifts in grading philosophy. That distinction is central for applications in pedagogy, where rubric compliance can be as important as mathematical truth.
3. Dataset composition and human annotation
QEDBench contains 272 upper-undergraduate/early-graduate proof-based problems spanning ten disciplines: Analysis, Complex Analysis, Abstract Algebra, Discrete Math, Probability, ODEs, Number Theory, Combinatorics, Algorithms, Graph Theory (Gonzalez et al., 24 Feb 2026). The solutions were generated by five state-of-the-art solver models: o3-deep-research, GPT-5 Pro, Claude Sonnet 4.5, Gemini 3.0 Pro, DeepSeek-Prover-V2.
Human ground truth is a major component of the benchmark. The paper reports 48 expert evaluators, described as PhD holders or candidates matched to their sub-fields, and approximately 1,000 hours of grading. The use of field-matched evaluators is significant because proof assessment in advanced mathematics is often highly domain-dependent; the adequacy of a contour integration argument, a combinatorial construction, or an algebraic derivation may depend on tacit disciplinary conventions.
The scoring scale is discrete:
According to the benchmark description, this scale distinguishes structural failures, minor expository oversights, and related grading categories. The presence of both $0.9$ and $1.0$ is notable, because it allows the benchmark to separate near-complete correctness from fully satisfactory solutions.
The released benchmark also includes the full problem set, dual rubrics, solver outputs in , human expert scores, justifications, markings, judge-model evaluation logs, and analysis scripts. It additionally reports an adversarial leakage audit. Taken together, these components make QEDBench not just a leaderboard substrate but a reproducible evaluation artifact.
4. Evaluation matrix and alignment metrics
The empirical core of QEDBench is a fully crossed Evaluator–Solver matrix under both rubrics (Gonzalez et al., 24 Feb 2026). The seven judges are GPT-5.2 Pro, Claude Opus 4.5, Gemini 3.0 Pro, Grok 4, Qwen 2.5 Max, DeepSeek-V3, Llama 4 Maverick. The five solvers are the models used to generate candidate proofs.
For each judge , solver , problem , and rubric , the benchmark records a score
Aggregate performance for each judge is reported through mean score and pass rate across all solver outputs and problems:
$0.9$0
$0.9$1
The principal calibration measure is the Alignment Gap:
$0.9$2
where $0.9$3 is the expert-assigned score on the same scale. A positive $0.9$4 denotes systematic leniency or grade inflation; a negative $0.9$5 denotes excessive strictness.
This metric is deliberately signed rather than absolute. That design choice matters because it distinguishes a judge that is merely noisy from one that is directionally biased. QEDBench therefore treats misalignment as a calibration problem, not only a correlation problem. The release materials further state that researchers can compute $0.9$6, pass-rate gaps, MAE, and Pearson $0.9$7 against human consensus, although the benchmark’s headline statistic is the Alignment Gap itself.
5. Quantitative findings
Using the Expert Common-Knowledge rubric unless otherwise noted, QEDBench reports substantial judge-side inflation for several frontier evaluators (Gonzalez et al., 24 Feb 2026). The benchmark highlights four models with large positive mean score deltas:
| Judge | Alignment Gap $0.9$8 | Additional note |
|---|---|---|
| Claude Opus 4.5 | $0.9$9 | Mean score inflation |
| DeepSeek-V3 | $1.0$0 | Mean score inflation |
| Qwen 2.5 Max | $1.0$1 | Mean score inflation |
| Llama 4 Maverick | $1.0$2 | 74.8% false-positive pass rate |
The best-aligned evaluator is reported to be GPT-5.2 Pro, with $1.0$3 and a pass-rate gap of approximately $1.0$4. In the benchmark’s framing, this does not imply perfect agreement with humans, but it does indicate substantially better calibration than the more inflationary judges.
On the solver side, the highest human-evaluated performance is attributed to Gemini 3.0 Pro, with average score 0.91 and 86.4% pass rate, followed by GPT-5 Pro with average score 0.84 and 76.8% pass rate. These figures are separate from judge calibration: QEDBench explicitly distinguishes the quality of generated proofs from the reliability of automated graders.
A major empirical result concerns discrete-domain collapse for some solver models under human evaluation:
| Domain | Gemini 3.0 Pro | GPT-5 Pro |
|---|---|---|
| Discrete Math | 0.89 | 0.72 |
| Graph Theory | 0.90 | 0.74 |
| Combinatorics | 0.73 | 0.68 |
The same benchmark description reports Claude Sonnet 4.5 scores of 0.63 in Discrete Math, 0.50 in Graph Theory, and 0.50 in Combinatorics (Gonzalez et al., 24 Feb 2026). By contrast, continuous domains such as ODEs and Complex Analysis are described as near-saturated by most solvers, with 100% pass rate in ODEs for GPT-5 Pro and Claude Sonnet 4.5.
These findings motivate two distinct conclusions. First, automated judges can be badly miscalibrated even when solver performance is strong. Second, solver competence itself is uneven across domains, with discrete mathematics emerging as a particularly difficult regime for several models. The paper explicitly notes that alignment failure is not uniform and correlates with domain structure, including contrasts such as constructive search versus template retrieval and human-model training biases.
6. Domain dependence, release, and research use
QEDBench reports that the bidirectional Alignment Gap is strongly domain-dependent (Gonzalez et al., 24 Feb 2026). In Discrete Math, the study highlights leniency from Llama 4 Maverick with $1.0$5 and Qwen 2.5 Max with $1.0$6. In Complex Analysis, the pattern reverses: DeepSeek-V3 has $1.0$7 and Grok 4 has $1.0$8, indicating harsher-than-human grading. The benchmark further notes minor negative bias in Algorithms for structured pseudo-code evaluators.
These asymmetries are important because they challenge the idea that a single scalar judge-quality score is sufficient. A judge can appear reasonable in aggregate while remaining systematically biased within particular subfields. This suggests that evaluator benchmarking for mathematics should be domain-stratified rather than globally averaged whenever deployment involves heterogeneous curricula or research areas.
QEDBench is publicly released at https://github.com/qqliu/Yale-QEDBench. The release includes the full problem set, dual rubrics, all solver outputs in $1.0$9, human expert scores and justifications, judge-model evaluation logs in JSON, and scripts to reproduce mean scores, pass rates, and 0 heatmaps. The benchmark description states that researchers can use it to benchmark new evaluator models or fine-tuned LLM judges, analyze domain-wise calibration or bias, and compare the effects of prompt engineering, fine-tuning, or chain-of-thought interventions on alignment metrics.
The release notes also state that QEDBench can support development of reward models or process-supervised pipelines targeting the identified Sycophancy Trap and Rubric Insensitivity limitations. Because those terms are presented as benchmark-specific failure modes, they should be understood in the context of QEDBench’s analysis rather than as universally standardized categories.
A common misconception would be to treat QEDBench as a benchmark for mathematical problem solving only. The benchmark is more specific: it is a framework for auditing the reliability of automated proof evaluation against expert human grading under explicitly contrasted rubrics. Its primary contribution is therefore methodological as much as empirical, providing a reusable instrument for studying when and how LLM judges diverge from mathematically trained human evaluators.