AlgGeoTest: Proof-Centric Geometry Benchmark

• AlgGeoTest is a proof-centric benchmark for algebraic geometry that tests LLMs on identifying two correct answers among six options based on proof validation.
• It uses a four-stage synthesis pipeline—including seed filtering, distractor generation, and aggregation—to automatically produce 456 hybrid items from The Stacks Project.
• Empirical findings show that even top-performing LLMs score moderately, revealing challenges in abstract reasoning and detecting subtle flaws in natural-language proofs.

AlgGeoTest is a proof-centric benchmark for algebraic geometry introduced as the demonstration domain of the Proof2Hybrid framework. It is designed to evaluate LLMs on advanced natural-language mathematics in settings where correctness depends on proof structure, theorem use, and subtle logical consistency rather than on a final numeric answer. In its current arXiv usage, the term primarily denotes a benchmark comprising 456 hybrid-formatted items synthesized automatically from The Stacks Project; in a separate computational-geometry tradition, closely related “algebraic geometry test” language refers to elimination-based criteria for classifying geometric statements in automated reasoning (Peng et al., 4 Aug 2025, Kovács et al., 2018).

1. Domain, motivation, and benchmark identity

AlgGeoTest was proposed in response to a benchmark gap identified for proof-centric mathematics. Existing evaluations such as GSM8K, MATH, MMLU-Pro (Math), and AIME-style sets mostly emphasize number-centric problems with definite answers, whereas much of modern mathematics is proof-centric and not naturally reducible to short strings. More difficult benchmarks and formal benchmarks improve challenge level, but manual expert curation and formalization in Lean, Coq, or Isabelle remain costly. AlgGeoTest is intended to fill the intersection that prior benchmarks largely miss: it is proof-centric, in natural language, large-scale, and fully automatic (Peng et al., 4 Aug 2025).

Algebraic geometry was chosen because it is described as a central frontier domain of modern mathematics, and because The Stacks Project provides a structured tagged corpus containing definitions, lemmas, propositions, theorems, and proofs. That structure enables automatic extraction of natural-language definitions and proposition-proof pairs as seed items. The benchmark therefore targets abstract reasoning over schemes, sheaves, associated and weakly associated points, specialization maps, descent theory, divisors, and related topics, rather than school-geometry construction problems (Peng et al., 4 Aug 2025).

The benchmark’s unit of evaluation is a hybrid question rather than a single proposition. Each AlgGeoTest item contains six options, exactly two of which are mathematically correct. Those options are assembled from correct source items and carefully filtered distractors derived from other source items. The resulting task is not theorem proving in a proof assistant, and it is not standard multiple choice in the usual one-correct-answer sense; it is a constrained proof-validation problem over natural-language mathematical material (Peng et al., 4 Aug 2025).

2. Proof2Hybrid pipeline and benchmark synthesis

AlgGeoTest is the concrete algebraic-geometry instantiation of the hierarchy Proof2X $\rightarrow$ Proof2Hybrid $\rightarrow$ benchmark. Proof2X is the conceptual roadmap for converting mathematical proofs into automatically verifiable question types; Proof2Hybrid is the operational framework; AlgGeoTest is the resulting benchmark instance. For AlgGeoTest, the source corpus is The Stacks Project, from which the authors randomly selected 1,100 seed items. The two seed-item types are mathematical definitions and mathematical proposition-proof pairs. For proposition-proof pairs, the proposition is kept fixed and only the proof is modified to create distractors, making the task centrally about proof validation rather than theorem recognition (Peng et al., 4 Aug 2025).

The synthesis pipeline has four stages: seed-item filtration, distractor generation, distractor filtration, and aggregation into hybrid questions. In seed-item filtration, each seed is judged by $m_1$ strong LLMs, each queried $n_1$ times, and retained if it is judged mathematically correct on at least $k_1$ occasions, where

$k_1 > \frac{m_1 n_1}{2}.$

For AlgGeoTest, the choices are $m_1=4$, $n_1=3$, and $k_1=8$, using o3, Gemini-2.5-Pro, DeepSeek-R1, and Qwen3-235B-A22B. In distractor generation, $m_2$ LLMs each generate $\rightarrow$0 flawed but close variants by altering keywords, conditions, formulas, or proof steps; for AlgGeoTest, $\rightarrow$1, $\rightarrow$2, and $\rightarrow$3, with DeepSeek-V3, Qwen2.5-72B-Instruct, GPT-4.1, Claude-4-Sonnet, and Gemini-2.5-Flash as generation models. Distractor filtration then keeps only distractors judged incorrect between $\rightarrow$4 and $\rightarrow$5 times by a separate judging pool, under

$\rightarrow$6

For AlgGeoTest, $\rightarrow$7, $\rightarrow$8, $\rightarrow$9, and $m_1$0, with the same four strong judging models used in seed filtering (Peng et al., 4 Aug 2025).

Pipeline component	AlgGeoTest setting
Seed source	The Stacks Project
Seed count selected	1,100 seed items
Seed types	Definitions; proposition-proof pairs
Seed filter	$m_1$1, $m_1$2, $m_1$3
Distractor generation	$m_1$4, $m_1$5, $m_1$6
Distractor filter	$m_1$7, $m_1$8, $m_1$9, $n_1$0
Final question format	$n_1$1, $n_1$2
Final benchmark size	456 items

Aggregation draws $n_1$3 seed items and $n_1$4 distractors subject to the condition that all $n_1$5 options come from distinct seed items. For AlgGeoTest, the benchmark uses $n_1$6 and $n_1$7, so each question is a 2-out-of-6 multiple judge task. The stated reasons for choosing 2-out-of-6 are that two correct answers are harder than one, the post-filter seed:distractor ratio is about $n_1$8, and six options keep context length manageable (Peng et al., 4 Aug 2025).

3. Question format, verification logic, and evaluation protocols

The central formal device in AlgGeoTest is the “$n_1$9-out-of-$k_1$0 multiple judge question.” Each option is either a correct original definition or proposition-proof pair, or a distractor produced by controlled corruption. All $k_1$1 options are drawn from distinct seed items, and exactly $k_1$2 are correct. For AlgGeoTest, $k_1$3 and $k_1$4. The random-guessing success probability is

$k_1$5

so for AlgGeoTest

$k_1$6

This is presented as substantially more robust than binary true/false, whose expected accuracy under guessing is $k_1$7, and less vulnerable than ordinary multiple choice to shortcutting by comparing variants of the same seed item (Peng et al., 4 Aug 2025).

The distractors are intentionally close to the originals. Representative corruptions described for AlgGeoTest include changing “Noetherian” to “locally Noetherian,” replacing $k_1$8 by $k_1$9, reversing the direction of a map $k_1 > \frac{m_1 n_1}{2}.$0 to $k_1 > \frac{m_1 n_1}{2}.$1, and changing $k_1 > \frac{m_1 n_1}{2}.$2 to $k_1 > \frac{m_1 n_1}{2}.$3 in the definition of a strict normal crossings divisor. These modifications target theorem-condition tracking, proof comprehension, map directionality, and sensitivity to local hypotheses rather than rote recognition (Peng et al., 4 Aug 2025).

Two evaluation protocols are reported. The main protocol is generation-based evaluation for instruction-tuned and API models. The model sees the full 2-out-of-6 question and outputs which two options are correct. All evaluations are API-based and use default hyperparameters of each service. Scoring is reported with a loose metric—full credit if both selected answers are correct, half credit if exactly one is correct, zero otherwise—and a tight metric—full credit only if both answers are correct. A separate perplexity-based evaluation is used especially for base models: the grouped hybrid questions are converted into standard multiple-choice form by packaging one seed item with its distractors and choosing the option with lowest perplexity, with weighted scoring used so that expected score under random guessing is equalized across questions (Peng et al., 4 Aug 2025).

4. Empirical performance and benchmark behavior

AlgGeoTest is empirically hard. The reported generation-based scores place Gemini 2.5 Pro first with 61.4, followed by Grok-4 at 59.0 and o3 at 45.6. The next cluster includes o4-mini at 30.9, Qwen3-235B at 23.9, Claude 4 Sonnet at 23.7, Claude-4-Opus at 23.0, DeepSeek-R1 at 18.6, GPT-4.1 at 11.7, Kimi-K2 at 11.4, and DeepSeek-V3 at 7.7. The paper summarizes the overall picture by stating that even the best-performing model achieves only a moderate score of around 60, with scores commonly lower than 20, and that reasoning models commonly outperform non-reasoning counterparts (Peng et al., 4 Aug 2025).

Model	AlgGeoTest score
Gemini 2.5 Pro	61.4
Grok-4	59.0
o3	45.6
o4-mini	30.9
Qwen3-235B	23.9
Claude 4 Sonnet	23.7
DeepSeek-R1	18.6
GPT-4.1	11.7

AlgGeoTest also reorders model rankings relative to mainstream mathematics benchmarks. The paper notes that o3 is No.1 on MATH-500 but No.3 on AlgGeoTest, Gemini 2.5 Pro rises from No.3 on MATH-500 to No.1 on AlgGeoTest, o4-mini rises from No.8 to No.4, and Claude-4-Opus falls from No.4 to No.7. The reported coefficient-of-determination values are $k_1 > \frac{m_1 n_1}{2}.$4 for AlgGeoTest versus MATH-500 and $k_1 > \frac{m_1 n_1}{2}.$5 for AlgGeoTest versus AIME24, indicating moderate rather than redundant correlation (Peng et al., 4 Aug 2025).

The perplexity-based experiments on base models exhibit monotonic gains with model scale. The reported plotted scores include Qwen 2.5 increasing from 48.4 at 1.5B to 81.7 at 72B, Qwen 3 increasing from 43.9 at 0.6B to 83.3 at 30B, and Llama 3.1 increasing from 56.6 at 8B to 67.8 at 70B. The paper interprets this as sensible scaling behavior. At the same time, the generation-based results are used to argue that current LLMs still have profound deficits in proof comprehension, abstract reasoning, theorem-condition tracking, mathematical structure understanding, and detection of subtle flaws in near-correct proofs (Peng et al., 4 Aug 2025).

5. Relation to earlier algebraic-geometric testing and adjacent geometry benchmarks

A distinct but relevant usage of “algebraic geometry test” predates the benchmark. In computational automated reasoning for geometry, the paper “Detecting truth, just on parts” presents an elimination-based criterion for deciding whether a geometric statement is generally true, generally false, or true on parts, false on parts. Given a statement $k_1 > \frac{m_1 n_1}{2}.$6 with thesis polynomial $k_1 > \frac{m_1 n_1}{2}.$7 and a maximum-size independent parameter set $k_1 > \frac{m_1 n_1}{2}.$8 satisfying $k_1 > \frac{m_1 n_1}{2}.$9, the method computes

$m_1=4$0

Then $m_1=4$1 iff the statement is generally true, $m_1=4$2 iff the statement is generally false, and $m_1=4$3 iff the statement is true on parts, false on parts. The paper explicitly characterizes this as the paper’s “AlgGeoTest” in essence, emphasizing Gröbner-basis elimination over primary decomposition (Kovács et al., 2018).

Modern AlgGeoTest is also sharply distinct from school-geometry and dynamic-geometry evaluation suites. “UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression” introduces a multimodal benchmark with 4,998 calculation problems and 9,543 proving problems in school plane geometry, evaluates sequence generation from diagrams and text, and uses top-10 proof accuracy rather than formal proof checking (Chen et al., 2022). “GeoGebra Tools with Proof Capabilities” documents a 200-case benchmark suite for automated Euclidean theorem proving inside GeoGebra and reports 175/200 solved correctly by the complex algebraic geometry prover via Singular on desktop, with 125/200 in the web setting (Kovács et al., 2016). “Towards a Geometry Automated Provers Competition” proposes a competition infrastructure centered on TGTP, I2GATP, and multi-criteria evaluation including CPU time, readable proofs, and educationally relevant validation-time thresholds (Baeta et al., 2020). These resources concern Euclidean or multimodal geometric reasoning, whereas AlgGeoTest targets natural-language proof validation in advanced algebraic geometry (Peng et al., 4 Aug 2025).

This contrast is substantive. UniGeo and GeoGebra-based benchmarks are built around Euclidean constructions, diagrams, incidence, angle and length relations, and theorem proving in geometry education or dynamic geometry systems. AlgGeoTest instead uses definitions and proposition-proof pairs extracted from The Stacks Project, focuses on proof-centric natural-language mathematics, and evaluates whether models can identify the mathematically correct subset among subtly corrupted options. A plausible implication is that the shared word “geometry” conceals markedly different benchmark semantics: multimodal school-geometry deduction on one side, and abstract proof validation in modern algebraic geometry on the other (Chen et al., 2022, Peng et al., 4 Aug 2025).

6. Reliability, limitations, and interpretive issues

The benchmark’s reliability rests on multiple filtering layers and an expert audit. The paper reports that over 98.75\% of model-generated distractors are mathematically incorrect yet deceptively plausible, and that more than 95\% of benchmark questions meet the same standard, with every distractor satisfying the stated plausibility criterion. These claims are supported by the combination of seed filtering, multi-model distractor generation, multi-model distractor filtering, and manual expert review (Peng et al., 4 Aug 2025).

Several limitations are also stated. The paper does not provide a detailed taxonomy of algebraic-geometry subdomains, proof styles, or source distribution across The Stacks Project. Dependence on LLM judges during construction could introduce stylistic or model-family bias. Because the benchmark is sourced from a public corpus, contamination risk is not deeply discussed, although the transformed proof-validation format is described as likely reducing trivial memorization. No full inter-rater or item-level reliability statistics are reported beyond the audit percentages, and the paper leaves room for more formal psychometric analysis (Peng et al., 4 Aug 2025).

Two misconceptions are especially important to avoid. First, AlgGeoTest is not a formal theorem-prover benchmark in the sense of Lean, Coq, or Isabelle: it is natural-language, automatically scored, and proof-centric, but it does not rely on machine-checked derivations. Second, it is not a geometry benchmark in the school-Euclidean sense exemplified by multimodal diagram reasoning datasets. Its domain is algebraic geometry and related topics in the Stacks Project, and its core task is proof validation under controlled corruption rather than construction-based theorem proving or symbolic diagram parsing (Peng et al., 4 Aug 2025, Chen et al., 2022).

Taken together, these properties make AlgGeoTest both a benchmark and a benchmark-construction demonstration. It operationalizes proof-centric evaluation at scale in a frontier mathematical domain, while also exposing the present gap between strong performance on answer-centric math datasets and substantially weaker performance on natural-language proof validation in algebraic geometry (Peng et al., 4 Aug 2025).