---
title: 'HAGeo-409: Olympiad Geometry Benchmark'
url: https://www.emergentmind.com/topics/hageo-409-benchmark
type: topic
---

# HAGeo-409: Olympiad Geometry Benchmark

HAGeo-409 is a comprehensive benchmark designed for the rigorous evaluation of automated theorem-proving systems in Olympiad-level Euclidean geometry. It comprises 409 problems spanning a broad range of human-assessed difficulty levels, explicitly addressing deficiencies in earlier benchmarks by ensuring both greater scale and precise, quantitative difficulty annotation. HAGeo-409 supports geometry-specific representations compatible with both synthetic (e.g., DDAR) and neural network–based (e.g., AlphaGeometry) inference, enabling fair and transparent comparison across diverse solution paradigms [2512.00097].

## 1. Motivation and Rationale

Existing benchmarks, most notably IMO-30, consist of only 30 problems extracted from International Mathematical Olympiads (2000–2022). IMO-30 lacks formal quantitative difficulty annotation, and subject-matter experts have identified its problems as largely “easy,” with an average difficulty of approximately $2.85$ on a $1$–$7$ scale. This limited size and range results in high variance during evaluation and constrains the assessment of solver capability, particularly for harder instances. HAGeo-409 was constructed to curate a larger and more representative set of Olympiad-level geometry theorems, each annotated with a human-assigned difficulty score, thereby enabling granular analysis of system performance relative to problem hardness. In addition, HAGeo-409 adopts a geometry-specific, numerically verified encoding, ensuring compatibility with both established synthetic engines and neural architectures [2512.00097].

## 2. Problem Acquisition and Verification Pipeline

Problem selection for HAGeo-409 begins with two principal sources: the Art of Problem Solving (AoPS) contest archive, contributing over 2,000 geometry problems across national, regional, and mock Olympiads, and ShuZhiMi, a WeChat mini-program for high-school mathematics, which contributed approximately 50 additional user-rated, Olympiad-style problems. 

The conversion and verification workflow operates as follows:
1. Each natural-language problem statement is translated into a construction-based, GeoGebra-inspired language by prompting GPT-4o with few-shot examples.
2. Preliminary, automated numeric verification is performed to rule out invalid or ill-posed theorems.
3. Problems failing conversion or verification (~50 %) undergo manual correction and inspection.

Problems retained in the benchmark meet the criteria of possessing a geometry-specific encoding, successful numeric verification, and a human-assigned difficulty rating. The final collection covers the gamut of standard Olympiad geometry topics, including triangle centers (incenter, centroid, circumcenter, orthocenter), circle theorems, angle-chasing, concurrency, collinearity, similarity, and power of a point [2512.00097].

## 3. Human-Assessed Difficulty Annotation

Each HAGeo-409 problem is annotated with a difficulty score on a 1–7 scale (1 = “very easy,” 7 = “very hard”), based directly on arithmetic mean ratings from ShuZhiMi users. No further normalization or rubrics are applied; thus, scores reflect the unweighted crowd-sourced consensus. For reporting and analysis, problems are grouped into five bins: $[1,3)$, $[3,4)$, $[4,5)$, $[5,6)$, $[6,7]$. 

This detailed annotation framework offers fine-grained insight into solver performance across the difficulty spectrum, which is absent in previous datasets such as IMO-30 [2512.00097].

## 4. Dataset Construction and Representation

Problems are encoded in a domain-specific textual language inspired by GeoGebra, comprising primitive geometric constructions:
- **Point definitions**: e.g., “A B C = triangle”
- **Lines**: “l = line A B”
- **Circles**: “ω = circle_center_point O P”
- **Intersections**: “X, Y = intersection l ω”
- **Special constructions**: midpoints, reflections, perpendicular feet, angle-equal loci, circumcenters, incenters, etc.

Each problem file concludes with a goal statement, such as “Prove: cong O A O K” or “Prove: collinear K O3 O6.” All geometric objects are referred by name; explicit coordinates are withheld from the solver. Numeric verification is used internally for solution validation.

For broader applicability, a manually corrected conversion into AlphaGeometry's point-only format is also provided, facilitating direct comparison between symbolic and neural approaches [2512.00097].

## 5. Statistical Overview

The benchmark's coverage and problem statistics are summarized below:

| Difficulty Bin | Number of Problems | Proportion (%) |
|:--------------:|:-----------------:|:--------------:|
| [1, 3)         | 161               | 39.4           |
| [3, 4)         | 112               | 27.4           |
| [4, 5)         |  71               | 17.4           |
| [5, 6)         |  43               | 10.5           |
| [6, 7]         |  22               | 5.4            |

- **Average difficulty**: 3.47 (compared to IMO-30’s 2.85).
- **Automated conversion**: ~50% of AoPS problems via GPT-4o succeeded; ~20% passed numeric verification automatically with the remainder revised manually.
- **Topic distribution**: Triangle centers & concurrency (~30%), circle theorems & power of a point (~25%), angle-chasing & similarity (~20%), collinearity & ratio arguments (~15%), advanced topics (~10%) [2512.00097].

## 6. Evaluation Methodology

Evaluation is standardized as follows:
- **Computational environment**: 64-core CPU for all DDAR-based and heuristic runs; 80GB A100 GPU leveraged exclusively for AlphaGeometry’s neural point proposer.
- **Per-problem runtime**: Each DDAR invocation receives a 60-second time limit.
- **Aggregate experiment duration**: Each experiment (e.g. pass@K evaluation) is limited to at most 1.5 hours on 64 cores.

**Pass@K metric definition**:
- For each problem, up to $K$ different auxiliary-point augmentations may be attempted.
- $S(K)$ denotes the number of problems solved within $K$ trials.
- $\mathrm{pass@}K~(\%) = 100 \times S(K) / 409$.

In experimental evaluation, $K$ is typically set to 2048 and 8192 to study solver scaling [2512.00097].

## 7. Baseline Performance and Comparative Analysis

HAGeo-409 enables systematic head-to-head comparison between neural (AlphaGeometry), random baseline, and heuristic methods (HAGeo). Success rates by difficulty bin and overall pass@K scores are summarized in the table below:

| Method                | [1,3) | [3,4) | [4,5) | [5,6) | [6,7] | Total (%)   |
|-----------------------|-------|-------|-------|-------|-------|-------------|
| AlphaGeometry         | 73.3% | 39.3% | 18.3% | 4.7%  | 0.0%  | 43.3        |
| Random @2048          | 78.9% | 55.4% | 18.3% | 4.7%  | 0.0%  | 49.9        |
| HAGeo @2048           | 87.6% | 77.7% | 40.8% | 11.6% | 4.5%  | 64.3        |
| Random @8192          | 79.5% | 61.6% | 25.4% | 7.0%  | 0.0%  | 53.3        |
| HAGeo @8192           | 92.5% | 83.0% | 50.7% | 16.3% | 9.1%  | 70.2        |

Key findings include an absolute improvement of approximately 20% by HAGeo over the random baseline at $K = 2048$ (64.3% vs 49.9%) and 17% at $K = 8192$ (70.2% vs 53.3%). Against AlphaGeometry, HAGeo achieves overall success rates increasing from 43.3% to 64.3% at $K = 2048$ and 70.2% at $K = 8192$. Notably, these gains are amplified in the more difficult bins ([3,4), [4,5), [5,6), [6,7]), confirming the heuristic method’s generalization to challenging Olympiad problems [2512.00097].

## 8. Adoption and Research Utility

HAGeo-409 is publicly accessible, with each problem provided as a plain-text GeoGebra-style file and an accompanying table of difficulty scores. Recommended benchmarking involves implementing the established pass@K protocol with a 60-second time budget per DDAR trial, reporting success rates by difficulty bin and overall, and comparing results to the baseline performance table. Alternate encoding in point-only format is provided for users focused on neural architectures; thus, HAGeo-409 defines a reproducible and challenging benchmark for advancing automated geometric theorem proving [2512.00097].

Source: https://www.emergentmind.com/topics/hageo-409-benchmark