---
title: Geoint Benchmark
url: https://www.emergentmind.com/topics/geoint-benchmark
type: topic
---

# Geoint Benchmark

Geoint Benchmark is a multimodal benchmark for formal geometric reasoning, introduced together with Geoint-R1 to evaluate whether models can jointly interpret natural-language geometry problems and diagrams, introduce necessary auxiliary constructions, and express solutions in a formally verifiable representation. It contains 1,885 geometry problems spanning plane, spatial, and solid geometry, and each instance is coupled with structured textual annotations, TikZ-normalized diagrams, Lean4 code for auxiliary constructions, and expert-verified solution steps [2508.03173].

## 1. Definition and task formulation

Geoint Benchmark is designed around formal geometric reasoning rather than answer-only problem solving. Each problem combines a textual description \(T_i\) and a visual diagram \(I_i\), and the target output is formalized as
\[
(P_i, C_i, A_i) = \mathcal{F}_\theta(T_i, I_i),
\]
where \(P_i\) is the reasoning process, \(C_i\) is the set of auxiliary constructions, and \(A_i\) is the final answer or proved conclusion; when no auxiliary construction is needed, \(C_i = \emptyset\) [2508.03173].

The benchmark covers both answer-based and proof-based problems. Its topics include plane, spatial, and solid geometry, and the paper states that the problems are categorized into nine knowledge types, listing Triangle, Solid Geometry, Quadrilateral, Function, Positional Relationship, Circle, Geometry Figures, and Intersecting/Parallel Lines [2508.03173]. This organization makes the benchmark broader than a single school-style subdomain while keeping a common formal target in Lean4.

A defining property of the benchmark is that it treats auxiliary construction as a first-class reasoning component. Rather than scoring only the final numeric or symbolic result, it evaluates whether a model can identify and specify extra points, lines, or segments that are not present in the original figure but are required to complete the proof or derivation [2508.03173].

## 2. Data collection, normalization, and annotation

Problems were collected from publicly available web resources and then manually reviewed, with items filtered and categorized into plane versus solid geometry before formalization [2508.03173]. The visual component of each problem was not left in raw form: original diagrams were extracted, converted into TikZ code, compiled back into rendered figures, and then compared with the original by human annotators, with only high-fidelity reconstructions retained [2508.03173].

For high-quality diagrams, DeepSeek-R1 was used to assist in translating TikZ geometry into Lean4 formal objects and proofs, after which mathematics experts manually checked the generated Lean4 code. The expert review covered the correctness of assumptions, the correctness and necessity of auxiliary constructions, and the validity of the proofs themselves [2508.03173].

Many source problems were multi-part. The benchmark therefore splits original items into multiple question–answer pairs under a prompt-plus-question structure. When a later sub-question depends on an earlier conclusion, that earlier result is explicitly inserted into context so that each benchmark instance remains logically well formed [2508.03173].

The curation process included extensive human oversight. The paper states that three mathematics professionals with master’s degrees were involved throughout, and every entry was double-checked for category accuracy and quality [2508.03173].

## 3. Benchmark instance structure and formal artifacts

Each benchmark instance contains a coordinated set of natural-language, visual, and formal components. The natural-language problem statement supplies the geometry description and, for multi-part items, any prior conclusions needed for the current subtask. The diagram is rendered from retained TikZ source, so the image and the formal representation are linked rather than independently authored [2508.03173].

Structured annotations record the problem’s knowledge type, whether it belongs to the “Auxiliary line” or “No auxiliary line” subset, and whether it is answer-based or proof-based. Lean4 code represents both the initial geometric configuration and any auxiliary constructions, while formal statements encode the target goal as a theorem or lemma. Many problems also include reference proofs and step-by-step solution traces aligned with the formal structure [2508.03173].

| Component | Role | Form |
|---|---|---|
| Problem statement | Describes configuration and target | Natural language |
| Diagram | Visualizes the geometry | TikZ-rendered image |
| Structured annotations | Records subset and task type | Metadata |
| Constructions | Encodes initial and auxiliary geometry | Lean4 code |
| Goal and proof | Formal target and solution | Lean4 theorem/proof |
| Solution trace | Human-readable derivation | Step-by-step text |

This composition allows the benchmark to support both informal explanatory evaluation and formal proof-oriented evaluation. It also ties visual interpretation to formal verification more tightly than benchmarks that provide only text and diagrams [2508.03173].

## 4. Auxiliary constructions and Lean4-centered reasoning

Dynamic auxiliary constructions are the benchmark’s central technical axis. The original figure is often insufficient for direct reasoning, so the solver must add new geometric elements—such as a parallel line through a point, an intersection point, or a line connecting derived points—based on the structure of the problem rather than from a fixed template [2508.03173].

The paper gives two illustrative cases. In a planar answer-based example, the successful solution requires drawing a parallel line through point \(E\), after which angle relationships become derivable. In a spatial proof-based example involving a pyramid, the solution introduces the intersection point \(O\) of base diagonals, connects \(O\) to \(E\), proves \(OE \parallel PC\), and then concludes \(PC \parallel\) plane \(BDE\) [2508.03173].

Within the associated Geoint-R1 framework, auxiliary correctness is formalized by a binary metric
\[
F_{aux} = 1 \text{ if the model’s auxiliary constructions exactly match the reference set; } 0 \text{ otherwise,}
\]
and this signal is incorporated into a verification-oriented reward design. The paper further defines
\[
F_{corr}(x)=
\begin{cases}
\min(1,\mathrm{acc}+\lambda), & \text{if auxiliary are correct},\\
\max(0,\mathrm{acc}-\lambda), & \text{otherwise,}
\end{cases}
\]
with total reward
\[
R(x)=\alpha F_{corr}(x)+\beta F_{aux}(x)+\gamma F_{fmt}(x),
\]
where \(F_{fmt}\) measures output-format compliance [2508.03173]. These formulas belong to the training and evaluation ecosystem around the benchmark, but they also clarify why auxiliary construction is not treated as incidental metadata.

Lean4 is used to encode constructions, assumptions, and proof goals. This gives the benchmark a theorem-proving orientation: auxiliary objects must not only be plausible geometric ideas but also be type-correct and consistent with the formal geometry environment [2508.03173].

## 5. Evaluation protocol and empirical behavior

Evaluation is automated with DeepSeek-V3. For answer-based questions, the final answer is extracted and scored as 1 if correct or mathematically equivalent and 0 otherwise. For proof-based questions, DeepSeek-V3 applies a weighted rubric over logical validity, completeness, correctness of the final conclusion, construction of auxiliary lines, and clarity, with aggregate score
\[
\text{score} = 0.3 \cdot \text{validity} + 0.2 \cdot \text{completeness} + 0.2 \cdot \text{correctness} + 0.2 \cdot \text{construction} + 0.1 \cdot \text{clarity}.
\]
The paper reports answer and proof results as percentages and also analyzes auxiliary-line and non-auxiliary-line subsets separately [2508.03173].

| Model | Answer (%) | Proof (%) | Overall (%) |
|---|---:|---:|---:|
| Geoint-R1 | 57.01 | 72.43 | 64.72 |
| Gemini-1.5-pro | 51.53 | 73.50 | 62.52 |
| GPT-4o | 42.68 | 77.99 | 60.34 |
| MMR1-Math-v0-7B | 49.08 | 71.93 | 60.51 |
| LLaVA-v1.6-7B | — | — | 22.76 |
| Yi-VL-6B | — | — | 18.87 |
| Phi-3.5-V-4B | — | — | 10.83 |

These results indicate that the benchmark is difficult even for strong multimodal systems. The paper notes that many open-source vision-language models remain below 30% average accuracy, while Geoint-R1 obtains the best overall average despite having 7B parameters [2508.03173].

The auxiliary-line subset is a particularly important stress point. On that subset, Geoint-R1 reaches 68.63% on answer-based items and 68.40% on proof-based items; on non-auxiliary-line items, it reaches 51.77% and 76.39%, respectively [2508.03173]. The benchmark therefore exposes a different difficulty profile for explicit construction problems than for direct proof continuation.

Ablation studies in Geoint-R1 reinforce this point. Removing the verification reward lowers the overall average from 64.72% to 55.78%; removing reinforcement learning lowers it to 55.33%; removing curriculum learning lowers it to 56.82% [2508.03173]. The paper interprets the largest degradation as evidence that auxiliary-line correctness is not a superficial attribute but a major source of difficulty.

## 6. Position among related benchmarks and known limitations

Geoint Benchmark differs from existing math and geometry benchmarks by coupling diagrams, structured annotations, and Lean4 formalizations with explicit auxiliary constructions and process-based proof evaluation [2508.03173]. This places it alongside newer diagnostic geometry evaluations, but with a distinct emphasis on formal code. For example, GeoBench organizes geometric reasoning into four levels—Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking—through six multiple-choice tasks generated from formally verified problems, whereas Geoint centers its benchmark structure on Lean4-coded constructions and proof artifacts [2512.24119].

The benchmark’s limitations are also clearly identified. Its coverage is broad within classical geometry, but it still focuses on school-level plane and solid geometry rather than areas such as non-Euclidean or projective geometry. Diagram styles are normalized through TikZ, which improves consistency but may reduce exposure to noisier or less standardized figures. Formalization currently depends on Lean4, and proof evaluation still relies in part on DeepSeek-V3 grading rather than exclusively on direct theorem checking. Auxiliary-line correctness is evaluated by exact matching, so alternative but valid constructions may not receive credit [2508.03173].

The paper points to several future directions: expanding to more geometry domains, increasing diagram diversity, strengthening evaluation through direct Lean4 type-checking, relaxing auxiliary-line evaluation so equivalent constructions can be recognized, and enabling interactive exploration in which models iteratively propose constructions and receive diagram feedback [2508.03173]. Taken together, these directions suggest a benchmark intended not only as a leaderboard instrument but also as an infrastructure for research on formally grounded multimodal reasoning.

Source: https://www.emergentmind.com/topics/geoint-benchmark