Geoint Benchmark

• Geoint Benchmark is a multimodal framework that integrates text descriptions and TikZ-rendered diagrams to support formal geometric reasoning.
• It emphasizes the role of auxiliary constructions by requiring models to introduce and verify additional geometric elements using Lean4 code.
• The benchmark evaluates both answer-based and proof-based problems with rigorous metrics and expert-verified solution traces.

Geoint Benchmark is a multimodal benchmark for formal geometric reasoning, introduced together with Geoint-R1 to evaluate whether models can jointly interpret natural-language geometry problems and diagrams, introduce necessary auxiliary constructions, and express solutions in a formally verifiable representation. It contains 1,885 geometry problems spanning plane, spatial, and solid geometry, and each instance is coupled with structured textual annotations, TikZ-normalized diagrams, Lean4 code for auxiliary constructions, and expert-verified solution steps (Wei et al., 5 Aug 2025).

1. Definition and task formulation

Geoint Benchmark is designed around formal geometric reasoning rather than answer-only problem solving. Each problem combines a textual description $T_i$ and a visual diagram $I_i$, and the target output is formalized as

$(P_i, C_i, A_i) = \mathcal{F}_\theta(T_i, I_i),$

where $P_i$ is the reasoning process, $C_i$ is the set of auxiliary constructions, and $A_i$ is the final answer or proved conclusion; when no auxiliary construction is needed, $C_i = \emptyset$ (Wei et al., 5 Aug 2025).

The benchmark covers both answer-based and proof-based problems. Its topics include plane, spatial, and solid geometry, and the paper states that the problems are categorized into nine knowledge types, listing Triangle, Solid Geometry, Quadrilateral, Function, Positional Relationship, Circle, Geometry Figures, and Intersecting/Parallel Lines (Wei et al., 5 Aug 2025). This organization makes the benchmark broader than a single school-style subdomain while keeping a common formal target in Lean4.

A defining property of the benchmark is that it treats auxiliary construction as a first-class reasoning component. Rather than scoring only the final numeric or symbolic result, it evaluates whether a model can identify and specify extra points, lines, or segments that are not present in the original figure but are required to complete the proof or derivation (Wei et al., 5 Aug 2025).

2. Data collection, normalization, and annotation

Problems were collected from publicly available web resources and then manually reviewed, with items filtered and categorized into plane versus solid geometry before formalization (Wei et al., 5 Aug 2025). The visual component of each problem was not left in raw form: original diagrams were extracted, converted into TikZ code, compiled back into rendered figures, and then compared with the original by human annotators, with only high-fidelity reconstructions retained (Wei et al., 5 Aug 2025).

For high-quality diagrams, DeepSeek-R1 was used to assist in translating TikZ geometry into Lean4 formal objects and proofs, after which mathematics experts manually checked the generated Lean4 code. The expert review covered the correctness of assumptions, the correctness and necessity of auxiliary constructions, and the validity of the proofs themselves (Wei et al., 5 Aug 2025).

Many source problems were multi-part. The benchmark therefore splits original items into multiple question–answer pairs under a prompt-plus-question structure. When a later sub-question depends on an earlier conclusion, that earlier result is explicitly inserted into context so that each benchmark instance remains logically well formed (Wei et al., 5 Aug 2025).

The curation process included extensive human oversight. The paper states that three mathematics professionals with master’s degrees were involved throughout, and every entry was double-checked for category accuracy and quality (Wei et al., 5 Aug 2025).

3. Benchmark instance structure and formal artifacts

Each benchmark instance contains a coordinated set of natural-language, visual, and formal components. The natural-language problem statement supplies the geometry description and, for multi-part items, any prior conclusions needed for the current subtask. The diagram is rendered from retained TikZ source, so the image and the formal representation are linked rather than independently authored (Wei et al., 5 Aug 2025).

Structured annotations record the problem’s knowledge type, whether it belongs to the “Auxiliary line” or “No auxiliary line” subset, and whether it is answer-based or proof-based. Lean4 code represents both the initial geometric configuration and any auxiliary constructions, while formal statements encode the target goal as a theorem or lemma. Many problems also include reference proofs and step-by-step solution traces aligned with the formal structure (Wei et al., 5 Aug 2025).

Component	Role	Form
Problem statement	Describes configuration and target	Natural language
Diagram	Visualizes the geometry	TikZ-rendered image
Structured annotations	Records subset and task type	Metadata
Constructions	Encodes initial and auxiliary geometry	Lean4 code
Goal and proof	Formal target and solution	Lean4 theorem/proof
Solution trace	Human-readable derivation	Step-by-step text

This composition allows the benchmark to support both informal explanatory evaluation and formal proof-oriented evaluation. It also ties visual interpretation to formal verification more tightly than benchmarks that provide only text and diagrams (Wei et al., 5 Aug 2025).

4. Auxiliary constructions and Lean4-centered reasoning

Dynamic auxiliary constructions are the benchmark’s central technical axis. The original figure is often insufficient for direct reasoning, so the solver must add new geometric elements—such as a parallel line through a point, an intersection point, or a line connecting derived points—based on the structure of the problem rather than from a fixed template (Wei et al., 5 Aug 2025).

The paper gives two illustrative cases. In a planar answer-based example, the successful solution requires drawing a parallel line through point $E$, after which angle relationships become derivable. In a spatial proof-based example involving a pyramid, the solution introduces the intersection point $O$ of base diagonals, connects $O$ to $I_i$0, proves $I_i$1, and then concludes $I_i$2 plane $I_i$3 (Wei et al., 5 Aug 2025).

Within the associated Geoint-R1 framework, auxiliary correctness is formalized by a binary metric

$I_i$4

and this signal is incorporated into a verification-oriented reward design. The paper further defines

$I_i$5

with total reward

$I_i$6

where $I_i$7 measures output-format compliance (Wei et al., 5 Aug 2025). These formulas belong to the training and evaluation ecosystem around the benchmark, but they also clarify why auxiliary construction is not treated as incidental metadata.

Lean4 is used to encode constructions, assumptions, and proof goals. This gives the benchmark a theorem-proving orientation: auxiliary objects must not only be plausible geometric ideas but also be type-correct and consistent with the formal geometry environment (Wei et al., 5 Aug 2025).

5. Evaluation protocol and empirical behavior

Evaluation is automated with DeepSeek-V3. For answer-based questions, the final answer is extracted and scored as 1 if correct or mathematically equivalent and 0 otherwise. For proof-based questions, DeepSeek-V3 applies a weighted rubric over logical validity, completeness, correctness of the final conclusion, construction of auxiliary lines, and clarity, with aggregate score

$I_i$8

The paper reports answer and proof results as percentages and also analyzes auxiliary-line and non-auxiliary-line subsets separately (Wei et al., 5 Aug 2025).

Model	Answer (%)	Proof (%)	Overall (%)
Geoint-R1	57.01	72.43	64.72
Gemini-1.5-pro	51.53	73.50	62.52
GPT-4o	42.68	77.99	60.34
MMR1-Math-v0-7B	49.08	71.93	60.51
LLaVA-v1.6-7B	—	—	22.76
Yi-VL-6B	—	—	18.87
Phi-3.5-V-4B	—	—	10.83

These results indicate that the benchmark is difficult even for strong multimodal systems. The paper notes that many open-source vision-LLMs remain below 30% average accuracy, while Geoint-R1 obtains the best overall average despite having 7B parameters (Wei et al., 5 Aug 2025).

The auxiliary-line subset is a particularly important stress point. On that subset, Geoint-R1 reaches 68.63% on answer-based items and 68.40% on proof-based items; on non-auxiliary-line items, it reaches 51.77% and 76.39%, respectively (Wei et al., 5 Aug 2025). The benchmark therefore exposes a different difficulty profile for explicit construction problems than for direct proof continuation.

Ablation studies in Geoint-R1 reinforce this point. Removing the verification reward lowers the overall average from 64.72% to 55.78%; removing reinforcement learning lowers it to 55.33%; removing curriculum learning lowers it to 56.82% (Wei et al., 5 Aug 2025). The paper interprets the largest degradation as evidence that auxiliary-line correctness is not a superficial attribute but a major source of difficulty.

6. Position among related benchmarks and known limitations

Geoint Benchmark differs from existing math and geometry benchmarks by coupling diagrams, structured annotations, and Lean4 formalizations with explicit auxiliary constructions and process-based proof evaluation (Wei et al., 5 Aug 2025). This places it alongside newer diagnostic geometry evaluations, but with a distinct emphasis on formal code. For example, GeoBench organizes geometric reasoning into four levels—Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking—through six multiple-choice tasks generated from formally verified problems, whereas Geoint centers its benchmark structure on Lean4-coded constructions and proof artifacts (Feng et al., 30 Dec 2025).

The benchmark’s limitations are also clearly identified. Its coverage is broad within classical geometry, but it still focuses on school-level plane and solid geometry rather than areas such as non-Euclidean or projective geometry. Diagram styles are normalized through TikZ, which improves consistency but may reduce exposure to noisier or less standardized figures. Formalization currently depends on Lean4, and proof evaluation still relies in part on DeepSeek-V3 grading rather than exclusively on direct theorem checking. Auxiliary-line correctness is evaluated by exact matching, so alternative but valid constructions may not receive credit (Wei et al., 5 Aug 2025).

The paper points to several future directions: expanding to more geometry domains, increasing diagram diversity, strengthening evaluation through direct Lean4 type-checking, relaxing auxiliary-line evaluation so equivalent constructions can be recognized, and enabling interactive exploration in which models iteratively propose constructions and receive diagram feedback (Wei et al., 5 Aug 2025). Taken together, these directions suggest a benchmark intended not only as a leaderboard instrument but also as an infrastructure for research on formally grounded multimodal reasoning.