GeoSketch Benchmark: Dynamic Geometry Reasoning

• GeoSketch Benchmark is a dynamic evaluation set that requires auxiliary constructions and affine diagram manipulations for solving geometric problems.
• It integrates a perception–reasoning–action loop, enabling models to execute diagram edits and verify updates through symbolic constraints.
• The benchmark features 390 curated problems with a geometry constraint solver to ensure consistency between visual edits and symbolic reasoning.

GeoSketch Benchmark is a geometry evaluation set introduced with the GeoSketch neural-symbolic framework to test multimodal reasoning under dynamic, verifiable diagram manipulation rather than static diagram interpretation. It consists of 390 curated problems that require auxiliary construction and/or affine transformations, and it is coupled to an interaction loop in which a model perceives a diagram, reasons symbolically, executes a sketch action, and updates the geometric state before producing a final answer. The benchmark is therefore not limited to reading diagrams or selecting answers from a fixed visual input; it is organized around executable edits whose consistency is checked against symbolic constraints and a geometry constraint solver (Weng et al., 26 Sep 2025).

1. Conceptual scope and problem setting

GeoSketch Benchmark is designed to move geometric multimodal reasoning from static diagram interpretation to dynamic, verifiable interaction. Its motivating claim is that most MLLMs process diagrams as static images and output text only, which excludes the iterative visuospatial operations that human solvers commonly use, especially drawing auxiliary lines and applying diagram-level transformations such as reflection, rotation, and translation. Within this formulation, geometric reasoning is recast as a perception–reasoning–action loop in which each visual edit is symbolically justified and computationally checked (Weng et al., 26 Sep 2025).

The benchmark specifically targets problems that humans typically solve by adding latent structure to a figure. The required actions include constructing angle bisectors, perpendiculars, parallels, midpoints, and extensions, and executing reflection, rotation, and translation. The benchmark’s emphasis is therefore not on passive visual recognition, but on whether a system can decide which construction or transformation is needed next and then maintain consistency between the edited image and its symbolic interpretation.

A common misconception is to treat GeoSketch Benchmark as a generic diagram-generation dataset. The benchmark is narrower and more technical than that. Its defining property is the requirement that auxiliary constructions and affine-style diagram manipulations be integral to solution trajectories, with each update checked by a geometry constraint solver. This suggests that the benchmark is best understood as an evaluation environment for closed-loop geometric reasoning rather than as a corpus of standalone geometry images.

2. Corpus composition, sources, and curation

The benchmark contains 390 curated problems selected from approximately 1,200 textbook/competition-style candidates. The retained problems had to require either auxiliary constructions or affine transformations, and they were translated into English and standardized to a decisive question-answer format. Multiple-choice, fill-in, and proof tasks were rewritten with definitive gold answers, expert annotators verified solution accuracy and diagram clarity, and problems that could not be decisively transformed were discarded (Weng et al., 26 Sep 2025).

Its answer-type distribution is explicitly reported: 201 Numerical problems, 108 Ratio problems, and 81 Descriptor problems. The corpus emphasizes multi-step reasoning, although exact step-count statistics per problem are not reported. The paper also does not provide exact counts for “auxiliary only” versus “transformation required” problems, nor fine-grained subtype counts for constructions such as angle bisectors, perpendiculars, parallels, midpoints, or extensions.

Each benchmark instance contains a structured JSON entry with a unique ID, a textual statement, paths to initial and solution images, a final gold answer, and a detailed step-by-step natural language proof or trajectory. Images are indispensable and not reconstructible from text alone. This sharply distinguishes the benchmark from text-complete geometry construction settings in which the full figure can be recovered from language alone.

The paper does not report train/validation/test splits for the 390 benchmark problems. In practice, the benchmark is presented as an evaluation set attached to the GeoSketch interaction protocol rather than as a conventional split-based supervised-learning dataset.

3. Representation: logic forms, sketch actions, and geometric state

GeoSketch Benchmark couples visual inputs with structured symbolic representations. Each problem includes an initial diagram image, a textual statement, and a gold solution trajectory; the framework additionally provides a logic form in JSON that describes entities, relations, and the goal. The logic form includes points, lines, segments, rays, circles, and polygons together with properties such as coordinates, endpoints, centers, and radii. Its constraints can encode parallelism, perpendicularity, collinearity, angle equality, congruence/similarity groups, midpoint relations, bisectors, incidence, length/angle measures, and transformation relationships (Weng et al., 26 Sep 2025).

The benchmark’s action space is correspondingly explicit. Primary actions include draw_line(p1, p2), reflect(object, line_of_symmetry), rotate(object, center, degrees), translate(object, vector), and label_point(name, coordinates). Auxiliary geometric operations such as constructing a perpendicular, constructing a parallel, bisecting an angle, marking a midpoint, and extending a line are represented as action commands plus symbolic constraints. For example, a perpendicular from $A$ to $BC$ is encoded by drawing a line from $A$ to a new point $D$ together with the constraints that $D$ lies on $BC$ and that $AD \perp BC$.

The benchmark’s treatment of transformations is formalized by the standard affine update

$x' = A x + b,$

or, in homogeneous coordinates,

$$p' = T p, \qquad T = \begin{pmatrix} A & b \ 0 & 1 \end{pmatrix}.$$

Within the benchmark and action API, however, the emphasized transformations are isometries: reflection, rotation, and translation. Rotation by $\theta$ around center $BC$0 uses

$BC$1

with the usual rotation matrix, reflection across a line uses the corresponding normal-vector formula, and translation uses $BC$2. The paper notes that collinearity and parallelism are preserved by affine transforms, ratios along a line are preserved, and areas scale by $BC$3, whereas angles are preserved only by similarity transforms (Weng et al., 26 Sep 2025).

A representative auxiliary-construction example in the benchmark encodes a problem about dropping a perpendicular from $BC$4 to $BC$5, then uses the resulting right-triangle structure and the Law of Cosines to compute $BC$6. This example illustrates the benchmark’s core design: the construction is not merely an annotation artifact but an executable action whose geometric consequences are part of the reasoning trace.

4. Evaluation protocol, symbolic verification, and learning setup

The paper reports two evaluation modes. In Pure Run (static), a model receives the text and initial image and outputs a final answer directly. In GeoSketch (dynamic), the model enters the perception–reasoning–action loop: at each step it receives the current logic form and history, outputs a reasoning sentence and a JSON action, the action is executed, and the loop continues until an answer is produced or a step cap is reached. The paper mentions a limit on the number of steps but does not give the number (Weng et al., 26 Sep 2025).

The reported benchmark metrics are centered on final answer quality. The primary metric is final problem-solving accuracy, with additional reporting by answer type: Numerical, Ratio, and Descriptor. Stepwise accuracy is described conceptually in the abstract, but it is not quantitatively reported; the limitations section states explicitly that evaluation focuses on final answer accuracy and does not assess intermediate step correctness. This is important because the benchmark supplies gold trajectories and symbolic state updates, yet its official reporting remains answer-centric.

Symbolic verification proceeds at two levels. First, action outputs are checked for JSON schema validity. Second, a geometry constraint solver (GCS) checks whether the updated logic form is consistent. If an action-induced state violates a constraint, the GCS reports a positive error, and an L-BFGS optimization adjusts coordinates until constraint errors are minimized to near zero. The resulting mechanism is intended to keep the visual diagram and symbolic constraints aligned after every edit.

The learning setup reported in the paper uses 2,000 symbolic-curated multi-step trajectories for supervised fine-tuning and then reinforcement learning. The RL initialization is the SFT checkpoint of Qwen2.5-VL-7B, and the optimization algorithm is Group Relative Policy Optimization (GRPO). The reward used in the main text is sparse and binary at trajectory end:

$BC$7

where $BC$8 rewards strict adherence to the JSON schema and $BC$9 rewards exact agreement with the gold answer. An objective tension appears in the paper’s presentation: the abstract refers to “dense, symbolic rewards,” whereas the main text describes sparse end-of-trajectory rewards without theorem-validity or stepwise consistency terms.

5. Empirical performance and reported findings

The benchmark evaluates GPT-4o, Qwen2.5-VL-72B, and Qwen2.5-VL-7B as baselines, together with GeoSketch-Qwen2.5-VL-7B after supervised fine-tuning and after reinforcement learning. The reported total accuracies on the 390 problems are as follows (Weng et al., 26 Sep 2025).

Model/setting	Pure Run total accuracy (%)	GeoSketch total accuracy (%)
GPT-4o	33.08	49.74
Qwen2.5-VL-72B	43.08	58.71
Qwen2.5-VL-7B	32.56	48.46
GeoSketch-Qwen2.5-VL-7B-SFT	33.85	52.56
GeoSketch-Qwen2.5-VL-7B-RL	35.64	59.23
Human	—	86.14

The same pattern holds across answer types. GPT-4o improves from 24.88/50.00/30.86 to 44.28/59.26/50.62 on Numerical/Ratio/Descriptor problems, Qwen2.5-VL-72B improves from 38.81/49.07/45.68 to 57.71/56.48/64.20, and the RL-tuned 7B model reaches 57.21/60.19/62.96. The paper highlights that dynamic interaction substantially improves all models over Pure Run, and that the RL-tuned 7B model slightly surpasses the 72B baseline in the GeoSketch setting, at 59.23% versus 58.71%.

The paper also reports an efficiency comparison with Visual Sketchpad. Using declarative logic forms rather than code generation reduces average token usage from approximately 12,899 to 5,693 input tokens and from approximately 1,056 to 901 output tokens. This is presented as a scalability advantage of the framework attached to the benchmark.

Beyond the benchmark itself, the training procedure is reported to improve performance on standard static benchmarks: Geometry3K reaches 28.79% versus a 19.30% base, and GeoQA reaches 72.52% versus a 47.28% base. A plausible implication is that the benchmark’s interaction model regularizes reasoning in ways that transfer beyond the 390-problem evaluation set, although the paper frames these numbers as empirical observations rather than as a general theorem about transfer.

6. Relation to prior geometry benchmarks and to GeoBuildBench

GeoSketch Benchmark is positioned against two earlier evaluation patterns. First, general multimodal math benchmarks include few geometry items and do not evaluate the temporal evolution of a solution sketch. Second, even geometry-focused work often requires manual diagram correction or treats sketching as a monolithic artifact rather than an executable, stepwise action space. GeoSketch Benchmark departs from both by centering evaluation on diagram edits that alter a structured logic form and are checked by a solver (Weng et al., 26 Sep 2025).

It is also useful to distinguish GeoSketch Benchmark from GeoBuildBench, which evaluates whether an agent can ground informal natural-language plane geometry problems into executable geometric constructions in a domain-specific language. GeoBuildBench contains 489 Chinese textbook-style plane-geometry problems, retains only text-complete and constructible instances, and evaluates whether a generated DSL program executes, creates all required geometric objects, and satisfies explicit machine-checkable constraints. Its metrics include success rate, average success steps, structural hallucinations per problem, missing-object counts, and failed-constraint counts (Kim et al., 13 May 2026).

The two benchmarks therefore target different regimes. GeoSketch Benchmark requires images and focuses on solving problems through dynamic manipulation of a given figure, with final answer accuracy as the headline metric. GeoBuildBench, by contrast, treats geometry diagram creation itself as the task and measures semantic validity of the construction program. This suggests complementary coverage: GeoSketch Benchmark stresses interactive multimodal problem solving with auxiliary constructions and transformations, while GeoBuildBench stresses grounded executable construction from language alone. The GeoBuildBench paper explicitly notes that, if one imagines a “GeoSketch Benchmark,” its own executable, constraint-verifiable core supplies a rigorous way to test sketching agents and diagnose failures such as missing objects or false parallelism (Kim et al., 13 May 2026).

7. Limitations, reproducibility, and open directions

Several limitations are explicit. The official evaluation focuses on final answer accuracy and does not provide an official stepwise verification metric, even though gold trajectories are part of the released artifact. The benchmark emphasizes auxiliary constructions and isometries—reflection, rotation, and translation—while scaling, shear, and general affine tasks are not enumerated in the problem set. Circle geometry is present, but not exhaustively categorized, and exact counts per auxiliary subtype are not reported (Weng et al., 26 Sep 2025).

The benchmark’s symbolic verification reduces but does not eliminate geometric fragility. The GCS plus L-BFGS auto-correction is intended to maintain consistent symbolic–visual states, yet broader generalization to complex or noisy real-world diagrams remains a challenge. This matters because the benchmark’s curated diagrams are designed for solver-mediated interaction rather than for unconstrained visual noise.

The reproducibility statement commits to releasing the GeoSketch Benchmark, the 2,000-trajectory fine-tuning dataset, code for perception components including YOLOv11, U-Net, and OCR, the reasoning prompts, the sketch action executor, the GCS, and pretrained weights for the SFT and RL variants. Reported toolchains include CUDA 12.4, 4×A800 GPUs for training, vLLM 0.10.2 for deployment, a matplotlib-based renderer, and L-BFGS optimization in the GCS. At the same time, specific repository links and license terms are not provided in the paper.

The benchmark’s future direction is correspondingly clear. Because it already supplies structured JSON, gold answers, and gold trajectories, it supports a natural extension from answer-level evaluation toward explicit stepwise correctness, theorem-application validity, and richer trajectory metrics. The paper presents that extension as planned work rather than as a completed part of the benchmark.