---
title: 'GeoSketch: Interactive Geometric Problem Solving'
url: https://www.emergentmind.com/topics/geosketch
type: topic
---

# GeoSketch: Interactive Geometric Problem Solving

Searching arXiv for the named GeoSketch paper and closely related sketch-based geometric reasoning work.
GeoSketch is a neural-symbolic framework for geometric multimodal reasoning that recasts geometric problem solving from static image interpretation into an interactive perception–reasoning–action loop. In this formulation, a system does not merely read a geometry diagram and produce text; it abstracts the diagram into a structured logic form, applies symbolic reasoning to decide the next deductive step, and executes diagram edits such as auxiliary line construction or affine transformations, thereby updating the diagram in a closed loop [2509.22460]. The framework is motivated by the observation that many geometry problems are not solvable by passive inspection alone, because human solvers routinely draw extra lines, extend structures, and apply transformations such as reflection, rotation, and translation to expose hidden relations. GeoSketch formalizes these operations as executable actions within a multimodal agent architecture and evaluates them on a dedicated benchmark of geometry problems requiring such interventions [2509.22460].

## 1. Conceptual basis and problem setting

GeoSketch studies Geometric Problem Solving (GPS) in a multimodal setting where the input consists of both a natural-language problem statement and a geometry diagram [2509.22460]. The central claim is that existing approaches process diagrams as static images and therefore lack the capacity for dynamic manipulation, even though auxiliary line construction and affine transformations are a core aspect of human geometric reasoning [2509.22460].

Within GeoSketch, the task is not framed as one-shot visual question answering. Instead, it is organized as a closed-loop hierarchical agent with three modules: a Perception module, a Symbolic Reasoning module, and a Sketch Action module [2509.22460]. This design treats geometry solving as a stateful interaction process in which the diagram itself can change over time. A plausible implication is that GeoSketch shifts multimodal reasoning from static interpretation toward executable, verifiable interaction, but the paper’s explicit emphasis remains on auxiliary constructions, affine-like transformations, and symbolic verification [2509.22460].

The motivation aligns with a broader pattern in sketch-based systems: static perception is often insufficient when downstream tasks require geometric manipulation rather than recognition alone. Related work on sketch-guided deformation for 3D Gaussian Splatting likewise treats a user sketch as a source of explicit geometric control rather than merely semantic input [2411.12168]. Similarly, systems such as GroundUp show that heterogeneous sketches can be combined to infer 3D structure when the workflow is designed around complementary geometric roles rather than a single passive rendering [2407.12739]. In GeoSketch, however, the target is not 3D reconstruction or modeling but iterative diagrammatic reasoning over Euclidean geometry problems [2509.22460].

## 2. Architecture and closed-loop operation

GeoSketch integrates three components: “(1) a Perception module that abstracts diagrams into structured logic forms, (2) a Symbolic Reasoning module that applies geometric theorems to decide the next deductive step, and (3) a Sketch Action module that executes operations such as drawing auxiliary lines or applying transformations, thereby updating the diagram in a closed loop” [2509.22460]. The loop repeats until the model outputs a final answer or reaches a predefined step limit [2509.22460].

The end-to-end flow begins with the geometry problem text and the initial diagram image. The Perception module detects visual primitives, parses labels with OCR, and converts the visual scene into a symbolic JSON-based logic form [2509.22460]. The Reasoning module then receives the problem text, the current logic form, and the history of prior reasoning and actions, and predicts the next reasoning step together with an executable action [2509.22460]. The Sketch Action executor applies the action to the logic form, updates coordinates or object lists, re-renders the diagram, and feeds the updated state back into the next perception–reasoning cycle [2509.22460].

The paper conceptually defines the current logic form as the state, the generated JSON command as the action, and the environment as the GeoSketch architecture, but it does not provide explicit transition equations such as \(s_{t+1}=T(s_t,a_t)\) or policy formulas such as \(a_t \sim \pi_\theta(\cdot \mid s_t)\) [2509.22460]. This should be noted because the framework is interactive and sequential in practice, yet only partially formalized mathematically in the manuscript. A plausible implication is that GeoSketch is best understood as a systems architecture with executable symbolic state, rather than as a fully specified MDP formalism.

This architecture differs from other sketch-based systems in an important respect. Query-by-sketch systems such as “Query by Semantic Sketch” encode user intent into a static retrieval vector over semantic layouts [1909.12526], whereas GeoSketch updates the visual workspace itself after each action [2509.22460]. The distinction is fundamental: the sketch in GeoSketch is not the query alone, but also the medium of intermediate reasoning.

## 3. Perception and symbolic state representation

The Perception module converts the image into a structured symbolic representation suitable for theorem reasoning and diagram manipulation [2509.22460]. It uses a fine-tuned YOLOv11 object detector, a fine-tuned U-Net segmentation model, and an OCR model [2509.22460]. Together these produce a JSON-based logic form enumerating geometric objects, their properties, and their relations [2509.22460].

The paper gives example entries such as:

```json
{type: 'line', points: ['A', 'B']}
```

and

```json
{type: 'circle', center: 'O', radius: 5}
```

These examples illustrate the intended role of the logic form as a symbolic digital twin of the diagram [2509.22460]. The paper states that the diagram can be re-rendered from this logic form “losslessly,” and this re-renderability is important because subsequent actions operate on symbolic geometry rather than on raw pixels [2509.22460].

A practical feature of the perception pipeline is an automatic correction mechanism. After parsing the diagram into a logic form, GeoSketch re-renders the diagram from that logic form, uses MLLMs to compare the rendered image against the original, and refines the logic form until they match [2509.22460]. In the appendix, this is strengthened by a geometry constraint solver: if points and lines do not satisfy the intended geometric relations, the solver produces a positive error, and L-BFGS is used to optimize this error to zero [2509.22460]. The exact analytical form of the error is not given.

This symbolic abstraction places GeoSketch in a lineage closer to systems that seek geometric understanding of sketches rather than pixel-level pattern recognition. In “Geometric Understanding of Sketches,” raster line drawings are converted into explicit planar graphs \(G(V,E)\) for downstream robotic reproduction [2204.06675]. GeoSketch adopts a different symbolic target, but the shared systems lesson is that sketches become computationally powerful when they are converted into editable structural representations rather than left as images.

## 4. Symbolic reasoning and executable actions

The Symbolic Reasoning module is described as the theorem-proving core of GeoSketch [2509.22460]. It receives the natural-language problem text, the current logic form, and the history of previous reasoning steps and actions, and produces a JSON output with two fields: `"reasoning"` and `"action"` [2509.22460]. The first is a natural-language explanation of the current deductive step; the second is an executable command for the sketch module [2509.22460].

The paper characterizes this module as applying geometric theorems to decide the next deductive step, but it does not present a formal theorem library, a symbolic calculus, or a theorem-ranking objective [2509.22460]. This matters because GeoSketch is explicitly symbolic in architecture yet only partially symbolic in mathematical exposition. A plausible implication is that theorem application is delegated to an MLLM conditioned on structured geometry state, rather than implemented as a fully separate hand-coded prover.

The executable action space is the operational novelty of the framework. GeoSketch explicitly supports:

- **`draw_line(point_i, point_j)`**: construct an auxiliary line [2509.22460]
- **`reflect(object, line_of_symmetry)`**: reflect an object about a line [2509.22460]
- **`rotate(object, center, degrees)`**: rotate an object about a center by a specified angle [2509.22460]
- **`translate(object, vector)`**: translate an object by a vector [2509.22460]
- **`label_point(name, coordinates)`**: annotate a new point [2509.22460]

The paper repeatedly refers to affine transformations, but it does not provide exact matrix formulas such as \(x' = Ax + b\) or explicit rotation/reflection matrices [2509.22460]. The representation is therefore operational rather than algebraically developed.

The action executor updates the logic form, regenerates the corresponding image, and passes the new state back to the next cycle [2509.22460]. This makes the diagram a mutable workspace rather than a fixed prompt. In this respect, GeoSketch resembles broader human-centered sketch systems in which sketch input is progressively refined or operationalized, but its action semantics are deductive rather than generative. For example, SketchFlex refines rough region sketches into stronger canny-based anchors to stabilize image generation [2502.07556], whereas GeoSketch refines logical state through auxiliary constructions and transformations to stabilize formal reasoning [2509.22460].

## 5. Training pipeline, benchmark, and empirical results

GeoSketch uses a two-stage training pipeline: supervised fine-tuning followed by reinforcement learning [2509.22460]. The SFT stage uses 2,000 expert trajectories from a fine-tuning dataset generated by a neural-symbolic pipeline with geometric templates, handcrafted rules, and support from multiple advanced LMs including Gemini-2.5-pro, Qwen3, Seed1.6-Thinking, and DeepSeek-R1 [2509.22460]. These trajectories are represented as multi-turn conversations in which each turn includes the history of previous auxiliary-line operations, newly generated geometric elements, and the current problem context, with target outputs consisting of a reasoning step and a JSON action [2509.22460].

The student model is Qwen2.5VL-7B, and SFT is framed as knowledge distillation from larger teacher models [2509.22460]. Appendix hyperparameters specify that the ViT module is frozen and only the projector and LLM modules are fine-tuned, with learning rate \(1\times 10^{-5}\) for both [2509.22460]. The paper does not provide an explicit supervised loss formula.

The RL stage uses Group Relative Policy Optimization (GRPO) [2509.22460]. The concrete reward formulation described in the methodology is sparse and rule-based, with two binary end-of-trajectory rewards: \(R_{\text{format}}\), which equals \(1.0\) if the final output strictly adheres to the predefined JSON schema and \(0\) otherwise, and \(R_{\text{result}}\), which equals \(1.0\) if the final computed answer matches the gold-standard solution and \(0\) otherwise [2509.22460]. The abstract describes “dense, symbolic rewards,” but the detailed method as provided specifies sparse terminal rewards; this is an internal inconsistency in the paper [2509.22460]. During RL, the ViT remains frozen and the learning rate is \(1\times 10^{-6}\) for both the LLM and projector [2509.22460].

The GeoSketch Benchmark contains 390 geometry problems requiring auxiliary constructions or affine transformations [2509.22460]. Problems are curated from roughly 1,200 candidates and stored as structured JSON objects containing a unique identifier, textual description, path to the initial diagram image, path to the solution diagram image, final gold answer, and detailed step-by-step natural-language proof [2509.22460]. The answer categories are numerical (\(201\)), ratio (\(108\)), and descriptor (\(81\)) [2509.22460].

The main evaluation metric is final answer accuracy. GeoSketch improves all tested models over static “Pure Run” inference. The headline results are:

| Model | Pure Run | GeoSketch |
|---|---:|---:|
| GPT-4o | 33.08 | 49.74 |
| Qwen2.5VL-72B | 43.08 | 58.71 |
| Qwen2.5VL-7B | 32.56 | 48.46 |
| GeoSketch-Qwen2.5VL-7B-SFT | 33.85 | 52.56 |
| GeoSketch-Qwen2.5VL-7B-RL | 35.64 | 59.23 |
| Human | — | 86.14 |

The strongest reported model, GeoSketch-Qwen2.5VL-7B-RL, reaches 59.23% on the benchmark, outperforming Qwen2.5VL-72B at 58.71% and GPT-4o at 49.74% under the GeoSketch framework [2509.22460]. On static geometry benchmarks, GeoSketch training also improves the base Qwen2.5VL-7B from 19.30 to 28.79 on Geometry3K and from 47.28 to 72.52 on GeoQA after RL [2509.22460]. The paper further reports a token-efficiency comparison with Visual Sketchpad, reducing average input tokens from 12899.12 to 5692.71 and average output tokens from 1055.75 to 901.46 [2509.22460].

## 6. Significance, relation to adjacent work, and limitations

GeoSketch’s significance lies in the claim that multimodal reasoning over geometry problems benefits from an executable visual workspace grounded in symbolic state, rather than static image interpretation alone [2509.22460]. This places it in contrast not only with static MLLM baselines but also with prior systems such as GeoGPT4V, G-LLAVA, and AutoGPS, which the paper characterizes as remaining essentially static or less centered on executable visual actions [2509.22460].

A broader research context suggests that GeoSketch belongs to a growing class of systems that treat sketches as operational constraints rather than passive depictions. In Geo-Sketcher, geologists sketch topographic and geological maps and the system turns these into 3D geological models by combining graph-based topological reasoning with Hermite-Birkhoff RBF interpolation [2308.12152]. In GroundUp, architects use top-down and perspective sketches to infer 3D city massing through occupancy-guided depth prediction and diffusion-based heightfield completion [2407.12739]. In Sketch2PQ, architectural line drawings are parsed into visible boundaries, occluded boundaries, contours, and feature lines to infer a B-spline surface and a conjugate direction field for planar quadrilateral meshing [2201.09367]. These systems differ in domain and output, but they share a design principle with GeoSketch: meaningful sketch-based intelligence often requires explicit intermediate structure, geometric constraints, and iterative interaction rather than end-to-end image-only prediction.

GeoSketch also has clear limitations. The paper explicitly states that evaluation is based only on final answer accuracy, not on the correctness of intermediate reasoning steps [2509.22460]. It does not provide module-wise ablations isolating the contribution of perception, reasoning, and action separately [2509.22460]. The symbolic and RL formalizations are incomplete: no explicit SFT loss, no GRPO objective, no transition equations, and no affine transformation matrices are given [2509.22460]. The reward presentation is inconsistent between the abstract and methodology [2509.22460]. A plausible implication is that the framework’s empirical gains are clearer than its mathematical specification.

There are also broader open questions. The logic-form abstraction depends on accurate perception of diagrams, and the paper gives little quantitative error analysis for parser failures [2509.22460]. The action set supports auxiliary lines and basic transformations, but not richer geometric operations such as circle construction from constraints, locus generation, or theorem-conditioned symbolic rewrites beyond the provided executor vocabulary. This suggests that GeoSketch is a strong proof of concept for dynamic geometric reasoning, but not yet a complete formal geometry environment.

Overall, GeoSketch establishes a distinct paradigm for multimodal geometry solving: perceive a diagram into logic, reason symbolically over that logic, act on the diagram through executable constructions and transformations, and repeat until the geometry problem is solved [2509.22460]. Its most consequential contribution is not a new static perception model, but the unification of hierarchical decision-making, executable sketch actions, and symbolic verification in a closed loop.

Source: https://www.emergentmind.com/topics/geosketch