GeoSketch: Interactive Geometric Problem Solving

• GeoSketch is a neural-symbolic framework for geometric reasoning that recasts static diagram interpretation into an interactive perception–reasoning–action loop.
• It integrates a perception module that converts diagrams into symbolic logic with a reasoning module that applies geometric theorems and executes diagram manipulations.
• The framework employs supervised fine-tuning and reinforcement learning to enhance geometric problem solving, outperforming static approaches on benchmark tests.

Searching arXiv for the named GeoSketch paper and closely related sketch-based geometric reasoning work. GeoSketch is a neural-symbolic framework for geometric multimodal reasoning that recasts geometric problem solving from static image interpretation into an interactive perception–reasoning–action loop. In this formulation, a system does not merely read a geometry diagram and produce text; it abstracts the diagram into a structured logic form, applies symbolic reasoning to decide the next deductive step, and executes diagram edits such as auxiliary line construction or affine transformations, thereby updating the diagram in a closed loop (Weng et al., 26 Sep 2025). The framework is motivated by the observation that many geometry problems are not solvable by passive inspection alone, because human solvers routinely draw extra lines, extend structures, and apply transformations such as reflection, rotation, and translation to expose hidden relations. GeoSketch formalizes these operations as executable actions within a multimodal agent architecture and evaluates them on a dedicated benchmark of geometry problems requiring such interventions (Weng et al., 26 Sep 2025).

1. Conceptual basis and problem setting

GeoSketch studies Geometric Problem Solving (GPS) in a multimodal setting where the input consists of both a natural-language problem statement and a geometry diagram (Weng et al., 26 Sep 2025). The central claim is that existing approaches process diagrams as static images and therefore lack the capacity for dynamic manipulation, even though auxiliary line construction and affine transformations are a core aspect of human geometric reasoning (Weng et al., 26 Sep 2025).

Within GeoSketch, the task is not framed as one-shot visual question answering. Instead, it is organized as a closed-loop hierarchical agent with three modules: a Perception module, a Symbolic Reasoning module, and a Sketch Action module (Weng et al., 26 Sep 2025). This design treats geometry solving as a stateful interaction process in which the diagram itself can change over time. A plausible implication is that GeoSketch shifts multimodal reasoning from static interpretation toward executable, verifiable interaction, but the paper’s explicit emphasis remains on auxiliary constructions, affine-like transformations, and symbolic verification (Weng et al., 26 Sep 2025).

The motivation aligns with a broader pattern in sketch-based systems: static perception is often insufficient when downstream tasks require geometric manipulation rather than recognition alone. Related work on sketch-guided deformation for 3D Gaussian Splatting likewise treats a user sketch as a source of explicit geometric control rather than merely semantic input (Xie et al., 2024). Similarly, systems such as GroundUp show that heterogeneous sketches can be combined to infer 3D structure when the workflow is designed around complementary geometric roles rather than a single passive rendering (Unlu et al., 2024). In GeoSketch, however, the target is not 3D reconstruction or modeling but iterative diagrammatic reasoning over Euclidean geometry problems (Weng et al., 26 Sep 2025).

2. Architecture and closed-loop operation

GeoSketch integrates three components: “(1) a Perception module that abstracts diagrams into structured logic forms, (2) a Symbolic Reasoning module that applies geometric theorems to decide the next deductive step, and (3) a Sketch Action module that executes operations such as drawing auxiliary lines or applying transformations, thereby updating the diagram in a closed loop” (Weng et al., 26 Sep 2025). The loop repeats until the model outputs a final answer or reaches a predefined step limit (Weng et al., 26 Sep 2025).

The end-to-end flow begins with the geometry problem text and the initial diagram image. The Perception module detects visual primitives, parses labels with OCR, and converts the visual scene into a symbolic JSON-based logic form (Weng et al., 26 Sep 2025). The Reasoning module then receives the problem text, the current logic form, and the history of prior reasoning and actions, and predicts the next reasoning step together with an executable action (Weng et al., 26 Sep 2025). The Sketch Action executor applies the action to the logic form, updates coordinates or object lists, re-renders the diagram, and feeds the updated state back into the next perception–reasoning cycle (Weng et al., 26 Sep 2025).

The paper conceptually defines the current logic form as the state, the generated JSON command as the action, and the environment as the GeoSketch architecture, but it does not provide explicit transition equations such as $s_{t+1}=T(s_t,a_t)$ or policy formulas such as $a_t \sim \pi_\theta(\cdot \mid s_t)$ (Weng et al., 26 Sep 2025). This should be noted because the framework is interactive and sequential in practice, yet only partially formalized mathematically in the manuscript. A plausible implication is that GeoSketch is best understood as a systems architecture with executable symbolic state, rather than as a fully specified MDP formalism.

This architecture differs from other sketch-based systems in an important respect. Query-by-sketch systems such as “Query by Semantic Sketch” encode user intent into a static retrieval vector over semantic layouts (Rossetto et al., 2019), whereas GeoSketch updates the visual workspace itself after each action (Weng et al., 26 Sep 2025). The distinction is fundamental: the sketch in GeoSketch is not the query alone, but also the medium of intermediate reasoning.

3. Perception and symbolic state representation

The Perception module converts the image into a structured symbolic representation suitable for theorem reasoning and diagram manipulation (Weng et al., 26 Sep 2025). It uses a fine-tuned YOLOv11 object detector, a fine-tuned U-Net segmentation model, and an OCR model (Weng et al., 26 Sep 2025). Together these produce a JSON-based logic form enumerating geometric objects, their properties, and their relations (Weng et al., 26 Sep 2025).

The paper gives example entries such as:

$a_t \sim \pi_\theta(\cdot \mid s_t)$5

and

$a_t \sim \pi_\theta(\cdot \mid s_t)$6

These examples illustrate the intended role of the logic form as a symbolic digital twin of the diagram (Weng et al., 26 Sep 2025). The paper states that the diagram can be re-rendered from this logic form “losslessly,” and this re-renderability is important because subsequent actions operate on symbolic geometry rather than on raw pixels (Weng et al., 26 Sep 2025).

A practical feature of the perception pipeline is an automatic correction mechanism. After parsing the diagram into a logic form, GeoSketch re-renders the diagram from that logic form, uses MLLMs to compare the rendered image against the original, and refines the logic form until they match (Weng et al., 26 Sep 2025). In the appendix, this is strengthened by a geometry constraint solver: if points and lines do not satisfy the intended geometric relations, the solver produces a positive error, and L-BFGS is used to optimize this error to zero (Weng et al., 26 Sep 2025). The exact analytical form of the error is not given.

This symbolic abstraction places GeoSketch in a lineage closer to systems that seek geometric understanding of sketches rather than pixel-level pattern recognition. In “Geometric Understanding of Sketches,” raster line drawings are converted into explicit planar graphs $G(V,E)$ for downstream robotic reproduction (Venkataramaiyer, 2022). GeoSketch adopts a different symbolic target, but the shared systems lesson is that sketches become computationally powerful when they are converted into editable structural representations rather than left as images.

4. Symbolic reasoning and executable actions

The Symbolic Reasoning module is described as the theorem-proving core of GeoSketch (Weng et al., 26 Sep 2025). It receives the natural-language problem text, the current logic form, and the history of previous reasoning steps and actions, and produces a JSON output with two fields: "reasoning" and "action" (Weng et al., 26 Sep 2025). The first is a natural-language explanation of the current deductive step; the second is an executable command for the sketch module (Weng et al., 26 Sep 2025).

The paper characterizes this module as applying geometric theorems to decide the next deductive step, but it does not present a formal theorem library, a symbolic calculus, or a theorem-ranking objective (Weng et al., 26 Sep 2025). This matters because GeoSketch is explicitly symbolic in architecture yet only partially symbolic in mathematical exposition. A plausible implication is that theorem application is delegated to an MLLM conditioned on structured geometry state, rather than implemented as a fully separate hand-coded prover.

The executable action space is the operational novelty of the framework. GeoSketch explicitly supports:

• draw_line(point_i, point_j): construct an auxiliary line (Weng et al., 26 Sep 2025)
• reflect(object, line_of_symmetry): reflect an object about a line (Weng et al., 26 Sep 2025)
• rotate(object, center, degrees): rotate an object about a center by a specified angle (Weng et al., 26 Sep 2025)
• translate(object, vector): translate an object by a vector (Weng et al., 26 Sep 2025)
• label_point(name, coordinates): annotate a new point (Weng et al., 26 Sep 2025)

The paper repeatedly refers to affine transformations, but it does not provide exact matrix formulas such as $x' = Ax + b$ or explicit rotation/reflection matrices (Weng et al., 26 Sep 2025). The representation is therefore operational rather than algebraically developed.

The action executor updates the logic form, regenerates the corresponding image, and passes the new state back to the next cycle (Weng et al., 26 Sep 2025). This makes the diagram a mutable workspace rather than a fixed prompt. In this respect, GeoSketch resembles broader human-centered sketch systems in which sketch input is progressively refined or operationalized, but its action semantics are deductive rather than generative. For example, SketchFlex refines rough region sketches into stronger canny-based anchors to stabilize image generation (Lin et al., 11 Feb 2025), whereas GeoSketch refines logical state through auxiliary constructions and transformations to stabilize formal reasoning (Weng et al., 26 Sep 2025).

5. Training pipeline, benchmark, and empirical results

GeoSketch uses a two-stage training pipeline: supervised fine-tuning followed by reinforcement learning (Weng et al., 26 Sep 2025). The SFT stage uses 2,000 expert trajectories from a fine-tuning dataset generated by a neural-symbolic pipeline with geometric templates, handcrafted rules, and support from multiple advanced LMs including Gemini-2.5-pro, Qwen3, Seed1.6-Thinking, and DeepSeek-R1 (Weng et al., 26 Sep 2025). These trajectories are represented as multi-turn conversations in which each turn includes the history of previous auxiliary-line operations, newly generated geometric elements, and the current problem context, with target outputs consisting of a reasoning step and a JSON action (Weng et al., 26 Sep 2025).

The student model is Qwen2.5VL-7B, and SFT is framed as knowledge distillation from larger teacher models (Weng et al., 26 Sep 2025). Appendix hyperparameters specify that the ViT module is frozen and only the projector and LLM modules are fine-tuned, with learning rate $1\times 10^{-5}$ for both (Weng et al., 26 Sep 2025). The paper does not provide an explicit supervised loss formula.

The RL stage uses Group Relative Policy Optimization (GRPO) (Weng et al., 26 Sep 2025). The concrete reward formulation described in the methodology is sparse and rule-based, with two binary end-of-trajectory rewards: $R_{\text{format}}$, which equals $1.0$ if the final output strictly adheres to the predefined JSON schema and $0$ otherwise, and $R_{\text{result}}$, which equals $1.0$ if the final computed answer matches the gold-standard solution and $a_t \sim \pi_\theta(\cdot \mid s_t)$0 otherwise (Weng et al., 26 Sep 2025). The abstract describes “dense, symbolic rewards,” but the detailed method as provided specifies sparse terminal rewards; this is an internal inconsistency in the paper (Weng et al., 26 Sep 2025). During RL, the ViT remains frozen and the learning rate is $a_t \sim \pi_\theta(\cdot \mid s_t)$1 for both the LLM and projector (Weng et al., 26 Sep 2025).

The GeoSketch Benchmark contains 390 geometry problems requiring auxiliary constructions or affine transformations (Weng et al., 26 Sep 2025). Problems are curated from roughly 1,200 candidates and stored as structured JSON objects containing a unique identifier, textual description, path to the initial diagram image, path to the solution diagram image, final gold answer, and detailed step-by-step natural-language proof (Weng et al., 26 Sep 2025). The answer categories are numerical ($a_t \sim \pi_\theta(\cdot \mid s_t)$2), ratio ($a_t \sim \pi_\theta(\cdot \mid s_t)$3), and descriptor ($a_t \sim \pi_\theta(\cdot \mid s_t)$4) (Weng et al., 26 Sep 2025).

The main evaluation metric is final answer accuracy. GeoSketch improves all tested models over static “Pure Run” inference. The headline results are:

Model	Pure Run	GeoSketch
GPT-4o	33.08	49.74
Qwen2.5VL-72B	43.08	58.71
Qwen2.5VL-7B	32.56	48.46
GeoSketch-Qwen2.5VL-7B-SFT	33.85	52.56
GeoSketch-Qwen2.5VL-7B-RL	35.64	59.23
Human	—	86.14

The strongest reported model, GeoSketch-Qwen2.5VL-7B-RL, reaches 59.23% on the benchmark, outperforming Qwen2.5VL-72B at 58.71% and GPT-4o at 49.74% under the GeoSketch framework (Weng et al., 26 Sep 2025). On static geometry benchmarks, GeoSketch training also improves the base Qwen2.5VL-7B from 19.30 to 28.79 on Geometry3K and from 47.28 to 72.52 on GeoQA after RL (Weng et al., 26 Sep 2025). The paper further reports a token-efficiency comparison with Visual Sketchpad, reducing average input tokens from 12899.12 to 5692.71 and average output tokens from 1055.75 to 901.46 (Weng et al., 26 Sep 2025).

6. Significance, relation to adjacent work, and limitations

GeoSketch’s significance lies in the claim that multimodal reasoning over geometry problems benefits from an executable visual workspace grounded in symbolic state, rather than static image interpretation alone (Weng et al., 26 Sep 2025). This places it in contrast not only with static MLLM baselines but also with prior systems such as GeoGPT4V, G-LLAVA, and AutoGPS, which the paper characterizes as remaining essentially static or less centered on executable visual actions (Weng et al., 26 Sep 2025).

A broader research context suggests that GeoSketch belongs to a growing class of systems that treat sketches as operational constraints rather than passive depictions. In Geo-Sketcher, geologists sketch topographic and geological maps and the system turns these into 3D geological models by combining graph-based topological reasoning with Hermite-Birkhoff RBF interpolation (Amorim et al., 2023). In GroundUp, architects use top-down and perspective sketches to infer 3D city massing through occupancy-guided depth prediction and diffusion-based heightfield completion (Unlu et al., 2024). In Sketch2PQ, architectural line drawings are parsed into visible boundaries, occluded boundaries, contours, and feature lines to infer a B-spline surface and a conjugate direction field for planar quadrilateral meshing (Deng et al., 2022). These systems differ in domain and output, but they share a design principle with GeoSketch: meaningful sketch-based intelligence often requires explicit intermediate structure, geometric constraints, and iterative interaction rather than end-to-end image-only prediction.

GeoSketch also has clear limitations. The paper explicitly states that evaluation is based only on final answer accuracy, not on the correctness of intermediate reasoning steps (Weng et al., 26 Sep 2025). It does not provide module-wise ablations isolating the contribution of perception, reasoning, and action separately (Weng et al., 26 Sep 2025). The symbolic and RL formalizations are incomplete: no explicit SFT loss, no GRPO objective, no transition equations, and no affine transformation matrices are given (Weng et al., 26 Sep 2025). The reward presentation is inconsistent between the abstract and methodology (Weng et al., 26 Sep 2025). A plausible implication is that the framework’s empirical gains are clearer than its mathematical specification.

There are also broader open questions. The logic-form abstraction depends on accurate perception of diagrams, and the paper gives little quantitative error analysis for parser failures (Weng et al., 26 Sep 2025). The action set supports auxiliary lines and basic transformations, but not richer geometric operations such as circle construction from constraints, locus generation, or theorem-conditioned symbolic rewrites beyond the provided executor vocabulary. This suggests that GeoSketch is a strong proof of concept for dynamic geometric reasoning, but not yet a complete formal geometry environment.

Overall, GeoSketch establishes a distinct paradigm for multimodal geometry solving: perceive a diagram into logic, reason symbolically over that logic, act on the diagram through executable constructions and transformations, and repeat until the geometry problem is solved (Weng et al., 26 Sep 2025). Its most consequential contribution is not a new static perception model, but the unification of hierarchical decision-making, executable sketch actions, and symbolic verification in a closed loop.