---
title: 'PyGeoX-Bench: Geometric Synthesis Benchmark'
url: https://www.emergentmind.com/topics/pygeox-bench
type: topic
---

# PyGeoX-Bench: Geometric Synthesis Benchmark

Searching arXiv for recent papers on PyGeoX-Bench and closely related geometric/geospatial benchmarks.
PyGeoX-Bench is a benchmark for precision-critical geometric synthesis built atop the PyGeoX symbolic–numeric geometry engine and domain-specific language (DSL). It evaluates whether a model can translate free-form natural-language descriptions of Euclidean configurations into executable Python programs that define numeric coordinates and circle radii satisfying all stated geometric constraints simultaneously. In the PyGeoX framework, benchmark verification is solver-grounded rather than answer-key-based: each generated construction is checked by compiling the target problem into per-constraint residuals, and a problem is counted as solved when the resulting residual vector satisfies the threshold $\|\mathbf{r}\|_2^2 < 10^{-3}$ [2606.09278]. PyGeoX-Bench was introduced together with PyGeoX in the study of solver-residual learning for geometric generation, where it serves both as an offline evaluation suite and as a reward-bearing environment for reinforcement learning with verifiable rewards [2606.09278].

## 1. Conceptual foundations

PyGeoX-Bench is inseparable from PyGeoX itself, a programmable 2D geometry environment designed for geometric constraint solving. PyGeoX exposes a declarative interface in which users or models specify geometric objects, relations, and scalar constraints through operations such as `scene.add`, `scene.relate`, and `scene.constraint`. These declarations are compiled into symbolic equalities and inequalities over free geometric variables $\mathbf{u}$, then aggregated into a differentiable objective function [2606.09278].

The underlying objective is defined as
$$
\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2
+ \beta \sum_j \max(0, -G_j(\mathbf{u}))^2
+ \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2
+ \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},
$$
with default $\alpha=\beta=\gamma=1.0$ and $\epsilon=10^{-4}$ [2606.09278]. In practical terms, this means that geometric validity is reduced to constraint satisfaction under a unified symbolic–numeric formalism.

PyGeoX-Bench uses this machinery for verification rather than interactive assistance. A model is not allowed to call PyGeoX during generation. Instead, the benchmark supplies a natural-language geometry problem, receives a Python program as output, executes that program, and then uses PyGeoX to reconstruct the intended constraint system and evaluate the candidate coordinates. This yields a residual vector
$$
\mathbf{r} = [r_1,\dots,r_C], \quad r_i \ge 0,
$$
where each $r_i$ measures the violation magnitude of one constraint [2606.09278]. A plausible implication is that PyGeoX-Bench operationalizes geometric reasoning as a synthesis-and-verification problem rather than a theorem-selection or multiple-choice task.

This design distinguishes PyGeoX-Bench from hierarchical diagnostic benchmarks such as GeoBench, which decompose geometric reasoning into perception, planning, theorem application, and backtracking tasks [2512.24119]. PyGeoX-Bench instead evaluates end-to-end constructive competence: the model must generate coordinates or radii that satisfy a coupled system of constraints, not merely identify correct premises or theorems [2606.09278].

## 2. Task formulation and benchmark composition

PyGeoX-Bench contains 300 problems stratified into three difficulty tiers: 100 Easy, 100 Medium, and 100 Hard [2606.09278]. Each problem is an open-ended geometric synthesis task described in natural language. The description names points and composite objects, specifies relations such as collinearity, perpendicularity, intersection, circumcircle or incircle conditions, and may impose metric or containment constraints [2606.09278].

The required output is a Python program that defines a dictionary `points` and a dictionary `circles`, where the former contains numeric coordinates for required points and the latter contains numeric radii for required circles [2606.09278]. The benchmark does not prescribe a solver strategy. Models may use constructive derivations, algebraic reductions, numerical optimization, or hybrids thereof, provided that the executed program produces a valid construction under PyGeoX verification [2606.09278].

The tiering reflects increasing combinatorial and geometric complexity.

| Tier | Problems | Typical scale |
|---|---:|---|
| Easy | 100 | mostly single-polygon configurations, roughly $\sim 13$ objects and $\sim 8$ constraints |
| Medium | 100 | two polygons, roughly $\sim 15$ objects and $\sim 10$ constraints |
| Hard | 100 | three polygons, roughly $\sim 23$ objects and $\sim 16$ constraints |

The paper notes that the number “16” for Hard is representative rather than a hard bound, but the Hard tier systematically involves dozens of interacting constraints [2606.09278]. This matters because benchmark difficulty does not arise only from longer textual descriptions; it arises from simultaneous satisfaction of many partially dependent geometric conditions.

The language of the tasks spans the full PyGeoX vocabulary. Objects include points, lines, line segments, rays, circles, arcs, angles, polygons, and specialized polygon classes such as `Square`, `Rectangle`, `Rhombus`, `Trapezoid`, `IsoscelesTrapezoid`, `RegularPentagon`, and `RegularOctagon` [2606.09278]. Relations include incidence, collinearity, orthogonality, parallelism, tangency, special points such as midpoints or orthocenters, circumcircle or incircle relations, containment, and transformations such as rotation or mirroring [2606.09278]. Property constraints support comparisons $\text{eq}, \text{neq}, \text{gt}, \text{lt}, \text{geq}, \text{leq}$ between symbolic scalar expressions such as lengths, angles, areas, or perimeters [2606.09278].

This broad object–relation space makes PyGeoX-Bench closer to a geometric constraint solving environment than to conventional geometry QA. By contrast, GeoBench emphasizes formally verified multimodal question answering over six task types derived from reasoning graphs, while PyGeoX-Bench asks for executable constructions whose correctness is numerically and symbolically verified [2512.24119; 2606.09278].

## 3. Verification formalism and evaluation protocol

The benchmark’s evaluation loop is single-shot. For each problem, the model receives a natural-language prompt and a fixed system instruction telling it to act as a “computational geometer and Python algorithmist” and to output Python code defining `points` and `circles` [2606.09278]. The generated code is executed in a sandbox with standard numerical libraries including NumPy, SciPy, and SymPy, under a 90-second time limit [2606.09278].

If execution fails because of exceptions, timeouts, or missing required variables, the attempt is treated as failure [2606.09278]. Otherwise, PyGeoX substitutes the supplied numeric values into the precompiled residual functions for that problem and computes the residual vector $\mathbf{r}$ [2606.09278].

The benchmark’s principal success criterion is binary:
$$
\|\mathbf{r}\|_2^2 < 10^{-3}.
$$
A problem satisfying this threshold is fully solved [2606.09278]. The primary metric is solving rate,
$$
\text{SR} = \frac{\text{\# problems with } \|\mathbf{r}\|_2^2 < 10^{-3}}{\text{total problems}},
$$
reported separately for Easy, Medium, and Hard tiers [2606.09278].

Several aspects of this protocol are notable. First, correctness is not based on exact symbolic matching to a reference derivation. Any program is acceptable if it yields coordinates satisfying the declared constraints. Second, verification is per-constraint and continuous prior to being collapsed into a binary solved/unsolved outcome, which enables richer reward shaping during training [2606.09278]. Third, the benchmark is execution-based: syntax, control flow, and numeric stability affect outcomes because the model must produce runnable code, not just a declarative sketch [2606.09278].

A plausible implication is that PyGeoX-Bench evaluates a compound capability involving language understanding, constructive planning, numerical reasoning, and program synthesis. That makes it complementary to benchmarks such as GeoBench, where theorem selection and sub-goal decomposition are explicit tasks, and to code-oriented geospatial benchmarks such as GeoAnalystBench, where workflow validity and CodeBLEU matter more than numerical constraint satisfaction [2512.24119; 2509.05881].

## 4. Residual-based rewards and Outlier Gradient Masking

A defining contribution associated with PyGeoX-Bench is the analysis of how solver residuals should be converted into rewards for learning. The benchmark exposes per-constraint residuals, which permits both global-norm rewards and decomposed per-constraint rewards [2606.09278].

A natural baseline is a global-norm reward based on sum of squared errors,
$$
\text{SSE} = \|\mathbf{r}\|_2^2 = \sum_i r_i^2,
$$
with a decreasing kernel such as
$$
\mathcal{R}_G(\mathbf{r}) = \exp\!\left(-\frac{\|\mathbf{r}\|_2^2}{T_{\text{mse}}}\right),
$$
where $T_{\text{mse}}=10$ in the reported experiments [2606.09278]. The paper argues that such rewards suffer from a failure mode termed **Outlier Gradient Masking**: a single severely violated constraint can drive the global reward close to zero, thereby suppressing the policy gradient associated with all other, partially improved constraints [2606.09278].

Two theoretical results support this claim. The first states that for global-norm rewards, the effective reward volume shrinks to zero as the number of constraints grows, whereas for additive per-constraint rewards the effective reward volume tends to occupy almost all of the residual space [2606.09278]. The second shows that gradients under global norms are coupled to the total error magnitude $L=\|\mathbf{r}\|_p$, while gradients under additive rewards remain local to individual residuals [2606.09278].

Empirically, on 3,893 partial solutions, the paper reports that under Saturating Additive Rewards, 97.1% of samples fall in a useful reward range $[0.1,0.9]$, whereas with the MSE-based reward about 60% collapse near 0 and another 20.6% near 1, leaving only 19% in a discriminative range [2606.09278]. This suggests that PyGeoX-Bench is not merely a test set; it is also a testbed for studying reward informativeness in multi-constraint environments.

The proposed reward family is **Saturating Additive Rewards (SAR)**:
$$
\mathcal{R}_{\text{SAR}}(\mathbf{r}) = \sum_{i=1}^C \phi(r_i),
$$
where $\phi$ is monotone decreasing and bounded, with the experiments using the Boltzmann kernel
$$
\phi(r) = e^{-r/T}, \quad T=0.1.
$$
To make the task precision-critical, SAR is combined with a sparse success bonus and a degeneracy penalty, yielding
$$
\mathcal{R} =
\underbrace{\frac{w}{C}\sum_{i=1}^C e^{-r_i/T}}_{\text{Dense Shaping (SAR)}}
+ \underbrace{\mathbb{I}_{\text{suc}} \cdot R_{\text{bonus}}}_{\text{Sparse Bonus}}
- \underbrace{\min(4, C_{\text{deg}})}_{\text{Degeneracy Penalty}},
$$
with $w=6.0$, $T=0.1$, and $R_{\text{bonus}}=4.0$ [2606.09278].

In this respect, PyGeoX-Bench differs sharply from geometry benchmarks oriented around final-answer multiple-choice accuracy. GeoBench, for example, diagnoses sub-goal decomposition or theorem selection via accuracy over distractor choices [2512.24119], whereas PyGeoX-Bench exposes dense solver-grounded reward structure at the level of individual constraints [2606.09278].

## 5. Experimental results and benchmark difficulty

PyGeoX-Bench was used to evaluate both base and trained models, with a strong focus on Qwen3-8B under supervised fine-tuning and reinforcement learning [2606.09278]. The benchmark’s core stress test is the Hard tier.

The reported Hard-tier solving rates for Qwen3-8B are as follows.

| Training condition | Hard SR |
|---|---:|
| Base model | 0.18 |
| SFT, SAR only | 0.23 |
| SFT, Sparse only | 0.23 |
| SFT, MSE only | 0.22 |
| SFT, SAR+S+D | **0.32** |
| SFT, MSE+S+D | **0.04** |
| RL, Sparse only | 0.35 |
| RL, SAR only | 0.09 |
| RL, MSE only | 0.10 |
| RL, MSE+S+D | 0.18 |
| RL, SAR+S+D | **0.41** |

The comparison between RL with SAR+S+D and RL with MSE+S+D is central: the Hard-tier solving rate improves from 0.18 to 0.41, a $2.3\times$ gain [2606.09278]. The paper also reports that on Hard-tier SFT, SAR+S+D reaches 0.32 whereas MSE+S+D drops to 0.04, described as catastrophic [2606.09278]. These results are presented as empirical evidence that dense global-norm rewards can be actively harmful in high-constraint synthesis tasks.

On the out-of-distribution PyGeoX-Wild set of 86 human-written problems, the same pattern persists: SAR+S+D under RL reaches 0.66 versus 0.59 for Sparse and 0.60 for MSE+S+D [2606.09278]. Frontier proprietary models evaluated zero-shot achieve Hard SR 0.51 at best, while the authors’ 8B RL model reaches 0.41 [2606.09278]. This indicates that the benchmark is non-trivial even for strong large-scale systems.

The paper further notes that the SAR+S+D-trained RL model uses approximately 22.8% fewer tokens on average than a sparse-trained RL model on PyGeoX-Bench [2606.09278]. This suggests that reward shaping affects not only solution quality but also generation efficiency.

A plausible implication is that PyGeoX-Bench measures an internalized constructive bias toward exact geometry rather than superficial pattern completion. The authors estimate the Hard-tier configuration space to exceed $10^{17}$, arguing that memorization is infeasible at that scale [2606.09278].

## 6. Relation to adjacent benchmark families

PyGeoX-Bench belongs to a broader 2025–2026 wave of geometry and geospatial benchmarks, but its problem definition is unusually specific. GeoBench evaluates multimodal geometric problem solving through hierarchical tasks including Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking [2512.24119]. Its tasks are multiple-choice and generated from formally verified reasoning graphs. PyGeoX-Bench, by contrast, does not decompose reasoning into subskills; it directly tests whether a model can synthesize a valid construction as executable code [2606.09278].

This contrast clarifies a common misconception: PyGeoX-Bench is not a textbook-style geometry QA benchmark, nor is it primarily a proof benchmark. It is a geometric generation benchmark. Its central object is a constraint-satisfying scene, not a theorem statement or numeric answer [2606.09278].

Another nearby family concerns geospatial AI rather than Euclidean geometry. GEO-Bench-2 focuses on evaluation protocols for geospatial foundation models across classification, segmentation, regression, detection, and instance segmentation, introducing “capability groups” to summarize strengths across modalities and resolutions [2511.15658]. GeoX-Bench targets cross-view geo-localization and pose estimation for large multimodal models using panoramic and satellite imagery [2511.13259]. GeoAnalystBench and GeoBenchX evaluate LLMs on geospatial workflow and tool-use tasks, respectively, emphasizing GIS reasoning, code generation, and hallucination rejection [2509.05881; 2503.18129]. These are methodologically relevant as examples of benchmark design, but they do not share PyGeoX-Bench’s solver-residual verification structure.

The naming similarity between PyGeoX-Bench and GeoX-Bench can itself be misleading. GeoX-Bench concerns cross-view geo-localization and pose estimation over real-world imagery [2511.13259], whereas PyGeoX-Bench concerns synthetic Euclidean geometric synthesis from natural language [2606.09278]. The shared “GeoX” string reflects distinct research agendas.

A plausible implication is that PyGeoX-Bench occupies a niche between symbolic theorem proving and program synthesis: closer to CAD-like construction validity than to standard mathematical QA. The paper explicitly points to future extensions toward 3D CAD, kinematics, and other precision-critical domains [2606.09278].

## 7. Release, extensibility, and limitations

PyGeoX-Bench is released as part of the PyGeoX GitHub repository, together with the PyGeoX engine, a 100k-problem training corpus and its generation code, PyGeoX-Wild, training scripts and configurations for supervised fine-tuning and reinforcement learning, and the evaluation harness used in the paper [2606.09278]. The benchmark is described as procedurally generated and extensible: new problems can be created by sampling different combinations of object and relation types and then solving and validating them with PyGeoX [2606.09278].

This extensibility is substantial because difficulty can be modulated by changing the number of objects, the number and type of constraints, and the degeneracy filters applied during generation [2606.09278]. In benchmark-design terms, this makes PyGeoX-Bench closer to a benchmark generator than to a fixed finite dataset.

The main limitations acknowledged in the paper are that the engine and benchmark are currently restricted to 2D static geometry, that experiments concentrate on a single strong base model family, and that PyGeoX-Wild is modest in size at 86 problems [2606.09278]. The authors suggest extending the framework to 3D and dynamic or kinematic constraints and exploring SAR-style reward shaping in other multi-constraint RLVR domains such as physics environments, robotic control, and scene layout [2606.09278].

These limitations are informative. They suggest that PyGeoX-Bench should not be interpreted as a general geometry benchmark in the educational sense, nor as a complete test of theorem-proving competence. Rather, it is a benchmark for a specific but demanding form of geometric intelligence: mapping free-form language to numerically exact, executable geometric constructions under verifiable multi-constraint semantics [2606.09278].

In that role, PyGeoX-Bench provides a distinctive research instrument. It couples declarative geometry, executable synthesis, dense residual analysis, and solver-grounded evaluation in a single framework. This combination makes it particularly valuable for studying how large language models can internalize geometric law when correctness is determined not by plausibility or style, but by strict constraint satisfaction [2606.09278].

Source: https://www.emergentmind.com/topics/pygeox-bench