PyGeoX-Bench: Geometric Synthesis Benchmark
- PyGeoX-Bench is a benchmark that tests models on converting natural language descriptions into executable Python programs generating valid Euclidean geometric configurations.
- It employs a solver-grounded verification process that evaluates constructions based on a residual vector threshold, enabling precise quantification of constraint satisfaction.
- The benchmark offers 300 stratified problems across Easy, Medium, and Hard tiers, challenging models with complex constraints and diverse geometric relations.
Searching arXiv for papers on PyGeoX-Bench and closely related geometric/geospatial benchmarks. PyGeoX-Bench is a benchmark for precision-critical geometric synthesis built atop the PyGeoX symbolic–numeric geometry engine and domain-specific language (DSL). It evaluates whether a model can translate free-form natural-language descriptions of Euclidean configurations into executable Python programs that define numeric coordinates and circle radii satisfying all stated geometric constraints simultaneously. In the PyGeoX framework, benchmark verification is solver-grounded rather than answer-key-based: each generated construction is checked by compiling the target problem into per-constraint residuals, and a problem is counted as solved when the resulting residual vector satisfies the threshold (Cabral et al., 8 Jun 2026). PyGeoX-Bench was introduced together with PyGeoX in the study of solver-residual learning for geometric generation, where it serves both as an offline evaluation suite and as a reward-bearing environment for reinforcement learning with verifiable rewards (Cabral et al., 8 Jun 2026).
1. Conceptual foundations
PyGeoX-Bench is inseparable from PyGeoX itself, a programmable 2D geometry environment designed for geometric constraint solving. PyGeoX exposes a declarative interface in which users or models specify geometric objects, relations, and scalar constraints through operations such as scene.add, scene.relate, and scene.constraint. These declarations are compiled into symbolic equalities and inequalities over free geometric variables , then aggregated into a differentiable objective function (Cabral et al., 8 Jun 2026).
The underlying objective is defined as
$\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$
with default and (Cabral et al., 8 Jun 2026). In practical terms, this means that geometric validity is reduced to constraint satisfaction under a unified symbolic–numeric formalism.
PyGeoX-Bench uses this machinery for verification rather than interactive assistance. A model is not allowed to call PyGeoX during generation. Instead, the benchmark supplies a natural-language geometry problem, receives a Python program as output, executes that program, and then uses PyGeoX to reconstruct the intended constraint system and evaluate the candidate coordinates. This yields a residual vector
where each measures the violation magnitude of one constraint (Cabral et al., 8 Jun 2026). A plausible implication is that PyGeoX-Bench operationalizes geometric reasoning as an overview-and-verification problem rather than a theorem-selection or multiple-choice task.
This design distinguishes PyGeoX-Bench from hierarchical diagnostic benchmarks such as GeoBench, which decompose geometric reasoning into perception, planning, theorem application, and backtracking tasks (Feng et al., 30 Dec 2025). PyGeoX-Bench instead evaluates end-to-end constructive competence: the model must generate coordinates or radii that satisfy a coupled system of constraints, not merely identify correct premises or theorems (Cabral et al., 8 Jun 2026).
2. Task formulation and benchmark composition
PyGeoX-Bench contains 300 problems stratified into three difficulty tiers: 100 Easy, 100 Medium, and 100 Hard (Cabral et al., 8 Jun 2026). Each problem is an open-ended geometric synthesis task described in natural language. The description names points and composite objects, specifies relations such as collinearity, perpendicularity, intersection, circumcircle or incircle conditions, and may impose metric or containment constraints (Cabral et al., 8 Jun 2026).
The required output is a Python program that defines a dictionary points and a dictionary circles, where the former contains numeric coordinates for required points and the latter contains numeric radii for required circles (Cabral et al., 8 Jun 2026). The benchmark does not prescribe a solver strategy. Models may use constructive derivations, algebraic reductions, numerical optimization, or hybrids thereof, provided that the executed program produces a valid construction under PyGeoX verification (Cabral et al., 8 Jun 2026).
The tiering reflects increasing combinatorial and geometric complexity.
| Tier | Problems | Typical scale |
|---|---|---|
| Easy | 100 | mostly single-polygon configurations, roughly objects and constraints |
| Medium | 100 | two polygons, roughly objects and 0 constraints |
| Hard | 100 | three polygons, roughly 1 objects and 2 constraints |
The paper notes that the number “16” for Hard is representative rather than a hard bound, but the Hard tier systematically involves dozens of interacting constraints (Cabral et al., 8 Jun 2026). This matters because benchmark difficulty does not arise only from longer textual descriptions; it arises from simultaneous satisfaction of many partially dependent geometric conditions.
The language of the tasks spans the full PyGeoX vocabulary. Objects include points, lines, line segments, rays, circles, arcs, angles, polygons, and specialized polygon classes such as Square, Rectangle, Rhombus, Trapezoid, IsoscelesTrapezoid, RegularPentagon, and RegularOctagon (Cabral et al., 8 Jun 2026). Relations include incidence, collinearity, orthogonality, parallelism, tangency, special points such as midpoints or orthocenters, circumcircle or incircle relations, containment, and transformations such as rotation or mirroring (Cabral et al., 8 Jun 2026). Property constraints support comparisons 3 between symbolic scalar expressions such as lengths, angles, areas, or perimeters (Cabral et al., 8 Jun 2026).
This broad object–relation space makes PyGeoX-Bench closer to a geometric constraint solving environment than to conventional geometry QA. By contrast, GeoBench emphasizes formally verified multimodal question answering over six task types derived from reasoning graphs, while PyGeoX-Bench asks for executable constructions whose correctness is numerically and symbolically verified (Feng et al., 30 Dec 2025, Cabral et al., 8 Jun 2026).
3. Verification formalism and evaluation protocol
The benchmark’s evaluation loop is single-shot. For each problem, the model receives a natural-language prompt and a fixed system instruction telling it to act as a “computational geometer and Python algorithmist” and to output Python code defining points and circles (Cabral et al., 8 Jun 2026). The generated code is executed in a sandbox with standard numerical libraries including NumPy, SciPy, and SymPy, under a 90-second time limit (Cabral et al., 8 Jun 2026).
If execution fails because of exceptions, timeouts, or missing required variables, the attempt is treated as failure (Cabral et al., 8 Jun 2026). Otherwise, PyGeoX substitutes the supplied numeric values into the precompiled residual functions for that problem and computes the residual vector 4 (Cabral et al., 8 Jun 2026).
The benchmark’s principal success criterion is binary:
5
A problem satisfying this threshold is fully solved (Cabral et al., 8 Jun 2026). The primary metric is solving rate,
6
reported separately for Easy, Medium, and Hard tiers (Cabral et al., 8 Jun 2026).
Several aspects of this protocol are notable. First, correctness is not based on exact symbolic matching to a reference derivation. Any program is acceptable if it yields coordinates satisfying the declared constraints. Second, verification is per-constraint and continuous prior to being collapsed into a binary solved/unsolved outcome, which enables richer reward shaping during training (Cabral et al., 8 Jun 2026). Third, the benchmark is execution-based: syntax, control flow, and numeric stability affect outcomes because the model must produce runnable code, not just a declarative sketch (Cabral et al., 8 Jun 2026).
A plausible implication is that PyGeoX-Bench evaluates a compound capability involving language understanding, constructive planning, numerical reasoning, and program synthesis. That makes it complementary to benchmarks such as GeoBench, where theorem selection and sub-goal decomposition are explicit tasks, and to code-oriented geospatial benchmarks such as GeoAnalystBench, where workflow validity and CodeBLEU matter more than numerical constraint satisfaction (Feng et al., 30 Dec 2025, Zhang et al., 7 Sep 2025).
4. Residual-based rewards and Outlier Gradient Masking
A defining contribution associated with PyGeoX-Bench is the analysis of how solver residuals should be converted into rewards for learning. The benchmark exposes per-constraint residuals, which permits both global-norm rewards and decomposed per-constraint rewards (Cabral et al., 8 Jun 2026).
A natural baseline is a global-norm reward based on sum of squared errors,
7
with a decreasing kernel such as
8
where 9 in the reported experiments (Cabral et al., 8 Jun 2026). The paper argues that such rewards suffer from a failure mode termed Outlier Gradient Masking: a single severely violated constraint can drive the global reward close to zero, thereby suppressing the policy gradient associated with all other, partially improved constraints (Cabral et al., 8 Jun 2026).
Two theoretical results support this claim. The first states that for global-norm rewards, the effective reward volume shrinks to zero as the number of constraints grows, whereas for additive per-constraint rewards the effective reward volume tends to occupy almost all of the residual space (Cabral et al., 8 Jun 2026). The second shows that gradients under global norms are coupled to the total error magnitude $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$0, while gradients under additive rewards remain local to individual residuals (Cabral et al., 8 Jun 2026).
Empirically, on 3,893 partial solutions, the paper reports that under Saturating Additive Rewards, 97.1% of samples fall in a useful reward range $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$1, whereas with the MSE-based reward about 60% collapse near 0 and another 20.6% near 1, leaving only 19% in a discriminative range (Cabral et al., 8 Jun 2026). This suggests that PyGeoX-Bench is not merely a test set; it is also a testbed for studying reward informativeness in multi-constraint environments.
The proposed reward family is Saturating Additive Rewards (SAR):
$\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$2
where $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$3 is monotone decreasing and bounded, with the experiments using the Boltzmann kernel
$\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$4
To make the task precision-critical, SAR is combined with a sparse success bonus and a degeneracy penalty, yielding
$\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$5
with $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$6, $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$7, and $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$8 (Cabral et al., 8 Jun 2026).
In this respect, PyGeoX-Bench differs sharply from geometry benchmarks oriented around final-answer multiple-choice accuracy. GeoBench, for example, diagnoses sub-goal decomposition or theorem selection via accuracy over distractor choices (Feng et al., 30 Dec 2025), whereas PyGeoX-Bench exposes dense solver-grounded reward structure at the level of individual constraints (Cabral et al., 8 Jun 2026).
5. Experimental results and benchmark difficulty
PyGeoX-Bench was used to evaluate both base and trained models, with a strong focus on Qwen3-8B under supervised fine-tuning and reinforcement learning (Cabral et al., 8 Jun 2026). The benchmark’s core stress test is the Hard tier.
The reported Hard-tier solving rates for Qwen3-8B are as follows.
| Training condition | Hard SR |
|---|---|
| Base model | 0.18 |
| SFT, SAR only | 0.23 |
| SFT, Sparse only | 0.23 |
| SFT, MSE only | 0.22 |
| SFT, SAR+S+D | 0.32 |
| SFT, MSE+S+D | 0.04 |
| RL, Sparse only | 0.35 |
| RL, SAR only | 0.09 |
| RL, MSE only | 0.10 |
| RL, MSE+S+D | 0.18 |
| RL, SAR+S+D | 0.41 |
The comparison between RL with SAR+S+D and RL with MSE+S+D is central: the Hard-tier solving rate improves from 0.18 to 0.41, a $\mathcal{E}(\mathbf{u}) = \alpha \sum_i E_i(\mathbf{u})^2 + \beta \sum_j \max(0, -G_j(\mathbf{u}))^2 + \beta \sum_k \max(0, \epsilon - S_k(\mathbf{u}))^2 + \gamma \sum_\ell \mathds{1}_{|N_\ell(\mathbf{u})| < \epsilon},$9 gain (Cabral et al., 8 Jun 2026). The paper also reports that on Hard-tier SFT, SAR+S+D reaches 0.32 whereas MSE+S+D drops to 0.04, described as catastrophic (Cabral et al., 8 Jun 2026). These results are presented as empirical evidence that dense global-norm rewards can be actively harmful in high-constraint synthesis tasks.
On the out-of-distribution PyGeoX-Wild set of 86 human-written problems, the same pattern persists: SAR+S+D under RL reaches 0.66 versus 0.59 for Sparse and 0.60 for MSE+S+D (Cabral et al., 8 Jun 2026). Frontier proprietary models evaluated zero-shot achieve Hard SR 0.51 at best, while the authors’ 8B RL model reaches 0.41 (Cabral et al., 8 Jun 2026). This indicates that the benchmark is non-trivial even for strong large-scale systems.
The paper further notes that the SAR+S+D-trained RL model uses approximately 22.8% fewer tokens on average than a sparse-trained RL model on PyGeoX-Bench (Cabral et al., 8 Jun 2026). This suggests that reward shaping affects not only solution quality but also generation efficiency.
A plausible implication is that PyGeoX-Bench measures an internalized constructive bias toward exact geometry rather than superficial pattern completion. The authors estimate the Hard-tier configuration space to exceed 0, arguing that memorization is infeasible at that scale (Cabral et al., 8 Jun 2026).
6. Relation to adjacent benchmark families
PyGeoX-Bench belongs to a broader 2025–2026 wave of geometry and geospatial benchmarks, but its problem definition is unusually specific. GeoBench evaluates multimodal geometric problem solving through hierarchical tasks including Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking (Feng et al., 30 Dec 2025). Its tasks are multiple-choice and generated from formally verified reasoning graphs. PyGeoX-Bench, by contrast, does not decompose reasoning into subskills; it directly tests whether a model can synthesize a valid construction as executable code (Cabral et al., 8 Jun 2026).
This contrast clarifies a common misconception: PyGeoX-Bench is not a textbook-style geometry QA benchmark, nor is it primarily a proof benchmark. It is a geometric generation benchmark. Its central object is a constraint-satisfying scene, not a theorem statement or numeric answer (Cabral et al., 8 Jun 2026).
Another nearby family concerns geospatial AI rather than Euclidean geometry. GEO-Bench-2 focuses on evaluation protocols for geospatial foundation models across classification, segmentation, regression, detection, and instance segmentation, introducing “capability groups” to summarize strengths across modalities and resolutions (Simumba et al., 19 Nov 2025). GeoX-Bench targets cross-view geo-localization and pose estimation for large multimodal models using panoramic and satellite imagery (Zheng et al., 17 Nov 2025). GeoAnalystBench and GeoBenchX evaluate LLMs on geospatial workflow and tool-use tasks, respectively, emphasizing GIS reasoning, code generation, and hallucination rejection (Zhang et al., 7 Sep 2025, Krechetova et al., 23 Mar 2025). These are methodologically relevant as examples of benchmark design, but they do not share PyGeoX-Bench’s solver-residual verification structure.
The naming similarity between PyGeoX-Bench and GeoX-Bench can itself be misleading. GeoX-Bench concerns cross-view geo-localization and pose estimation over real-world imagery (Zheng et al., 17 Nov 2025), whereas PyGeoX-Bench concerns synthetic Euclidean geometric synthesis from natural language (Cabral et al., 8 Jun 2026). The shared “GeoX” string reflects distinct research agendas.
A plausible implication is that PyGeoX-Bench occupies a niche between symbolic theorem proving and program synthesis: closer to CAD-like construction validity than to standard mathematical QA. The paper explicitly points to future extensions toward 3D CAD, kinematics, and other precision-critical domains (Cabral et al., 8 Jun 2026).
7. Release, extensibility, and limitations
PyGeoX-Bench is released as part of the PyGeoX GitHub repository, together with the PyGeoX engine, a 100k-problem training corpus and its generation code, PyGeoX-Wild, training scripts and configurations for supervised fine-tuning and reinforcement learning, and the evaluation harness used in the paper (Cabral et al., 8 Jun 2026). The benchmark is described as procedurally generated and extensible: new problems can be created by sampling different combinations of object and relation types and then solving and validating them with PyGeoX (Cabral et al., 8 Jun 2026).
This extensibility is substantial because difficulty can be modulated by changing the number of objects, the number and type of constraints, and the degeneracy filters applied during generation (Cabral et al., 8 Jun 2026). In benchmark-design terms, this makes PyGeoX-Bench closer to a benchmark generator than to a fixed finite dataset.
The main limitations acknowledged in the paper are that the engine and benchmark are currently restricted to 2D static geometry, that experiments concentrate on a single strong base model family, and that PyGeoX-Wild is modest in size at 86 problems (Cabral et al., 8 Jun 2026). The authors suggest extending the framework to 3D and dynamic or kinematic constraints and exploring SAR-style reward shaping in other multi-constraint RLVR domains such as physics environments, robotic control, and scene layout (Cabral et al., 8 Jun 2026).
These limitations are informative. They suggest that PyGeoX-Bench should not be interpreted as a general geometry benchmark in the educational sense, nor as a complete test of theorem-proving competence. Rather, it is a benchmark for a specific but demanding form of geometric intelligence: mapping free-form language to numerically exact, executable geometric constructions under verifiable multi-constraint semantics (Cabral et al., 8 Jun 2026).
In that role, PyGeoX-Bench provides a distinctive research instrument. It couples declarative geometry, executable synthesis, dense residual analysis, and solver-grounded evaluation in a single framework. This combination makes it particularly valuable for studying how LLMs can internalize geometric law when correctness is determined not by plausibility or style, but by strict constraint satisfaction (Cabral et al., 8 Jun 2026).