GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation

Abstract: Referring remote sensing image segmentation isolates the object named by a natural-language expression in an aerial image. Existing training-free methods resolve the expression through implicit vision-language activations or region-text similarity, which gives weak control over the spatial, comparative, and ordinal relations that dominate aerial referring: they cannot represent constructions such as the largest ship or the second court from the left. We propose GeoSelect, a training-free pipeline that reframes referring as the execution of a typed spatial program. A frozen, text-only LLM synthesises the expression into a small domain-specific language, a well-formedness checker accepts the program, and a deterministic executor runs it. The central abstraction is a single scored candidate set type under which every operator composes: continuous geometric fields realise position and proximity as dense pixel-level maps, while discrete set and order operators add the extremum, ordinal, counted-union, and relational constructions that fields alone cannot express. Because execution is explicit, every intermediate program, field, and ranking is inspectable, and a reliability ladder degrades any failing program to a field-only special case, so every expression returns an answer. GeoSelect attains 58.86 mIoU on RRSIS-D test and 55.27 mIoU on RISBench test, more than twice the best prior training-free method on RRSIS-D, with no referring supervision and on a single GPU. A controlled comparison with candidates and segmenter fixed attributes the gain to explicit execution, not the backbone; an oracle decomposition localises the residual gap to detection recall on RRSIS-D and selection on RISBench, and an exposure audit confirms robustness to pretraining leakage. Code will be released upon acceptance at the project page https://avalon-s.github.io/GeoSelect/.

• The paper introduces GeoSelect, a training-free approach that synthesizes and executes explicit spatial programs for remote sensing image segmentation, achieving near-supervised performance.
• The methodology leverages a frozen text-only language model, an open-vocabulary detector, and a promptable segmenter to interpret complex spatial expressions with compositional reasoning.
• Experimental results show significant improvements, with explicit program execution yielding a +12.06 mIoU gain over implicit methods in cluttered settings.

GeoSelect: Explicit Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation

Introduction

Referring remote sensing image segmentation (RRSIS) seeks to segment the object referenced by a natural-language expression within an aerial image, often under conditions of significant same-class clutter and complex spatial relations. The core challenge extends beyond object recognition to precise instance disambiguation—requiring models not just to find the right class, but to interpret spatial, comparative, and ordinal cues inherent in referring expressions. While recent supervised methods achieve strong performance, they require dense annotation and per-benchmark supervision, limiting their scalability and adaptability.

GeoSelect proposes a paradigm shift: reframing referring segmentation as typed spatial-program execution, thereby achieving strong performance without any referring supervision or retraining. The pipeline directly synthesizes an executable spatial program from the expression using a frozen LLM, executes the program over a candidate set generated by an open-vocabulary detector, and segments the selected box using a frozen promptable segmenter. Every component is frozen—no weights are updated on benchmark data—leading to a highly interpretable, training-free neuro-symbolic pipeline that achieves state-of-the-art performance among training-free methods.

Motivation for Explicit Program Execution

Prior training-free approaches rely on implicit region–text matching or similarity-based ranking to ground referring expressions. However, these are inherently limited in representing complex spatial, comparative, and ordinal relations—categories that dominate in remote sensing imagery, where multiple same-class objects co-occur in structured spatial layouts.

GeoSelect addresses this conceptual shortcoming by introducing explicit, typed program execution. Instead of relying solely on black-box model activations, it parses expressions into compositional symbolic programs, executes them with geometric fidelity, and thus enables fine-grained, interpretable reasoning Figure 1.

Figure 1: For a spatial expression over several same-class objects, implicit region–text matching (left, GeoRSCLIP) fails, whereas GeoSelect synthesizes and executes a typed spatial program that selects the ground-truth instance (right), using an explicit, compositional scoring of candidate boxes.

In controlled comparisons where the candidate detector and segmenter are fixed, the advantage of explicit execution over implicit selection is pronounced—especially for spatially complex or compositional expressions, and in scenes with heavy instance clutter.

Methodology

Pipeline Overview

GeoSelect's pipeline factors the RRSIS task into three frozen stages:

1. Candidate Generation: An open-vocabulary detector, queried via category and attribute terms parsed from the expression, proposes candidate boxes.
2. Selection via Program Execution: The core selector parses the referring expression with a frozen, text-only LLM into a program written in a compact spatial DSL. This program is checked for well-formedness and executed deterministically over the scored set of candidate boxes.
3. Segmentation: The top-scoring box prompts a frozen segmenter (e.g., SAM-ViT-L) to produce the binary referent mask.

Only the selection mechanism—i.e., the execution of the synthesized program—varies among alternatives. Every other module is frozen. The system guarantees output for every expression via a "reliability ladder": if the program fails or is ill-formed, it falls back to a field-only scoring mechanism that supports a closed set of positional and attribute predicates.

Figure 2: GeoSelect executes a typed spatial program—composed of continuous geometric filters and discrete set/order operators—over a scored candidate set, with each stage and intermediate trace inspectable and grounded in geometric reasoning.

Typed Spatial Program DSL

The program DSL supports compositional reasoning through:

• Continuous Field Operators: Filters applying geometric kernels (direction, center, proximity) parameterized by anchors, generating soft spatial priors.
• Discrete Set/Order Operators: Argmax/argmin (superlative), nth (ordinal), counted unions (multi-instance reference), and binary relations (near, between, contains, etc.).
• Boolean Set Operations: And/or/not combine constraints on candidate sets.

Every intermediate is a "scored candidate set”—a list of boxes with associated scores—permitting free composition of fields and discrete operations, crucial for capturing the layered complexity of natural referring expressions.

Program Synthesis and Execution

A frozen text-only instruction LLM (e.g., Qwen3-4B) synthesizes the program from the language expression using a handcrafted schema, outputting JSON representations of the DSL. The program is checked for well-formedness and executed bottom-up by a deterministic interpreter. If failure or ill-formedness occurs, the process falls back to the less expressive field-only mechanism, guaranteeing an answer and bounding the risk of synthesis errors.

The geometric scoring uses closed-form, anchor-relative kernels for directional, center, and proximity predicates. Extremum and ordinal selection leverages axis-based softmaxes or ranking; binary and multi-instance reference use well-defined set-based algorithms.

Detection and Segmentation

The candidate generator is an aerial-pretrained open-vocabulary detector (e.g., LAE-DINO), and segmentation is performed via promptable, foundation model segmenters (e.g., SAM-ViT-L). Proposals and segmentation are frozen and interchangeable, isolating scientific contribution to the selection mechanism.

Experimental Results

Main Results and Comparisons

GeoSelect attains 58.86 mIoU on RRSIS-D and 55.27 mIoU on RISBench without any referring supervision, over twice the best prior training-free result on RRSIS-D (2607.03869). The field-only special case (no discrete operators) captures most of the RRSIS-D performance gap, but explicit program execution and discrete operators yield significant additional gains—especially on expressions requiring compositional or superlative reasoning.

GeoSelect's accuracy approaches 93% of the supervised specialist RMSIN on RRSIS-D, a remarkable result for a fully training-free pipeline. Importantly, in a controlled ablation with candidate proposals and segmentation fixed, the gain from explicit program execution over the strongest implicit baseline is +12.06 mIoU.

Figure 3: Qualitative traces across the pipeline: input, candidate boxes, spatially-scored candidate set (brightness proportional to score), GeoSelect's output mask, and ground truth.

Operator and Stratified Analysis

Program-level discrete operators—argmax/argmin, ordinal, and counted-union—are key for RISBench, which contains many complex superlative and compositional expressions. On RISBench, these operators provide an additional +2.22 mIoU on the superlative stratum, and an overall stratified improvement not attainable by closed-form fields alone.

Oracle Analysis and Error Attribution

An oracle-based analysis reveals that most errors are now selection errors rather than detection errors on RISBench: the correct candidate is usually proposed, but program synthesis or execution may err in ranking. Conversely, on RRSIS-D, detection recall remains the main bottleneck, with most residual error attributable to missed proposals rather than selection or segmentation errors.

Figure 4: Failure modes by pipeline stage: (a) Detection—object not proposed, (b) Selection—wrong box chosen among good candidates, (c, d) Synthesis—malformed or semantically incorrect program.

Figure 5: Pipeline-wide failure attribution: synthesis errors rise notably on RISBench, quantifying the growing importance of robust program parsing and execution.

Efficiency, Robustness, and Transfer

GeoSelect is highly efficient, requiring no referring training runs and operating in single-sequence parsing and proposal generation on commodity hardware (RTX 3090). A pretraining-leakage audit shows that performance inflation due to overlapping pretraining data is minimal, with unseen images achieving nearly the same mIoU as the full test set.

On cross-dataset transfer (RISBench, with the RRSIS-D-tuned configuration held frozen), GeoSelect generalizes strongly, outperforming prior training-free systems by a wide margin. The system consistently outpaces implicit matchers in scenarios with dense same-class instances and compositional relational expressions.

Figure 6: Traces on RISBench, including superlative, ordinal, and compositional cases, illustrating GeoSelect's capability in challenging, compositional scenarios.

Implications and Future Directions

GeoSelect demonstrates that explicit program synthesis and execution—a neuro-symbolic approach—offers decisive advantages for complex spatial reasoning tasks in remote sensing image understanding. The paradigm allows for interpretability, reliability, and compositional coverage, narrowing the gap to supervised methods without any additional supervision.

The rigorous oracle and failure analyses direct future development: improvements in anchor binding, robustness in axis estimation, and the expansion of the DSL to better capture compositional semantics under detector noise will be critical, especially for more synthetic or multi-instance referring tasks.

From a broader perspective, GeoSelect underscores the power and practicality of hybrid neuro-symbolic systems: integrating frozen, foundation perception modules with symbolic reasoning provides both accuracy and interpretability, especially in domains where training data annotation is intrinsically costly or ambiguous.

Conclusion

GeoSelect reframes referring remote sensing image segmentation as explicit, training-free, spatial-program execution. Using a frozen, text-only LLM for program synthesis and deterministic geometric execution, the approach unifies continuous fields and discrete set/order operators within a single scored-candidate-set framework. On established benchmarks, GeoSelect delivers substantial improvements over prior training-free methods, reaching close to specialist supervised accuracy while retaining interpretability and efficiency. Its success confirms the viability of explicit neuro-symbolic reasoning for compositional vision–language tasks, providing both a strong practical solution and a principled platform for further research in compositional, open-vocabulary visual grounding.