Geo-Eval: Geospatial Evaluation Frameworks
- Geo-Eval is a suite of frameworks that adapts evaluation methods to the unique spatial properties and semantics of geospatial tasks.
- It integrates tailored datasets, spatially aware metrics, and specialized cross-validation to assess machine learning, code generation, and linguistic tasks.
- The approach emphasizes reproducibility and diagnostic analysis by distinguishing overall accuracy from task-specific capability and model transferability.
Geo-Eval denotes a family of evaluation frameworks and methodologies for geospatial AI, geospatial machine learning, geo-localization, geospatial code generation, geoparsing, geographic language understanding, Earth observation foundation models, geological reasoning, and geoscience visual reasoning. Across these uses, the term refers not to a single benchmark but to a recurring evaluative program: the construction of task-specific datasets, protocols, metrics, and reporting standards that account for the distinctive statistical, semantic, spatial, or operational properties of geospatial problems. In one foundational formulation, Geo-Eval is a geospatially aware evaluation methodology for spatial machine learning that emphasizes spatial dependence, sampling gaps, covariate shift, edge effects, and the distinction between map accuracy as a population parameter and a model’s spatial generalization ability (Rolf, 2023). In later works, the name is also applied to unified ecosystems for image geo-localization, geospatial code generation, GeoSQL generation, embodied geo-localization, Earth observation model benchmarking, geological reasoning, and related domains, each adapting evaluation to the structure of the underlying task.
1. Geospatially aware model evaluation in spatial machine learning
A central use of Geo-Eval appears in the evaluation of geospatial machine learning with global or remotely sensed datasets. In this formulation, evaluation departs from off-the-shelf machine learning practice because spatial data exhibit positive spatial autocorrelation, non-random label availability, scale dependence, and domain shift. Nearby locations tend to have more similar values, so classical k-fold or random train/test splits assume independence and will overestimate real-world performance. Covariates and labels correlate across space, and models may “ride” these correlations rather than learn true predictors; when moved to new regions, these shortcuts fail. The modifiable areal unit problem and ecological fallacy imply that the choice of spatial aggregation unit changes both the data distribution and the evaluation outcome. In addition, uneven data availability means that a design-based probability sample is required to estimate map accuracy as a population parameter, whereas many studies reuse opportunistic data (Rolf, 2023).
Within this Geo-Eval formulation, standard regression and classification metrics are retained but spatially adapted. Mean Absolute Error, Root Mean Squared Error, and are weighted by sampling probability, area, or population, yielding weighted variants such as , , and . AUROC or Precision-Recall may also be computed with spatially stratified sampling and weights to estimate map accuracy in up-sampled or down-sampled classes. This emphasizes that a metric is not only a numerical summary but an estimator whose interpretation depends on the sampling design and the target estimand (Rolf, 2023).
The same framework also formalizes spatially adapted cross-validation. Block CV partitions the study area into non-overlapping spatial blocks; leave-one-region-out CV removes entire countries or ecoregions; buffered CV enforces a minimum train-test separation; checkerboard or environmental clustering imposes separation in geographic and covariate space; and nested CV combines an outer spatial CV for performance estimation with an inner spatial CV for model or hyperparameter selection. These procedures are intended to simulate the intended prediction regime, whether interpolation or extrapolation, while avoiding spatial leakage (Rolf, 2023).
Residual diagnostics are another defining component. Moran’s is used to quantify spatial autocorrelation in residuals, with positive values indicating residual spatial structure. Suggested corrections include explicit spatial random effects, Gaussian Processes, geostatistical models such as conditional autoregressive or Matérn covariance formulations, post-estimation smoothing baselines, and retraining with spatially lagged predictors or residual autocovariate terms. This establishes evaluation as an iterative diagnostic process rather than a terminal score-reporting exercise (Rolf, 2023).
The best-practice checklist associated with this version of Geo-Eval organizes evaluation around five pillars: spatially aware metrics, structured cross-validation, autocorrelation diagnostics, weighted aggregation, and rigorous reporting. It distinguishes between population-level map accuracy and model transferability to new regions, advocates weighted metrics and uncertainty reporting, recommends visualizing train-test splits and comparing against spatial baselines, addresses edge effects and covariate shift, and calls for transparent release of split indices, weight schemes, code, and explicit model cards (Rolf, 2023). This suggests that, in its original methodological sense, Geo-Eval functions as a normative protocol for valid geospatial model assessment rather than a single benchmark.
2. Geo-localization and multimodal reasoning benchmarks
A second major use of Geo-Eval concerns image geo-localization and multimodal reasoning. In the "GRE Suite: Geo-localization Inference via Fine-Tuned Vision-LLMs and Enhanced Reasoning Chains," Geo-Eval is the name given to the unified data-model-benchmark ecosystem comprising GRE30K, the GRE model, and GREval-Bench (Wang et al., 24 May 2025). GRE30K is assembled from a 5% sample of the MP16-Pro corpus, with 20,000 images for chain-of-thought demonstrations and 10,000 images for reasoning-quality judgments. It evaluates localization at five distance-based tasks defined by thresholds of 1 km, 25 km, 200 km, 750 km, and 2500 km, corresponding respectively to street, city, region, country, and continent. Each chain of thought is labeled according to semantic category: scene attribute, local detail, or high-level inference. Coverage is characterized by a Shannon diversity index over coarse regions (Wang et al., 24 May 2025).
The GRE model begins with Qwen2.5-VL-7B and uses cold-start supervised fine-tuning followed by two-stage GRPO reinforcement learning. Its chain-of-thought is organized into three stages: Scene Attributes, Local Details, and Semantic Inference. Evaluation in GREval-Bench measures both final coordinate accuracy and the quality of intermediate reasoning. Top-1 accuracy is defined per threshold, geodesic error is computed by spherical distance, and chain-of-thought quality is the average of Recall, RefCLIPScore, and BERTScore. Statistical significance between methods is assessed with paired t-tests over per-image indicator outcomes. Reported results show gains over existing methods on Im2GPS3k and GWS15k, with significance at (Wang et al., 24 May 2025).
GeoArena extends Geo-Eval in a different direction by replacing coordinate-only evaluation with pairwise human preference judgments on worldwide image geo-localization (Jia et al., 4 Sep 2025). It is described as the first live, user-centered geo-eval platform that ingests in-the-wild images, crowdsources pairwise human preferences, ranks large vision-LLMs through a Bradley-Terry and Elo-style pipeline, preserves privacy by never requiring ground-truth GPS, and continuously updates a public leaderboard. Users upload images, two anonymous model outputs are shown side by side, and votes are recorded as win, loss, or tie. The platform estimates latent model strengths through Bradley-Terry modeling, transforms them into Elo-style scores, and reports 95% confidence intervals via bootstrap. Its design is explicitly motivated by concerns about data leakage from static benchmarks and by the privacy implications of storing user-level coordinates (Jia et al., 4 Sep 2025).
ERGeoBench broadens this geo-evaluation paradigm to embodied geo-localization (Xue et al., 29 May 2026). It contains 2,207 globally distributed street-view panoramas from 56 countries and evaluates models in three settings: single-view, panorama-view, and embodied-view. Beyond final geo-localization reasoning, it measures foundational perception, spatial awareness, and common-sense reasoning. Localization is summarized by a unified Geo-Localization Score combining semantic alignment, metric precision across thresholds of 1, 25, 200, 750, and 2500 km, and an error penalization term based on median distance. Reported results indicate that models can infer high-level geographic semantics but still struggle with fine-grained perceptual operations, metric localization, and spatial consistency across views (Xue et al., 29 May 2026).
These systems differ in what they regard as the proper object of evaluation. GREval-Bench evaluates coordinate accuracy together with the quality of intermediate reasoning (Wang et al., 24 May 2025); GeoArena evaluates which output better aligns with human expectations (Jia et al., 4 Sep 2025); ERGeoBench treats geo-localization as one capability within a broader embodied reasoning problem (Xue et al., 29 May 2026). A plausible implication is that Geo-Eval in geo-localization has evolved from distance-threshold scoring toward multi-component assessments incorporating reasoning traces, human preference, active sensing, and privacy constraints.
3. Automated evaluation for geospatial code and query generation
Geo-Eval is also used for automated, execution-based assessment of LLMs on geospatial programming tasks. AutoGEEval is presented as the first multimodal, unit-level automated evaluation framework for geospatial code generation on Google Earth Engine (Hou et al., 19 May 2025). It consists of AutoGEEval-Bench, a submission program, and a judge program. The benchmark contains 1,325 unit tests over 26 Google Earth Engine data types, derived from the official GEE Reference after excluding deprecated functions. Each test case is structured as a 6-tuple containing a function header, reference code, parameter list, output type, output path, and expected answer. Because the GEE JavaScript sandbox is closed, the framework uses the Python API for full automation and error capture (Hou et al., 19 May 2025).
AutoGEEval reports pass@n, overall accuracy, coefficient of variation, stability-adjusted accuracy, resource-consumption metrics such as token usage, inference time, and code lines, operational efficiency metrics, and error categories including Syntax Error, Parameter Error, Invalid Answer, and Network/Server Error. Benchmarking across 18 models showed best pass@5 results for DeepSeek-V3, DeepSeek-R1, and Gemini-2.0, while parameter errors dominated failure patterns, suggesting that general coding ability exceeded task-specific knowledge of Earth Engine parameters and catalogs (Hou et al., 19 May 2025).
AutoGEEval++ extends this approach to 6,365 test cases covering 26 GEE data types and three levels of complexity: unit, combo, and theme tests (Hou et al., 12 Jun 2025). Its benchmark is YAML-defined, and theme tests are drawn from 93 peer-reviewed papers. The framework keeps the submission program and judge program architecture but expands the task hierarchy and metrics. Accuracy@1, pass@n, coefficient of variation, stability-adjusted accuracy, resource consumption, operational efficiency, and error-type logs are all reported. Experimental results indicate that combo tests are easier than theme tests, multi-round gains diminish beyond three attempts, and boundary-test pass rates vary substantially across models (Hou et al., 12 Jun 2025).
GeoJSEval applies a similar Geo-Eval logic to JavaScript-based geospatial computation and visualization (Chen et al., 28 Jul 2025). It is described as the first multimodal, function-level automatic evaluation framework for JavaScript-based geospatial code generation, with 432 function-level tasks and 2,071 structured test cases spanning five JavaScript geospatial libraries and 25 geospatial data types. Its three components are GeoJSEval-Bench, a submission program, and a judge program. The framework addresses fragmented APIs, complex data types and multimodal outputs, browser versus Node execution environments, and robustness under boundary conditions through an “edge_test” mechanism. Accuracy metrics include pass@n, single-round accuracy, coefficient of variation, and stability-adjusted accuracy; resource and efficiency metrics are also reported. Runtime failures are classified into Syntax Error, Attribute/Parameter Error, Output Type Error, Invalid Answer, Runtime Error, and Other Error (Chen et al., 28 Jul 2025).
GeoAnalystBench shifts attention from unit-level function generation to multi-step Python GIS workflows (Zhang et al., 7 Sep 2025). It comprises 50 tasks drawn from public GIS tutorials, academic publications, and ESRI ModelBuilder examples, each paired with a minimum deliverable product consisting of a geoprocessing workflow plan and a runnable Python script. The benchmark evaluates workflow validity, structural alignment through Mean Absolute Deviation, semantic similarity via sentence-transformer cosine similarity, and code quality through CodeBLEU. Results show substantially stronger performance for proprietary models than smaller open-source models, with “Finding best locations and paths” and “Determining how places are related” emerging as the most difficult categories (Zhang et al., 7 Sep 2025).
GeoSQL-Eval extends automated geo-evaluation into spatial databases (Hou et al., 28 Sep 2025). It is described as the first end-to-end automated evaluation framework for PostGIS query generation, grounded in Webb’s Depth of Knowledge model. GeoSQL-Bench comprises 14,178 questions across three task types, 340 PostGIS functions, and 82 domain-specific databases. The framework measures syntax accuracy, table hit rate, field hit rate, function name rate, argument matching accuracy, execution pass rate, geometric result correctness, pass@n, coefficient of variation, stability-adjusted accuracy, and entropy-weighted overall scores. Reported findings show high syntax accuracy among top models but much lower execution and semantic alignment accuracy, with a substantial fraction of errors arising from PostGIS function misuse and SQL syntax mistakes (Hou et al., 28 Sep 2025).
Across these systems, Geo-Eval denotes an automated pipeline in which model output is executed, compared against reference behavior or expected outputs, and analyzed along accuracy, stability, efficiency, and error-type dimensions. This contrasts with purely text-overlap evaluation and reflects the operational semantics of geospatial code, where correctness depends on execution, data types, spatial functions, and environment-specific behavior.
4. Geo-Eval in geoparsing and geographic language understanding
In geographic natural language processing, Geo-Eval is used to standardize evaluation across extraction, disambiguation, and language understanding tasks. "A Pragmatic Guide to Geoparsing Evaluation" presents a geoparsing Geo-Eval framework centered on task clarification, metric consolidation, and a pragmatic taxonomy of toponyms (Gritta et al., 2018). The framework distinguishes geotagging from toponym resolution and argues that evaluation must account for different kinds of place-name use rather than treating all toponyms as semantically identical. Literal toponyms, literal modifiers, mixed forms, coercion, and embedded literals are separated from associative categories such as metonymy, demonyms, languages, associative modifiers, embedded associatives, and homonyms. This taxonomy is intended to reduce precision errors from non-locations and recall errors from non-canonical but valid place mentions (Gritta et al., 2018).
The same framework recommends evaluating geotagging with Precision, Recall, and , and evaluating resolution with Mean Error Distance, thresholded accuracy such as Acc@161 km, and Area Under the Error-Distance Curve. It also advocates McNemar’s test for geotagging significance and Wilcoxon signed-rank tests for geocoding significance. GeoWebNews, a corpus of 200 English-language articles and 2,720 toponyms linked to Geonames IDs, provides the shared data resource for this protocol (Gritta et al., 2018).
A complementary Geo-Eval for Twitter user geolocation is proposed in "A Practical Guide for the Effective Evaluation of Twitter User Geolocation" (Mourad et al., 2019). There, evaluation is organized around continuous error-distance metrics, discrete classification metrics, and mixed thresholded accuracy. The guide emphasizes that metric choice can substantially alter conclusions, especially across country, region, city, and grid granularities. It recommends reporting Accuracy, Acc@161 km, micro and macro Precision/Recall/, and median and mean error distance, using Kendall’s to assess rank consistency among metrics and applying different significance tests for micro-level and macro-level comparisons. One of its central empirical findings is that a majority-class baseline remains competitive at coarse granularity (Mourad et al., 2019).
EUPEG operationalizes geoparsing Geo-Eval as an extensible web platform for comparative experimentation (Wang et al., 2020). It hosts eight corpora, nine end-to-end geoparsers, and eight performance measures, providing a common JSON format and a comparison engine. Metrics include Precision, Recall, , Accuracy, Mean Error Distance, Median Error Distance, Acc@161 km, and AUC. It stores experiments in an SQLite archive and allows new geoparsers to be registered through RESTful endpoints. This platform is intended to reduce the effort required to prepare datasets, deploy systems, and reproduce comparative results (Wang et al., 2020).
GeoGLUE applies a benchmark paradigm to geographic language understanding more broadly (Li et al., 2023). It defines six tasks: GeoTES-recall, GeoTES-rerank, GeoETA, GeoCPA, GeoWWC, and GeoEAG. These span retrieval, sequence tagging, and text classification. Metrics include MRR@5, MRR@1, Micro-0, and Macro-1. Baseline experiments with BERT, RoBERTa, ERNIE, Nezha, and StructBERT show strong performance on structured address tagging and weaker performance on colloquial query understanding and entity alignment. This benchmark is presented as a unified standard for geographic language understanding, analogous in function to general-domain language evaluation suites (Li et al., 2023).
Taken together, these works show that Geo-Eval in geographic NLP is chiefly concerned with comparability: clarifying the task boundary, specifying the proper metric family for each stage, exposing biases due to granularity or toponym type, and providing common corpora or platforms. This suggests a continuity between geospatial machine learning evaluation and geographic language evaluation: in both cases, Geo-Eval arises where standard metrics become misleading once spatial semantics, ambiguity, or geographic scale are ignored.
5. Earth observation, remote sensing, and geoscience benchmark ecosystems
In Earth observation and geoscience, Geo-Eval increasingly denotes large, prescriptive benchmark ecosystems rather than only metric suites. GEO-Bench-2 is described as an end-to-end evaluation framework for Geospatial Foundation Models that aggregates 19 open-license datasets across classification, semantic segmentation, regression, object detection, and instance segmentation (Simumba et al., 19 Nov 2025). It introduces “capability groups” such as Core, Pixel-wise, Classification, Detection, Multi-Temporal, resolution-based groupings, RGB/NIR, and Multi-Spectral-Dependent. Standardized metrics are used per task: Accuracy, 2, mIoU, RMSE, and mAP. The protocol fixes dataset splits, per-band Z-score normalization, augmentations, hyperparameter-search budgets, and repeatability guidelines while allowing methodological innovation in adaptation strategies. Aggregate scores are obtained by linearly remapping dataset scores and computing repeated, stratified bootstrap estimates aggregated through the interquartile mean (Simumba et al., 19 Nov 2025).
Geo3DVQA defines a benchmark for height-aware, 3D geospatial reasoning from RGB-only aerial imagery (Tsujimoto et al., 8 Dec 2025). It contains 110,000 question-answer pairs over 16 task categories grouped into three tiers: single-feature inference, multi-feature reasoning, and application-level spatial analysis. Inputs are derived from RGB imagery, LiDAR Digital Surface Models, and semantic segmentation from GeoNRW, but evaluation focuses on what vision-LLMs can infer from RGB alone. Metrics include short-answer accuracy, Jaccard similarity for multi-label land-cover tasks, tolerance rules for height and Sky View Factor estimation, and a free-form 1–5 rubric on Observation, Logic, and Conclusion. Results indicate low zero-shot performance and substantial gains from domain-specific fine-tuning of Qwen2.5-VL-7B (Tsujimoto et al., 8 Dec 2025).
GeoR-Bench evaluates geoscience visual reasoning through reasoning-informed visual editing (Zheng et al., 12 May 2026). It contains 440 samples across six geoscience categories and 24 task types, covering Earth observation imagery, maps, and scientific diagrams. Each task requires editing an input image according to a process-based textual instruction and is scored along reasoning, consistency, and quality dimensions. Strict accuracy is defined as the fraction of outputs that exceed threshold scores in all three dimensions simultaneously. Reported results show that visual consistency and image quality often exceed scientific correctness, with the top model reaching 42.7% overall strict accuracy and the best open-source model reaching 10.3% (Zheng et al., 12 May 2026).
These systems use Geo-Eval not merely to compare models but to characterize capability structure. GEO-Bench-2 decomposes model competence by modality, task, temporal structure, and spectral dependence (Simumba et al., 19 Nov 2025). Geo3DVQA decomposes it by reasoning complexity and geospatial modality (Tsujimoto et al., 8 Dec 2025). GeoR-Bench decomposes it into reasoning, consistency, and quality and thereby reveals a gap between visually plausible outputs and geoscientifically valid outputs (Zheng et al., 12 May 2026). A plausible implication is that Geo-Eval in Earth observation has shifted from leaderboard construction toward capability diagnosis.
6. Domain-specific scientific reasoning and broader conceptual significance
The term Geo-Eval is also used for domain-specific scientific reasoning benchmarks beyond GIS and remote sensing. In "Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning," Geo-Eval is a benchmark for open-ended, multi-step deductive reasoning in solid-earth geology (Guo et al., 24 May 2026). It consists of 387 expert-vetted questions drawn from a larger set of 2,591 textbook-derived items, selected through difficulty-aware adversarial mining and reviewed by geology professors. Questions are grouped into Concept, Process, and Engineering levels. Evaluation uses per-question scores on a 0–10 scale, summarized by average score, performance gain over base models, and paired t-tests for significance. Reported results show that domain-aligned fine-tuned models can outperform larger open-weight generalists and GPT-4o on this specialized task (Guo et al., 24 May 2026).
A much earlier but conceptually relevant use appears in the evaluation of geo-semantic relatedness and similarity. GeReSiD, the Geo Relatedness and Similarity Dataset, provides 50 geographic term pairs rated by 203 human subjects and serves as an evaluation baseline for computational measures of geo-semantic relatedness and similarity (Ballatore et al., 2014). Relatedness and similarity are formally distinguished, human ratings are aggregated and normalized, inter-rater reliability and agreement are quantified, and computational models are evaluated via rank and linear correlations against human judgments. Although GeReSiD predates many later uses of the Geo-Eval label, it exemplifies the same underlying pattern: geospatial tasks require domain-specific ground truth, task-appropriate metrics, and explicit modeling of human or expert judgment (Ballatore et al., 2014).
Across all these settings, several recurrent motifs define Geo-Eval. First, evaluation is tied to the ontology of the task: spatial dependence in gridded prediction, toponym pragmatics in text, execution semantics in code, or process realism in geoscientific image editing. Second, standard generic metrics are often insufficient unless weighted, stratified, decomposed, or paired with complementary diagnostics. Third, benchmark design increasingly combines datasets, model interfaces, evaluation engines, and reproducibility infrastructure into integrated ecosystems. Fourth, many Geo-Eval frameworks explicitly distinguish between overall performance and capability-specific competence, whether by region, granularity, modality, task category, or reasoning stage.
This suggests that Geo-Eval is best understood not as a single benchmark lineage but as an evaluative design philosophy for geospatial AI. Its unifying premise is that geographically grounded tasks cannot be assessed adequately without accounting for spatial structure, geographic semantics, domain-specific operational constraints, and the intended use regime. Under that interpretation, the diverse frameworks named Geo-Eval collectively mark the maturation of geospatial evaluation from ad hoc metric reporting toward reproducible, theory-informed, and task-native benchmarking (Rolf, 2023).