GeoX-Bench: Geospatial AI Benchmark

• GeoX-Bench is a family of modern, large-scale geospatial AI benchmarks that evaluate multimodal and language models on challenging spatial reasoning and perception tasks.
• It features tailored challenges like cross-view geo-localization and geo-pose estimation with balanced, human-verified datasets spanning diverse global regions.
• The benchmark supports instruction tuning and rigorous evaluation, making it pivotal for reproducible comparisons and advancements in GeoAI research.

GeoX-Bench denotes a family of modern, large-scale geospatial AI benchmarks, each systematically designed to evaluate advanced machine learning models—particularly large multimodal and LLMs—across challenging, realistic geospatial reasoning and perception tasks. Distinct from earlier generic vision and NLP benchmarks, recent GeoX-Bench frameworks explicitly target either (a) multistep tool-driven spatial analysis typical of commercial GIS workflows, or (b) cross-view geo-localization and geo-pose estimation from paired ground and satellite imagery. These initiatives have introduced novel data resources, structured evaluation protocols, and support for instruction tuning, collectively shaping the emerging standard for empirical comparison in the GeoAI subdomain.

1. Large-Scale Benchmark Construction

GeoX-Bench encompasses both “GeoBenchX” for LLM-driven multistep geospatial reasoning (Krechetova et al., 23 Mar 2025) and “GeoX-Bench” for LMM-based cross-view localization and pose estimation (Zheng et al., 17 Nov 2025). The latter provides:

• 10,859 panoramic–satellite image pairs across 128 cities in 49 countries, balanced for urban, residential, forest, agriculture, rangeland, and water land-cover categories.
• 755,976 structured QA pairs generated using seven task templates, with 42,900 QA pairs held out for standardized benchmarking and 713,076 pairs reserved for instruction tuning.
• Sampling procedures ensuring class, geographic, and task template balance, including human verification to eliminate mismatches and low-quality prompts.

The benchmark’s granularity, scale, and sampling protocols reflect a commitment to reproducibility and thorough model stress-testing across diverse real-world scenarios.

2. Task Design and Category Definitions

GeoX-Bench defines two primary families of model challenges:

• Cross-View Geo-Localization: Determines physical position using a ground image and candidate satellite tiles.
  • Variants include: binary classification (is the image within a specified tile with or without a positional prior), intra-map localization (relative quadrant or offset), and cross-map retrieval (selecting the correct tile among candidates).
• Geo-Pose Estimation: Predicts the oriented heading (azimuth) of the ground camera relative to north, including both fixed and random in-tile positions.

Each task is formalized as a question–answer pair, often using FoV crops, map grids, and unambiguous templates. Geodesic evaluation is used for positional errors:

$$d(p_1, p_2) = R \cdot \arccos\big( \sin\phi_1\sin\phi_2 + \cos\phi_1\cos\phi_2\cos(\lambda_1-\lambda_2) \big),\quad R = 6371~\text{km}$$

For orientation, error is expressed as:

$$e_\theta = \min\left(\left|\theta_\text{pred} - \theta_\text{gt}\right|,\, 360^\circ - \left|\theta_\text{pred} - \theta_\text{gt}\right|\right)$$

Tasks are balanced to reflect the statistical distribution of the underlying geography and to avoid trivializing the problem space.

3. Model Evaluation and Performance Metrics

GeoX-Bench has been extensively used to evaluate 25 state-of-the-art LMMs of varying sizes (2B to 78B parameters), including both open- and closed-source families (InternVL, Qwen2.5VL, DeepSeek-VL, LLaVA, mPLUG-Owl, GPT-4o, Claude-Sonnet-4, Gemini-2.5-Pro, o3). Models generally combine a substantial vision encoder (typically ViT-derived) with a (often frozen) LLM backbone. Key evaluation metrics include:

• Accuracy@1: Proportion of correctly identified location (or correct bin for pose).
• Geodesic Distance Error: For coordinate outputs, mean and median errors in kilometers.
• Orientation Error: Degrees of azimuthal deviation from ground truth; 4-way bucketed accuracy for coarse evaluation.
• Cross-class Option Bias: Tendency for models to default to specific direction categories, particularly among smaller architectures.

Performance on geo-localization tasks is typically high, with top closed-source models (GPT-4o, Gemini-Pro) achieving 80–90%+ accuracy under various conditions. In contrast, pose estimation remains challenging: pre-tuned models rarely exceed 35% accuracy on four-way direction prediction, with significant improvements (+30–40 percentage points) realized only after instruction tuning via LoRA adapters on GeoX-Bench’s dedicated training QA pool.

4. Instruction Tuning and Model Adaptations

Systematic instruction tuning, leveraging the large QA-pair pool of GeoX-Bench, has proven critical for maximizing performance, particularly in pose estimation and complex localization settings. Standard LoRA-based fine-tuning protocols are applied to all linear layers of paired ViT–LLM stacks. Empirically:

• Average accuracy across all tasks increases dramatically (from ~40% to over 70% for leading models such as InternVL3-8B) after tuning.
• Statistical significance of improvements is rigorously supported (paired t-tests, $p < 0.001$ on major tasks).
• Scaling model size improves localization performance until saturation at 32B parameters, but does not by itself suffice for pose tasks.
• Instruction tuning enables models to move beyond naive option bias and exploit subtle visual/geospatial cues.

A plausible implication is that future foundation models for geospatial reasoning will require not only data scale, but also explicit exposure to spatially-structured, cross-view, and orientation-labeled corpora for robust performance.

5. Limitations, Error Analysis, and Research Directions

Empirical evaluation exposes fundamental limitations in current LMMs:

• Persistent difficulty with pose estimation derives from a lack of mechanisms for explicit 3D or perspective projection, impairing the establishment of geometric correspondences between ground and satellite views (e.g., matching façades or street-grid layouts).
• Smaller models exhibit strong prior biases, defaulting to specific directions regardless of image content.
• All tested architectures implicitly reason about geometry, with no access to geometry-aware modules or auxiliary sensors (e.g. LiDAR or depth maps).

Recommended directions for overcoming these constraints include:

• Extension of datasets to include multi-season, multi-sensor (including IR), and time-of-day variations to induce more robust, bias-resistant models.
• Integration of geometry-aware modules and explicit positional encodings tied to global coordinates or local scene graphs.
• Pretraining on synthetic, camera-parameterized renderings to provide ground-truth pose and projection supervision.
• Multimodal fusion, including LiDAR, depth, and panoramic video, with contrastive losses across cross-view pairs.

6. Representative Protocols and Examples

GeoX-Bench covers a spectrum of granular QA examples reflecting real cross-view challenges:

Task Type	Input Example	Model Output Type
Pose Estimation	Sat tile (512×512), 4 FoV crops, N/E/S/W	“North”, “East”, etc.
Localization (with prior)	Tile at (φ₀,λ₀), ground image	“Yes”/“No”
Cross-Map Retrieval	Ground image, four candidate sat tiles {T₁,…,T₄}	“3”
Intra-Map Localization	Ground image in tile T	“NE”, “SW”, etc.

This breadth ensures the benchmark captures the core reasoning and correspondence challenges anticipated in real-world multi-source navigation, mapping, and environment understanding settings.

7. Data Availability and Reproducibility

GeoX-Bench releases all data, code, and evaluation scripts under an open-source license, accessible via [https://github.com/IntMeGroup/GeoX-Bench]. Resources include:

• Raw panoramic–satellite pairs and full QA sets
• Train/benchmark splits, dataloaders, and template scripts
• Sample model evaluation results and leaderboard
• Documentation for benchmarking and instruction tuning workflows

Such reproducibility provisions are critical for sustained methodological progress and rigorous comparison across the evolving landscape of geospatial AI models (Zheng et al., 17 Nov 2025).

---

GeoX-Bench serves as a reference standard for the empirical evaluation of geospatial reasoning in large multimodal models, simultaneously revealing current limitations of state-of-the-art approaches (notably in fine-grained geo-pose estimation) and charting priority directions for research and development within this fast-growing subfield.