GPSBench: Do Large Language Models Understand GPS Coordinates?

Abstract: LLMs are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench

• The paper introduces GPSBench, a comprehensive benchmark with 57,800 samples across 17 tasks to evaluate LLMs’ geospatial reasoning and GPS coordinate manipulation.
• The paper finds that while most LLMs excel in applied geographic tasks, they significantly struggle with pure geometric coordinate computations, with flagship models achieving over 75% accuracy.
• The paper highlights regional biases, scaling effects, and finetuning trade-offs, underlining limitations in fine-grained spatial cognition crucial for applications like navigation and robotics.

GPSBench: Evaluating LLMs' Geospatial Reasoning

Motivation and Benchmark Design

GPSBench addresses a critical gap in the assessment of LLMs by systematically probing whether current frontier models possess genuine geospatial reasoning capabilities, especially those related to GPS coordinate understanding. Existing benchmarks predominantly focus on small-scale, perceptual spatial tasks and tool-augmented workflows, providing little insight regarding direct coordinate manipulation and world-knowledge integration necessary for real-world applications such as navigation, robotics, and geographic information systems.

GPSBench comprises 57,800 samples across 17 tasks, divided into two tracks:

• Pure GPS Track: Tests geometric coordinate operations (distance, bearing, coordinate transformation, interpolation, spherical geometry), requiring mathematical reasoning over coordinates but not world knowledge.
• Applied Track: Evaluates geographic reasoning tasks (coordinate-to-place mapping, spatial relationships, name disambiguation, proximity, route analysis, terrain classification), demanding integration of world knowledge with spatial computation.

Locations are globally sampled from 18,196 cities in the GeoNames database, representing all continents and diverse regions, mitigating typical dataset regional biases.

Figure 1: Geographic coverage of GPSBench. 18,196 unique locations from GeoNames span six continents, ensuring comprehensive geographic representation.

Experimental Methodology

Fourteen current LLMs—including GPT-5.x, Gemini-2.5, Claude-Haiku-4.5, Qwen3, and Mistral-Large—were evaluated under standardized zero-shot prompting. Metrics include standard accuracy for classification and $1 - \text{MAPE}$ for numerical computations, normalizing across tasks for aggregation.

Models were strictly assessed on intrinsic geospatial capabilities, excluding tool-augmented workflows or specialized prompts. Coordinates in all tasks are grounded in the WGS84 standard; geometric answers are computed using rigorous geodetic formulae.

Main Findings

Applied vs Pure GPS Reasoning

Most models demonstrate higher performance on applied geographic tasks involving world knowledge than on pure geometric coordinate computations (+9.9% mean gap across all models), with only flagship reasoning models (GPT-5.1, Gemini-2.5-Pro) excelling in coordinate computation (84.4%, 76.7% Pure GPS accuracy, respectively). Model families exhibit distinct patterns: GPT and Gemini show relatively balanced strengths, whereas Mistral, Claude, and Qwen heavily favor world-knowledge tasks.

Figure 2: Applied vs Pure GPS Track performance. Most models cluster below the diagonal, favoring Applied tasks, while only GPT-5.1 and Gemini-2.5-Pro excel at Pure GPS computation.

Task Difficulty Stratification

Tasks cluster into three regimes:

• Solved ($>$95%): Name Disambiguation, Bounding Box—solvable via coarse heuristics or deterministic mapping.
• Brittle (25–95%): Distance Calculation, Spatial Patterns—variable model competence, significant performance spread.
• Unsolved ($<$25%): Place Association (city-level), Polygon Area, Coordinate Interpolation—require dense coordinate-to-place mappings or multi-step spherical geometry reasoning, which are not learned robustly.

  Figure 3: Mean accuracy across all models per task, sorted by difficulty, demonstrates stratification into solved, brittle, and unsolved tiers.

Regional and Granularity Bias

Applied tasks exhibit pronounced regional disparities; North America and Western Europe are better represented and have higher Place Association accuracy ($>$10%), compared to South/East Asia ($<$5%), revealing training data bias in world-knowledge encoding. In contrast, pure geometric computation (distance, bounding box, format conversion) shows minimal regional variation (differences $<$10%), confirming computation is region-agnostic.

Figure 4: Regional performance by subregion across all GPSBench tasks. Applied Track exhibits substantial geographic bias; Pure GPS Track remains more uniform.

Hierarchical degradation is evident: country-level Place Association accuracy exceeds 70%, province-level drops to 30–70%, city-level collapses to $<$25% for all models. Fine-grained localization is consistently absent.

Figure 5: Hierarchical degradation in geographic knowledge. All models drop sharply in accuracy when moving from country to city-level resolution.

Robustness to Coordinate Noise

Coordinate noise (10m–1km) does not degrade performance at any granularity, implying LLMs rely on generalized representations, not memorization of training-coordinate mappings.

Figure 6: Models' accuracy is flat across coordinate noise levels, indicating robustness and absence of memorization-based shortcuts.

GPS Augmentation in Downstream Tasks

Explicit GPS coordinate augmentation in external benchmarks (MapEval, Hierarchical Spatial) results in consistent performance improvements (MapEval +6.1%, Hierarchical Spatial +22.7%), especially for spatial relationship queries previously susceptible to hierarchical and alignment bias.

Figure 7: GPS coordinate augmentation improves performance and eliminates certain reasoning biases in downstream geospatial tasks.

Finetuning Impact

Finetuning Qwen3-30B on GPSBench induces trade-offs: geometric computation tasks improve (+4.3% Pure GPS), but world-knowledge tasks degrade (–1.6% Applied track). Strong gains (e.g., Spatial Patterns +56.5%) are offset by severe drops elsewhere (Boundary Analysis –25.2%), confirming that task-specific finetuning may erode pretrained general knowledge.

Figure 8: Finetuning Qwen3-30B yields improved computation but degraded world knowledge, highlighting capability trade-offs.

Model Scaling

Scaling effects are clear: increasing parameter count in Mistral and Qwen3 families correlates with higher accuracy in both tracks (Qwen3 8B→235B: Pure GPS +19.7%, Applied +11.1%).

Figure 9: Scaling analysis for Mistral and Qwen3 model families shows steady improvement with increased scale.

Error Analysis and Regional Bias Visualization

Qualitative analysis reveals prevalent error modes:

• Place Association: substitution for nearby prominent cities, correct coarse-level but incorrect fine-grained mapping.
• Polygon Area: formula stated but not executed.
• Coordinate Interpolation: cumulative rounding errors.
• Format Conversion: precision mismatches and hemisphere label swaps.

  Figure 10: Per-task accuracy heatmaps underscore strong variability, with geometric computation tasks showing greater model-dependent differences.

Regional bias visualization confirms the strongest disparities in applied tasks (Place Association), while computation tasks remain nearly uniform.

Figure 11: Regional accuracy heatmaps for Applied and Pure GPS tracks illustrate bias in knowledge tasks and agnosticism in computation tasks.

Implications and Future Directions

GPSBench demonstrates that current LLMs encode robust coarse world-knowledge and basic geometric computation, but lack dense coordinate-to-location associations and multi-step geodetic reasoning, especially at city- or sub-city granularity. Applied world-knowledge tasks are susceptible to regional training data bias, limiting deployment in underrepresented geographies. Model scaling improves performance, but does not resolve memorization deficiencies at fine levels.

The implications are substantial for AI deployments in navigation, spatial planning, robotics, and emergency response: reliance on LLMs for fine-grained localization or nuanced geospatial relationships remains unsafe. Downstream augmentation with GPS coordinates mitigates some reasoning biases, but does not fundamentally solve limitations in spatial cognition. Finetuning requires deliberate strategy to avoid catastrophic forgetting of world-knowledge.

Future research should explore:

• Continual learning protocols that jointly preserve world-knowledge and add coordinate computation.
• Integration of multimodal spatial data, leveraging maps, imagery, and sensor inputs.
• Cross-lingual geospatial reasoning and inclusion of rural, maritime, and polar regions.
• Model architectures or tokenization schemes optimized for coordinate-value representation and computation.

Conclusion

GPSBench establishes a comprehensive and technically rigorous framework for GPS reasoning evaluation in LLMs, revealing capability stratification, regional bias in world knowledge, absence of memorized coordinate mappings, and trade-offs induced by finetuning. Models encode coarse geographic structure, but fundamental limitations in fine-grained spatial cognition persist. This work informs safer deployment strategies and motivates physically grounded AI research addressing geospatial reasoning deficiencies.

Reference: "GPSBench: Do LLMs Understand GPS Coordinates?" (2602.16105)