IMAGEO-Bench Geolocalization Benchmark

• IMAGEO-Bench is a benchmark that evaluates image geolocalization in multimodal LLMs by requiring structured predictions with latitude-longitude, addresses, and confidence scores.
• It combines three complementary datasets—global street scenes, U.S. POIs, and a private set—to test performance under varied visual anchors and reduce contamination risk.
• The evaluation uses metrics like identification rate, great-circle distance, and regression diagnostics to quantify geographic precision, computational cost, and geospatial bias.

IMAGEO-Bench is a benchmark for evaluating image geolocalization ability in LLMs and multimodal LLMs: given an input image, a model must infer where it was taken and return a structured geographic prediction together with a reasoning trace. The benchmark was introduced to move beyond top-line localization accuracy toward a broader evaluation of geographic precision, geospatial bias, computational cost, and diagnostic reasoning behavior, using three complementary datasets that combine global street scenes, U.S. point-of-interest imagery, and a private held-out image collection (Li et al., 3 Aug 2025).

1. Benchmark objective and task formalization

IMAGEO-Bench targets image geolocation as a structured prediction problem. The model receives an image $\mathbf{x}$ and is asked to produce

$G = (C,\;A,\;(\phi,\lambda)),$

where $(\phi,\lambda)$ are predicted latitude and longitude in decimal degrees, $A$ is a structured textual address, and $C \in \{1,2,3,4,5\}$ is the model’s self-reported confidence score. The optimization target is the great-circle distance between predicted and true coordinates, written in the paper as minimizing $d((\phi,\lambda),(\phi^{\star},\lambda^{\star}))$ (Li et al., 3 Aug 2025).

The benchmark was motivated by several gaps in prior geolocalization evaluation for LLMs. The paper argues that earlier benchmarks were limited or outdated relative to rapidly improving multimodal models, often emphasized only top-line accuracy, and did not systematically measure geographic distance error, geospatial bias, computational cost, or reasoning process. A further concern was data contamination: publicly reused benchmark images may already appear in training corpora, so IMAGEO-Bench explicitly includes a private held-out split to reduce this risk (Li et al., 3 Aug 2025).

A notable design choice is that IMAGEO-Bench does not evaluate only the final answer. It requires a structured JSON output with three parts: image information, reasoning, and reverse geocoding output. The image-information fields describe the scene at a coarse semantic level through environment (indoor/outdoor), scene_type (building/scenery/street/mixed), and setting (urban/suburban/rural/natural). The reasoning block is organized into four evidence classes: landmark recognition, text and signage, cultural indicators, and spatial context. The reverse-geocoding block contains the final confidence score, address, and coordinates. This design supports both geolocation scoring and post-hoc diagnosis of model behavior.

2. Dataset composition and complementarity

The benchmark is built from three datasets: Dataset-GSS, Dataset-UPC, and Dataset-PCW. Their roles are deliberately different. GSS provides global, cue-rich street imagery; UPC provides U.S.-wide but often ambiguous POI imagery; and PCW provides private unseen images intended to reduce contamination risk (Li et al., 3 Aug 2025).

Dataset	Final scale	Distinctive role
Dataset-GSS	6,152 images	Global street-level outdoor imagery
Dataset-UPC	2,929 images	U.S. POI crowdsourced imagery
Dataset-PCW	220 images	Private, held-out original photos

Dataset-GSS is derived from the NUS Global Streetscapes dataset, originally about 10 million streetscape images from Mapillary and KartaView spanning 688 cities worldwide. IMAGEO-Bench starts from the manually labeled subset with all 8 contextual labels, retains 8,568 images, filters to images with quality = good, reflection = no, and glare = no, leaving 6,178 images, and then manually removes images lacking sufficient geographic cues. The final set contains 6,152 images from 396 cities across 123 countries. It is overwhelmingly outdoor, street-oriented, and urban, making it the benchmark’s most cue-rich split. The appendix reports 5,601 daytime images, 319 dusk/dawn images, and 232 night images; 4,638 clear-weather images; and model-extracted feature counts of 6,144 outdoor scenes, 4,636 street scenes, and 4,350 urban scenes (Li et al., 3 Aug 2025).

Dataset-UPC is built from a Google Maps POI dataset released by UC San Diego. The source dataset contains 4,963,111 unique POIs across the United States. IMAGEO-Bench uses a multi-stage stratified sampling strategy over 17 POI categories, selecting 3 POIs per category for each U.S. state, yielding a candidate pool of 4,325 POIs. Entries with broken image URLs are removed, as are advertisement images and images containing identifiable human faces. The final benchmark contains 2,929 images spanning all 50 states and Washington, D.C. Compared with GSS, UPC is substantially harder: many images are indoors, building-focused, suburban, or otherwise weakly geocoded. Appendix counts report 2,400 outdoor and 522 indoor images; 1,719 building scenes; and a suburban-heavy setting distribution with 1,675 suburban examples (Li et al., 3 Aug 2025).

Dataset-PCW is the benchmark’s private held-out set. It consists of 220 original photo–address pairs collected directly by the authors, with diverse natural, rural, and urban scenes; sunny, cloudy, and snowy conditions; and both daytime and nighttime imagery. Its primary purpose is to reduce overlap with public benchmark sources and likely training corpora. The paper notes a minor inconsistency: appendix model-feature counts sum to 270, but the main text and performance tables consistently introduce PCW as 220 images. The benchmark description treats 220 as the intended size (Li et al., 3 Aug 2025).

Taken together, the three datasets create a spectrum from globally distributed street scenes to domestically constrained but ambiguity-heavy POIs to private unseen imagery. This suggests that IMAGEO-Bench is not only measuring whether a model can identify famous landmarks, but whether it can sustain geolocation performance under progressively weaker visual anchors.

3. Evaluation protocol and metrics

IMAGEO-Bench evaluates six dimensions: identification rate, geographic accuracy, distance error, computational cost, geospatial bias, and reasoning quality. Identification rate is defined as the percentage of images for which the model returns a complete and correctly formatted JSON. The paper defines this quantity but does not report a dedicated identification-rate table in the provided text (Li et al., 3 Aug 2025).

Geographic accuracy is reported at different administrative levels depending on dataset. The benchmark uses country accuracy on GSS and PCW, state accuracy on UPC, and city accuracy on all three datasets. These are semantic correctness measures over discrete address components rather than coordinate errors. Distance error is measured using great-circle distance in kilometers via the Haversine formula:

$$d = 2r \arcsin \left(\sqrt{ \sin^2(\Delta\phi/2) + \cos(\phi_p)\cos(\phi_t)\sin^2(\Delta\lambda/2)} \right),$$

with

$$\Delta\phi = \phi_t - \phi_p,\qquad \Delta\lambda = \lambda_t - \lambda_p,$$

and Earth radius $r = 6{,}371$ km. The paper emphasizes both mean and median distance, noting that the error distribution is heavy-tailed, so median distance is especially informative (Li et al., 3 Aug 2025).

The benchmark also tracks computational cost through average token usage and average monetary cost per image query. This allows cost–performance comparisons across API-accessed models. The reasoning dimension is not scored through a human rubric; instead, it is assessed indirectly through the structure of the reasoning outputs, regression diagnostics, and lexical analyses of the clues invoked by each model.

The experimental setup evaluates 10 multimodal LLMs accessed through their designated APIs: o3, gpt-4.1, gpt-4.1-mini, gemini-1.5-pro, gemini-2.5-flash, gemini-2.5-pro, claude-3.5-haiku, llama-3.2-11b, llama-3.2-90b, and llama-4-17b. The prompting protocol is standardized and is not pure zero-shot: it uses a system prompt, a main prompt requesting step-by-step chain-of-thought reasoning, a one-shot example based on the Empire State Building in New York, and a strict JSON schema. All models are queried at temperature $0.7$, and the benchmark explicitly prohibits external APIs and GPS metadata, making it a closed-book geolocation task from pixels alone (Li et al., 3 Aug 2025).

4. Diagnostic modeling and reasoning analysis

A distinctive feature of IMAGEO-Bench is its post-hoc diagnostic layer. From each model’s JSON output, the authors extract one-hot encoded categorical predictors based on the coarse semantic tags: environment type, scene category, setting, and landmark presence. These predictors are then used in two regression analyses. For city-level correctness, the paper fits a logistic regression

$G = (C,\;A,\;(\phi,\lambda)),$0

where $G = (C,\;A,\;(\phi,\lambda)),$1. For distance error, it fits a ridge regression on log-distance,

$G = (C,\;A,\;(\phi,\lambda)),$2

with $G = (C,\;A,\;(\phi,\lambda)),$3 and regularization

$G = (C,\;A,\;(\phi,\lambda)),$4

These models are not part of the benchmark score itself; they quantify how predictive the model’s own scene descriptors are of downstream geolocation success (Li et al., 3 Aug 2025).

The diagnostic results indicate that successful geolocalization is strongly associated with urban settings, outdoor environments, street-level imagery, and identifiable landmarks. Conversely, indoor scenes, rural settings, natural environments, and scenery without explicit anchors are associated with poorer performance. The paper interprets this as evidence that current models rely heavily on surface semantic anchors such as signage, roads, landmarks, and dense infrastructure rather than holistic place understanding.

The regression analyses also expose model-specific anomalies. Claude-3.5-haiku is reported as a landmark outlier: it labels landmark = yes on 99.67% of images, versus 41.77% for all other models combined. The paper interprets this as over-identification of landmarks, which weakens the predictive value of that feature for Claude. On the held-out PCW set, llama-3.2-90b shows relatively strong alignment between its semantic tags and outcomes, achieving the highest $G = (C,\;A,\;(\phi,\lambda)),$5 among models in the ridge analysis despite not being top-tier in raw geolocation performance (Li et al., 3 Aug 2025).

The benchmark supplements regression with word-cloud analyses of reasoning traces. For strong models, especially o3, the most frequent lexical cues concern traffic and regulatory infrastructure: “sign,” “street,” “road,” “lane,” and “traffic light.” Color terms such as “red,” “white,” “yellow,” “blue,” and “green” also appear frequently, suggesting indirect reliance on road-marking conventions and sign-color standards. The paper argues that vegetation, climate, and terrain are comparatively underused. This suggests a reasoning style dominated by discrete, text-rich anchors rather than environmental synthesis.

5. Empirical results and observed biases

Across the benchmark, closed-source models substantially outperform open models, though open models are far cheaper. On Dataset-GSS, gemini-2.5-pro is the strongest model by city and country accuracy, reaching 44.1% city accuracy and 92.4% country accuracy, with a median distance error of 4.2 km. o3 follows at 42.0% city accuracy and 88.7% country accuracy, with 8.2 km median distance. gemini-2.5-flash reaches 41.7% city accuracy and 91.2% country accuracy. An important nuance is that gemini-1.5-pro, despite lower semantic accuracy, achieves the lowest mean distance on GSS at 161.4 km and an even lower median distance of 3.5 km, indicating fewer catastrophic long-tail failures on average (Li et al., 3 Aug 2025).

On Dataset-UPC, all models perform markedly worse, reflecting the ambiguity of indoor, suburban, and building-heavy POI imagery. gemini-2.5-pro again leads by city and state accuracy at 26.4% and 49.5%, with 150.4 km median distance. o3 reaches 23.9% city and 45.8% state accuracy, while gpt-4.1 reaches 20.7% and 44.2%. As on GSS, gemini-1.5-pro has the best distance statistics despite lower semantic accuracy, with mean distance 406.5 km and median distance 109.5 km (Li et al., 3 Aug 2025).

On the private Dataset-PCW, o3 attains the best city accuracy at 60.8%, gpt-4.1 the best country accuracy at 97.7%, and gemini-2.5-pro the best median distance at 0.7 km. All models perform better on PCW than on UPC in city accuracy, which the paper interprets as evidence that the private set, while unseen, often contains more geolocatable cues than the U.S. POI imagery (Li et al., 3 Aug 2025).

Dataset	Best city/state/country result	Best distance result
GSS	gemini-2.5-pro: 44.1% city, 92.4% country	gemini-1.5-pro: 161.4 km mean, 3.5 km median
UPC	gemini-2.5-pro: 26.4% city, 49.5% state	gemini-1.5-pro: 406.5 km mean, 109.5 km median
PCW	o3: 60.8% city; gpt-4.1: 97.7% country	gemini-2.5-pro: 0.7 km median

The benchmark’s most substantive finding is geospatial bias. Models consistently perform best in high-resource, visually distinctive, and heavily represented regions such as North America, Western Europe, Australia, California, Florida, and Hawaii, while degrading in underrepresented or visually ambiguous areas. The paper treats this as performance disparity rather than reducing it to a single scalar fairness score. The regional heatmaps and qualitative examples show models snapping to famous or overrepresented urban prototypes when evidence is weak: Philadelphia City Hall is misattributed to Honolulu, a Mexican coastal bar is mapped to New York City, a Mexican inlet to Florida, and a rural Luxembourg scene drifts to Germany (Li et al., 3 Aug 2025).

A more specific U.S. failure mode appears in the longitude analysis on UPC. Predicted longitudes cluster into a central-U.S. band rather than following the true longitude diagonal. The paper interprets this as a centralization bias: latitude can often be inferred from climate or vegetation, but longitude is less visually distinguishable within the United States, so models default to plausible mid-continent priors. This reveals a form of geospatial uncertainty handling that is plausible but poorly calibrated.

Confidence estimates are informative within a model but not across models. gemini-2.5-pro tends to report the highest confidence and o3 the lowest. Within some models, including gpt-4.1, o3, and claude-3.5-haiku, higher confidence correlates with better predictions; across models, however, confidence is not a reliable absolute measure of quality.

6. Position within the benchmark landscape and limitations

IMAGEO-Bench occupies a specific niche within Earth-observation and multimodal evaluation. GEO-Bench is a benchmark comprised of six classification and six segmentation tasks for Earth monitoring and is primarily oriented toward downstream transfer evaluation for foundation models (Lacoste et al., 2023). Landsat-Bench is a sensor-specific suite for evaluating geospatial foundation models on Landsat imagery, emphasizing comparability under consistent Landsat inputs and frozen-feature transfer (Corley et al., 10 Jun 2025). EarthSpatialBench benchmarks spatial reasoning capabilities of multimodal LLMs on Earth imagery, with over 325K question–answer pairs focused on distance, direction, topology, and geometry-aware references (Xu et al., 17 Feb 2026). By contrast, IMAGEO-Bench is centered on image geolocalization: it evaluates semantic correctness at country, state, and city levels, coordinate precision through Haversine distance, regional bias, cost, and structured reasoning traces (Li et al., 3 Aug 2025).

The benchmark has several explicit or implicit limitations. The authors note the need for broader geographic and scene diversity. Reasoning analysis is intentionally coarse, using only a few high-level semantic fields in the regression layer. Geospatial fairness is demonstrated descriptively through regional disparities rather than formalized as a single fairness metric. Because external APIs and GPS metadata are disallowed, the benchmark measures raw model capability rather than production geolocation systems that might use retrieval or GIS support. Residual contamination risk remains for the public splits, although the private PCW set partially mitigates it. The explanatory power of the regression diagnostics is also limited: even in the best case, more than 60% of variance in distance errors remains unexplained by the coarse semantic features (Li et al., 3 Aug 2025).

The benchmark’s central implication is that current image geolocalization in multimodal LLMs is genuine but uneven. The strongest systems can often recover broad geography and sometimes achieve kilometer-level precision, yet their performance remains heavily dependent on urban outdoor anchors such as signage and landmarks. This suggests that present-day geolocation behavior is less a general spatial understanding capability than a strong but regionally skewed pattern-recognition competence conditioned on richly represented visual cues.