DualGeo: Two-Stage Geo-Localization Framework

• DualGeo is a two-stage image geo-localization framework that fuses RGB and semantic segmentation features to predict precise geographic coordinates.
• It employs dual-view contrastive learning and bidirectional cross-attention, followed by geo-cognitive refinement using clustering and large multimodal model reasoning.
• Empirical results demonstrate significant gains at street and city scales, validating its effectiveness in handling environmental shifts and spatial outliers.

Searching arXiv for the exact DualGeo paper and closely related geo-localization work. DualGeo is a two-stage framework for worldwide image geo-localization that predicts geographic coordinates for a single photograph taken anywhere on Earth, spanning street, city, regional, national, and continental scales. It addresses two limitations identified in prior work: visual features that are sensitive to environmental variations such as lighting, season, and weather, and retrieval pipelines that lack effective post-processing to filter outlier candidates. The framework therefore combines a geo-representational foundation based on RGB–semantic fusion and GPS alignment with a geo-cognitive refinement stage based on geographic clustering and large multimodal model reasoning (Cui et al., 28 Apr 2026).

1. Task formulation and design rationale

Worldwide image geo-localization predicts geographic coordinates, expressed as latitude and longitude, for a single image. Accuracy is reported across five geodesic thresholds: street ($<1$ km), city ($<25$ km), region ($<200$ km), country ($<750$ km), and continent ($<2{,}500$ km) (Cui et al., 28 Apr 2026).

The motivating observation is that RGB-only matching is brittle under lighting, season, and weather changes, while retrieval-based systems that simply select top-1 or apply naive voting are vulnerable to spatial outliers. DualGeo’s central premise is that semantic segmentation captures scene structure that is more invariant under environmental variation, and that post-retrieval geospatial reasoning can further reject implausible candidates. This leads to a two-stage decomposition: a representation stage that aligns image content with GPS coordinates, and a refinement stage that imposes spatial coherence and leverages world knowledge (Cui et al., 28 Apr 2026).

The framework is organized as follows. Stage 1, termed the geo-representational foundation, fuses RGB and semantic segmentation features via bidirectional cross-attention and aligns both views to GPS coordinates using symmetric dual-view contrastive learning. The resulting embeddings populate a global retrieval database called the World Index. Stage 2, termed geo-cognitive refinement, first re-ranks retrieved candidates by geospatial clustering and then submits curated candidates together with the query image to a large multimodal model for final coordinate prediction (Cui et al., 28 Apr 2026).

2. Geo-representational foundation

The Stage 1 input consists of an RGB image $I$ and its semantic segmentation map $S$. The segmentation map is single-channel with class IDs and is generated by SegFormer-B5 pretrained on ADE20K with 150 classes. To support large-scale dual-view training, the work constructs MP16-SEG, containing 4.12M segmentation maps aligned with the MP16 training set (Cui et al., 28 Apr 2026).

The encoder stack contains three components. The RGB encoder follows the image encoder used in G3, with output dimensionality 768. The GPS encoder follows the GPS encoder design in G3 and maps latitude and longitude to a 768-d embedding. The segmentation encoder is a ResNet-18 adapted to single-channel input, with the first convolution modified and the final fully-connected layer removed to preserve spatial structure; its output is projected to 768 dimensions to match the RGB branch (Cui et al., 28 Apr 2026).

DualGeo fuses the RGB and segmentation token sequences by bidirectional cross-attention. Let $F_{\text{RGB}}$ and $F_{\text{SEG}}$ denote tokenized RGB and semantic features. The RGB branch enhanced by SEG is defined by

$$Q_{\text{RGB}} = W_Q F_{\text{RGB}}, \quad K_{\text{SEG}} = W_K F_{\text{SEG}}, \quad V_{\text{SEG}} = W_V F_{\text{SEG}},$$

$<25$0

$<25$1

$<25$2

The SEG branch is updated analogously by swapping RGB and SEG. The learned projection matrices, residual connections, and LayerNorm stabilize training while allowing each modality to attend to the other (Cui et al., 28 Apr 2026).

This design explicitly couples appearance-driven information with structure-driven information. The paper’s interpretation is that both branches should align to GPS coordinates, rather than treating semantic segmentation as auxiliary preprocessing only. A plausible implication is that the segmentation branch acts as a structural regularizer for retrieval under environmental shift, but the concrete mechanism claimed in the paper is the fusion-and-alignment pipeline itself (Cui et al., 28 Apr 2026).

3. Dual-view contrastive learning and the World Index

After cross-attention, DualGeo aligns both RGB and segmentation embeddings to GPS coordinates through a symmetric, CLIP-style, bidirectional contrastive objective. For a batch of size $<25$3, with image embeddings $<25$4, segmentation embeddings $<25$5, location embeddings $<25$6, cosine similarity $<25$7, and temperature $<25$8, the per-direction loss is

$<25$9

where $<200$0 indexes the positive pair. The similarity matrices are

$<200$1

$<200$2

The total loss is

$<200$3

In experiments, the contrastive temperature is $<200$4 (Cui et al., 28 Apr 2026).

The learned retrieval database, called the World Index, stores the enhanced RGB features $<200$5 for all reference images. The embedding dimensionality is 768, and nearest-neighbor retrieval at inference uses top-$<200$6 candidates. The paper does not specify the index library or normalization strategy; it states that similarity follows the training objective, with cosine similarity implied by $<200$7 (Cui et al., 28 Apr 2026).

Within the overall architecture, the World Index acts as the global retrieval substrate on which subsequent geo-cognitive reasoning operates. This suggests that Stage 1 is not an end point but a calibrated front end for candidate generation: it constructs a geographically aligned feature space that Stage 2 can further regularize (Cui et al., 28 Apr 2026).

4. Geo-cognitive refinement: clustering and multimodal reasoning

Stage 2 consists of two modules: GeoCluster-Rerank and Geo-Thinker. For a query image, the system retrieves the top-$<200$8 most similar coordinates

$<200$9

and also forms the top-$<750$0 most dissimilar set $<750$1, which is later supplied to the large multimodal model (Cui et al., 28 Apr 2026).

GeoCluster-Rerank clusters the retrieved coordinates on the sphere using DBSCAN with the Haversine distance as the metric. Coordinates are converted to radians, the neighborhood radius is specified in kilometers and normalized internally to radians, and the default radius is $<750$2 km unless otherwise stated. The DBSCAN min_samples parameter is not specified in the paper (Cui et al., 28 Apr 2026).

The Haversine distance used for clustering and re-ranking is

$<750$3

where $<750$4 are latitudes in radians, $<750$5 are longitudes in radians, and $<750$6 km (Cui et al., 28 Apr 2026).

After clustering, each candidate receives a cluster label $<750$7, where $<750$8 denotes noise. Among non-noise clusters, the largest cluster is selected as the main cluster and its geometrical center $<750$9 is computed. If at least one cluster exists, its center $<2{,}500$0 becomes the reference; otherwise the fallback is the mean of all candidates,

$<2{,}500$1

The system then computes

$<2{,}500$2

and re-ranks candidates in ascending order,

$<2{,}500$3

producing the reordered set

$<2{,}500$4

The top-1 of $<2{,}500$5 is the refined retrieval prediction (Cui et al., 28 Apr 2026).

Geo-Thinker then invokes the large multimodal model Qwen3-VL-Plus. The prompt contains the query image, the top-$<2{,}500$6 candidates from $<2{,}500$7, and the top-$<2{,}500$8 candidates from the dissimilar set $<2{,}500$9. The value of $I$0 is dataset-dependent: IM2GPS uses $I$1, IM2GPS3k uses $I$2, and YFCC4k uses $I$3. The LMM acts as an arbiter rather than a regressor and returns the final predicted coordinate (Cui et al., 28 Apr 2026).

A common misconception would be to interpret the LMM as replacing retrieval. The paper’s pipeline does not support that interpretation. Instead, the LMM is positioned after retrieval and clustering, and the ablations indicate that LMM-only zero-shot reasoning is strong at coarse scales but weak at street and city scales; the full model combines clustering with LMM reasoning to balance fine and coarse performance (Cui et al., 28 Apr 2026).

5. Training protocol, datasets, and inference procedure

DualGeo is implemented in PyTorch and trained for 10 epochs with AdamW, batch size 256, and initial learning rate $I$4 with dynamic weight decay. RGB, SEG, and GPS embeddings all have dimensionality 768. The retrieval stage uses top-$I$5, clustering uses DBSCAN with $I$6 km, and the LMM stage uses Qwen3-VL-Plus with dataset-specific context sizes as described above. The loss is the symmetric dual-view contrastive objective; the paper states that no triplet loss or additional margins are used (Cui et al., 28 Apr 2026).

Training data are based on MP16 augmented with MP16-SEG, which provides 4.12M semantic segmentation maps generated using SegFormer-B5 pretrained on ADE20K. Test benchmarks are IM2GPS, IM2GPS3k, and YFCC4k. Semantic maps are generated for all test images as well (Cui et al., 28 Apr 2026).

The end-to-end inference procedure is concise. First, the query image $I$7 is segmented to produce $I$8. Second, the RGB and SEG encoders produce $I$9 and $S$0, which are fused by bidirectional cross-attention into $S$1 and $S$2. Third, $S$3 queries the World Index to obtain the top-$S$4 similar set $S$5 and the $S$6 most dissimilar set $S$7. Fourth, GeoCluster-Rerank clusters $S$8 with DBSCAN under the Haversine metric and reorders candidates according to distance from the cluster reference. Fifth, Geo-Thinker forms the LMM prompt from the query image and the selected candidates, and Qwen3-VL-Plus outputs the final latitude and longitude (Cui et al., 28 Apr 2026).

The paper does not specify hardware, training time, parameter counts for the encoders or fusion blocks, runtime, memory usage, or concrete index details such as FAISS parameters. That absence is itself material for implementation assessment because it limits direct systems-level comparison (Cui et al., 28 Apr 2026).

6. Empirical performance, ablations, and limitations

DualGeo is evaluated by the fraction of test queries whose Haversine error falls below fixed thresholds: 1 km, 25 km, 200 km, 750 km, and 2,500 km (Cui et al., 28 Apr 2026).

On IM2GPS, DualGeo reports 23.20% at $S$9 km, 52.32% at $F_{\text{RGB}}$0 km, 65.40% at $F_{\text{RGB}}$1 km, 78.90% at $F_{\text{RGB}}$2 km, and 89.87% at $F_{\text{RGB}}$3 km. The paper summarizes these as gains of 16.58% at street level and 8.77% at city level over the best baselines, while noting slight drops at coarser scales due to the method’s emphasis on precise, local structure (Cui et al., 28 Apr 2026).

On IM2GPS3k, the reported results are 17.25%, 41.47%, 55.76%, 71.71%, and 85.05% at the same thresholds. Relative to the G3 baseline, the gains are +3.60% at street level, +1.29% at city level, and +0.36% at region level, with minor decreases at country and continent scales (Cui et al., 28 Apr 2026).

On YFCC4k, DualGeo reports 27.49%, 36.45%, 45.03%, 61.58%, and 75.92%. Relative to G3, the gains are +14.59% at street level and +1.56% at city level, with decreases at region, country, and continent scales (Cui et al., 28 Apr 2026).

The ablation study on IM2GPS3k isolates the contribution of each component. The RGB-only contrastive baseline $F_{\text{RGB}}$4 achieves 11.11 / 29.83 / 40.61 / 55.86 / 73.41. Adding dual-view contrastive learning without cross-attention ($F_{\text{RGB}}$5) raises performance to 12.25 / 31.97 / 42.64 / 57.59 / 75.19. Adding bidirectional cross-attention ($F_{\text{RGB}}$6) yields 12.81 / 32.30 / 43.24 / 58.79 / 76.37. Adding GeoCluster-Rerank with $F_{\text{RGB}}$7 km ($F_{\text{RGB}}$8) yields 14.81 / 34.50 / 45.41 / 60.19 / 76.78. The LMM-only zero-shot configuration ($F_{\text{RGB}}$9) obtains 8.54 / 33.13 / 52.12 / 70.84 / 83.32. The full model reaches 17.25 / 41.47 / 55.76 / 71.71 / 85.05 (Cui et al., 28 Apr 2026).

These ablations support three conclusions explicitly stated in the paper. First, dual-view learning and cross-attention consistently improve over RGB-only matching. Second, geographic clustering yields a large jump at fine scales. Third, LMM-only reasoning is strong at coarse scales but weak at street and city scales; combining clustering with LMM reasoning gives the best balance (Cui et al., 28 Apr 2026).

Hyperparameter sensitivity further clarifies the behavior of the refinement stage. For the LMM context size $F_{\text{SEG}}$0, performance saturates quickly as $F_{\text{SEG}}$1 increases, which the paper attributes to clustering-based re-ranking already filtering implausible candidates and conserving token budget. For the clustering radius $F_{\text{SEG}}$2 km, small $F_{\text{SEG}}$3 favors fine-grained accuracy at $F_{\text{SEG}}$4 km and $F_{\text{SEG}}$5 km, while larger $F_{\text{SEG}}$6 improves coarse-scale accuracy. The paper identifies $F_{\text{SEG}}$7 km as a good fine-grained default (Cui et al., 28 Apr 2026).

The limitations are also explicit. DualGeo remains challenged by ambiguous, structure-poor scenes and indoor images. The paper notes failures in visually or structurally ambiguous scenes such as many similar beaches or forests, as well as in regions poorly covered in the index. It also identifies dependence on segmentation quality and geographic coverage of the World Index. Finally, the slight drop at coarse scales suggests a bias toward fine-grained structural consistency; the authors propose future work on better GPS encoding and uncertainty modeling, adaptive multi-scale clustering and hierarchical reasoning, joint training with the LMM and segmentation encoder, and efficient global indexing with dynamic update strategies (Cui et al., 28 Apr 2026).