VLM2GeoVec: Unified Geographic Embeddings

• VLM2GeoVec is a unified embedding model that integrates visual, textual, geographic, and spatial data into a single representation for robust retrieval and reasoning.
• The architecture uses a modified Qwen2-VL backbone with LoRA adapters to seamlessly fuse images, captions, bounding boxes, and geo-coordinates into one token stream.
• Empirical results on the RSMEB benchmark demonstrate superior performance in scene classification, cross-modal retrieval, and region-level spatial reasoning compared to dual-encoder methods.

VLM2GeoVec refers to a family of methods and a concrete architecture that leverage vision–LLMs (VLMs) for learning universal geographic embeddings capable of interleaving visual, textual, spatial, and semantic modalities. The overarching goal of the VLM2GeoVec concept is to overcome the fragmentation of remote sensing and geo-localization pipelines by providing a single embedding space wherein images, captions, bounding boxes, and geographic coordinates are jointly encoded for unified retrieval and reasoning. This approach is motivated by demonstrated limitations in prior dual-encoder retrieval methods and generative assistants, particularly in region-level spatial reasoning and in the ability to unify fine-grained grounding with scalable search (Aimar et al., 12 Dec 2025).

1. Historical Context and Motivation

Conventional geo-localization and remote sensing workflows rely on dual-encoder models that separately embed visual and textual inputs, then compare them in a shared space through late fusion. Although effective for large-scale retrieval, these architectures struggle to model fine-grained spatial relationships, interleave heterogeneous inputs, or reason about region-level queries without bespoke modules. Generative VLMs excel at zero-shot region interpretation and semantic reasoning but lack scalable retrieval capabilities and alignment with specialized spatial tasks (Aimar et al., 12 Dec 2025). An earlier body of work, including GeoVLM reranking and hybrid VLM+VPR retrieval, established the value of fusing VLM outputs with conventional retrieval or placing reranking pipelines (Dagda et al., 19 May 2025, Waheed et al., 23 Jul 2025). These approaches, however, required multistage processing or external text encoders for pseudo-embedding.

This suggests that the field has converged on the need for unified, single-encoder architectures able to absorb and relate all spatial, textual, and region information at both local and global scales.

2. Model Architecture and Input Representation

VLM2GeoVec embodies a single-encoder paradigm, implemented as a modified Qwen2-VL (ViT-L/14 backbone) that interleaves all modalities—RGB images, natural language, bounding boxes, and serialized geographic coordinates—into a single input token stream. Key aspects include:

• Visual tokens: Images resized to 336×336, tokenized as 14×14 patch embeddings (1,024-dim each).
• Textual tokens: Captions, class labels, and instructions, preprocessed by the VLM tokenizer.
• Geographic coordinates: Latitude/longitude as text ("(lat, lon)"), supporting integration into the language stream.
• Bounding boxes: Normalized to [0,100] and encoded as continuous scalar tokens.
• Instructional prompting: Each input optionally prefixed by instructions relevant to the downstream task.
• Adaptation: All original weights are frozen; task-specific LoRA adapters (rank 8) are inserted into every self-attention and MLP layer, initialized from a universal embedder and updated on remote sensing tasks.

The encoder yields a single 1,024-dimensional vector for any input—image, text, region, or geo-query—enabling cosine-based retrieval, grounding, or classification in a shared space (Aimar et al., 12 Dec 2025).

3. Training Objectives and Losses

VLM2GeoVec is optimized end-to-end with an InfoNCE contrastive loss applied to interleaved multimodal pairs. For a minibatch of $N$ paired examples $\{(q_i, t^+_i)\}$, the loss is:

$$L_i = -\log \frac{\exp[\cos(h_{q_i}, h_{t^+_i})/\tau]}{\exp[\cos(h_{q_i}, h_{t^+_i})/\tau] + \sum_{j\neq i}\exp[\cos(h_{q_i}, h_{t^+_j})/\tau]}$$

where $h_{q_i}$ and $h_{t^+_i}$ are encoder outputs for query and target, and $\tau$ is a learned temperature. Positive pairs are constructed across scene classification (image, label), cross-modal (image, caption), composed region retrieval, grounding, and semantic geo-localization (text+lat/lon, image). In-batch negatives are employed, and GradCache enables an effective batch size of 1,024 on an 8×A100 cluster (Aimar et al., 12 Dec 2025).

4. Benchmarks and Evaluation Protocols

To assess versatility, the RSMEB (Remote-Sensing Multimodal Embedding Benchmark) unifies 21 remote-sensing and geo-localization tasks into a retrieval/ranking format across six meta-tasks:

• Scene classification (6 aerial datasets; top-1 accuracy)
• Cross-modal search (I⇌T and region+modifier→image; Recall@k)
• Compositional retrieval (region+modifier→image)
• Visual question answering (multichoice; P@1)
• Visual grounding/referring expression (image+text→region in set)
• Spatial localization (image+bbox→caption, text+bbox→image)
• Semantic geo-localization (text+lat/lon→image, P@1 over 2,000 geocandidates)

This design exposes the model's capacity to generalize from global scene retrieval to region-level spatial reasoning and semantic geospatial search within a unified vector space (Aimar et al., 12 Dec 2025).

5. Empirical Results and Comparative Analysis

On RSMEB, the 7B-parameter VLM2GeoVec model demonstrates the following performance highlights:

• Scene classification: 77.25% (AID), 64.82% (Million-AID), 44.54% (RSI-CB), outperforming CLIP, GeoRSCLIP, and SkyCLIP on most datasets.
• Cross-modal retrieval: Ranks second overall, surpassing all dual-encoders except for RemoteCLIP.
• Visual grounding (RefExp): P@1 = 32.50% (+20.0pp vs. CLIP 13.15%).
• Spatial localization (Region-caption): 26.56% (+25.5pp vs. CLIP), (GrT2I): 13.70% (+12.7pp vs. CLIP).
• Semantic geo-localization (GeoT2I): 17.80% P@1, more than threefold the prior best result (SkyCLIP 5.10%).

The overall Friedman score across all 21 RSMEB tasks is 1.93, indicating best aggregate performance among tested models (Aimar et al., 12 Dec 2025).

6. Methodological Comparisons and Evolution of VLM2GeoVec

Earlier works (Dagda et al., 19 May 2025, Waheed et al., 23 Jul 2025) instantiated VLM2GeoVec as a "companion feature" obtained through two-stage pipelines—first extracting base image embeddings, then augmenting with VLM-generated descriptions or coarse coordinate priors for reranking. In GeoVLM (Dagda et al., 19 May 2025), zero-shot BLIP-2 VQA responses are structured into natural language, encoded with text-embedding models, and aligned with visual embeddings via a learned reranker. Hybrid architectures fuse VLM priors with place recognition backbones, using clustering or country-based submaps and final haversine-based reranking (Waheed et al., 23 Jul 2025).

The present single-encoder design eliminates these modular dependencies, integrating all modalities from the outset and supporting end-to-end contrastive training, resulting in superior region-level and semantic-geospatial retrieval.

7. Strengths, Limitations, and Future Directions

Strengths

• Single-encoder design enables deep cross-modal and spatial interaction, unifying retrieval, region reasoning, and classification.
• Scalable retrieval across large candidate archives.
• High performance on region-level and semantic-geo tasks, surpassing previous dual-encoder and generative models.
• Instruction conditioning increases robustness to prompt or query variation.

Limitations

• Restricted to single-view RGB imagery and text; lacks SAR, multispectral, LiDAR, or temporal stream support.
• Geographic coordinates are encoded as text, limiting learned spatial topology.
• Dependence on in-batch negatives necessitates extensive training data, especially for rare query types.

Future directions proposed include direct encoding of geodesic spatial relationships, multimodal extension (SAR, multispectral), and hybrid contrastive–generative pretraining (Aimar et al., 12 Dec 2025). This suggests further convergence between universal embedding models and open-ended VLM-based reasoning for remote sensing and planetary-scale geo-localization.