Turning Generators into Retrievers: Unlocking MLLMs for Natural Language-Guided Geo-Localization

Abstract: Natural-language Guided Cross-view Geo-localization (NGCG) aims to retrieve geo-tagged satellite imagery using textual descriptions of ground scenes. While recent NGCG methods commonly rely on CLIP-style dual-encoder architectures, they often suffer from weak cross-modal generalization and require complex architectural designs. In contrast, Multimodal LLMs (MLLMs) offer powerful semantic reasoning capabilities but are not directly optimized for retrieval tasks. In this work, we present a simple yet effective framework to adapt MLLMs for NGCG via parameter-efficient finetuning. Our approach optimizes latent representations within the MLLM while preserving its pretrained multimodal knowledge, enabling strong cross-modal alignment without redesigning model architectures. Through systematic analysis of diverse variables, from model backbone to feature aggregation, we provide practical and generalizable insights for leveraging MLLMs in NGCG. Our method achieves SOTA on GeoText-1652 with a 12.2% improvement in Text-to-Image Recall@1 and secures top performance in 5 out of 12 subtasks on CVG-Text, all while surpassing baselines with far fewer trainable parameters. These results position MLLMs as a robust foundation for semantic cross-view retrieval and pave the way for MLLM-based NGCG to be adopted as a scalable, powerful alternative to traditional dual-encoder designs. Project page and code are available at https://yuqichen888.github.io/NGCG-MLLMs-web/.

• The paper introduces a parameter-efficient adaptation strategy employing LoRA modules and InfoNCE loss to tune MLLM embeddings for cross-modal geo-localization.
• The approach achieved a 12.2% improvement in Text-to-Image Recall@1 on GeoText-1652 and top-1 performance on 5 of 12 CVG-Text subtasks with minimal trainable parameters.
• The study demonstrates that leveraging [EOS] token pooling and focused adaptation strategies preserves multimodal priors while significantly boosting retrieval accuracy.

Parameter-Efficient Adaptation of Multimodal LLMs for Natural Language-Guided Geo-Localization

Motivation and Problem Formulation

Natural-language Guided Cross-view Geo-localization (NGCG) is a critical extension of standard cross-view geo-localization (CVGL) tasks, enabling retrieval of geo-referenced satellite imagery using only textual descriptions of ground scenes. This capability is essential in scenarios where visual input is unavailable or impractical, such as emergency response or large-scale mapping platforms. Conventional NGCG solutions predominantly rely on CLIP-style dual-encoder vision-LLMs (VLMs), which separately encode ground text and satellite imagery before projecting them into a shared embedding space for contrastive retrieval. However, these architectures inherently limit modality interaction and demand elaborate alignment modules to compensate for weak cross-modal generalization (Figure 1).

Figure 1: Structural comparison of CLIP-style dual encoders and the unified MLLM framework emphasizing joint modality processing.

The recent emergence of Multimodal LLMs (MLLMs) has revealed substantial semantic reasoning potential for visual-language tasks. MLLMs jointly process visual and textual input in a unified architecture, facilitating rich cross-modal interactions. However, they are optimized for generative or instruction-following objectives, not retrieval, resulting in embeddings that lack the discriminativity required for NGCG. Empirical validation—such as zero-shot application of InternVL3.5-1B yielding only 0.92% R@1 in CVG-Text New York—underscores this limitation.

Framework Architecture and Adaptation Mechanism

The proposed framework leverages a pre-trained parameter-efficient MLLM backbone as a unified feature encoder, adapting its latent space for cross-modal retrieval via LoRA modules and InfoNCE loss. Both text and satellite visual data are processed through the shared weights of the MLLM, extracting final-layer hidden states, which are aggregated using several strategies ([EOS]-token pooling, query-based pooling, and average pooling). The adaptation mechanism employs LoRA to inject trainable low-rank updates across attention and MLP layers, preserving original reasoning ability while enabling targeted task adaptation. Crucially, the InfoNCE loss with fixed temperature aligns text and image embeddings for discriminative cross-modal retrieval.

Figure 2: The adaptation pipeline: MLLM encoding, LoRA-injected parameter-efficient tuning, and InfoNCE-based contrastive alignment for NGCG.

Empirical Results and Numerical Analysis

Extensive evaluation was performed on GeoText-1652 and CVG-Text benchmarks. The framework yielded SOTA performance, including a 12.2% absolute improvement in Text-to-Image Recall@1 on GeoText-1652, and top-1 performance in 5 out of 12 CVG-Text subtasks, with strong parameter efficiency—only 9M trainable parameters versus hundreds of millions in classic dual-encoder baselines.

Key ablation studies highlight the following:

• Model scale: Even smaller MLLMs (SmolVLM-256M) retain substantial pretrained knowledge and deliver competitive retrieval with minimal parameter overhead.
• Feature aggregation: [EOS] token pooling outperformed query-based and average pooling, reflecting the intrinsic contextual summarization of autoregressive models.
• Adaptation strategy: LoRA adaptation consistently surpassed full fine-tuning, particularly on smaller backbones (SmolVLM-500M), due to improved generalization and preserved multimodal priors.
• LoRA scaling: Higher update magnitude ($\alpha$) was required for robust task adaptation, maximizing retrieval scores at $\frac{\alpha}{r}=8$.
• InfoNCE temperature: Fixed $\tau=0.03$ outperformed learnable temperature, stabilizing gradient estimation and enhancing discriminativity.

Efficiency analysis demonstrates a new performance-latency-mem frontier. The smallest model (Ours-S) required only 683 MB GPU memory and minimal latency, enabling potential deployment on edge devices.

Qualitative Evaluation and Failure Analysis

Qualitative results on CVG-Text illustrate both successful and failed retrievals. The model accurately integrates architectural and spatial cues in diverse queries, but struggles in ranking when candidate images share similar visual characteristics or textual descriptions lack distinctive spatial references.

Figure 3: Qualitative visualization of text-to-satellite retrieval, highlighting both correct matches (green) and ranking errors (red) across descriptive ambiguity and visual similarity.

Theoretical and Practical Implications

The study reveals several theoretical implications. First, MLLMs possess latent multimodal priors that can be efficiently activated for cross-modal retrieval with minimal parameter overhead. Second, parameter-efficient fine-tuning (LoRA) offers superior generalization and robustness compared to full fine-tuning, primarily due to regularization and retention of pretrained knowledge.

Practically, the approach unlocks scalable, deployable NGCG solutions, overcoming the memory and tuning complexity of dual-encoder baselines. The fixed temperature contrastive objective and use of contextual summarization tokens offer actionable design guidelines for future NGCG frameworks. The methodology directly enables practical language-guided search, navigation, and mapping at scale.

Future Directions

The results motivate research in several areas:

• Knowledge distillation and quantization for further latency reduction and edge deployment.
• Improved text-image alignment via leveraging richer spatial context and hierarchical pooling mechanisms.
• Expansion to generalized multimodal retrieval tasks and integration into autonomous systems for natural language-guided tasks.

Conclusion

This work establishes MLLMs as robust, scalable foundations for NGCG, providing strong empirical and theoretical evidence for parameter-efficient adaptation. Systematic investigation of adaptation variables delivers precise practical guidelines for deploying MLLM-based cross-modal retrieval systems, setting new benchmarks and enabling broader adoption in geo-localization and multimodal search domains.