Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Abstract: Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems for tasks such as image geolocation and spatial reasoning. In this work, we analyze the geospatial representations acquired by three model families: vision-only architectures (e.g., ViT), vision-LLMs (e.g., CLIP), and large-scale multimodal foundation models (e.g., LLaVA, Qwen, and Gemma). By evaluating across image clusters, including people, landmarks, and everyday objects, grouped based on the degree of localizability, we reveal systematic gaps in spatial accuracy and show that textual supervision enhances the learning of geospatial representations. Our findings suggest the role of language as an effective complementary modality for encoding spatial context and multimodal learning as a key direction for advancing geospatial AI.

• The paper demonstrates that textual supervision significantly enhances geospatial encoding in VLMs, achieving R² values up to 0.88 on landmark-rich clusters.
• It employs systematic linear probing with ridge regression on summary tokens to assess spatial information across vision-only and vision-language architectures.
• The study reveals that geospatial features are concentrated in compact, manipulable subspaces, enabling controlled model steering and targeted debiasing.

Textual Supervision Drives Geospatial Representation Emergence in Vision-LLMs

Introduction

The paper "Textual Supervision Enhances Geospatial Representations in Vision-LLMs" (2606.07172) addresses the mechanisms by which geospatial information is encoded in the latent spaces of both vision-only and vision-LLMs (VLMs). The analysis spans a range of model families, covering standard vision-only architectures (ViTs, DINOv2, MAE), vision-LLMs (CLIP, MetaCLIP), and recent end-to-end multimodal foundation models (LLaVA, Qwen2.5, Gemma). The work’s central thesis is that textual supervision, even absent explicit geospatial annotation, is highly effective at driving the formation of semantically robust geospatial representations within VLMs. This hypothesis is interrogated via large-scale, systematic linear probing on internal representations, utilizing datasets with varying degrees of localizability, and directly comparing performance with and without prompts referring to spatial reasoning.

Probing Methodology for Geospatial Representation

The researchers formalize geospatial representation as the degree to which a model’s internal activations (at various layers) contain linearly decodable information about an image’s physical location (latitude and longitude). To quantify this, they employ linear probing using ridge regression on summary tokens (the [CLS] token or equivalent summary embedding) from each layer of the model, regressing directly onto ground-truth geocoordinates. This approach operationalizes geospatial knowledge extraction in a model-agnostic, interpretable way.

Figure 1: Schematic of the linear probing setup fitting ridge regression models on summary embeddings to predict geolocation.

Through systematic evaluation of layer-wise probing $R^2$ performance across a comprehensive suite of models and data clusters, the study characterizes where, when, and to what extent geospatial representations emerge. This is supported by careful sampling and balancing from the YFCC100M and Google Landmarks datasets, coupled with unsupervised semantic clustering to allow interpretable, subject-specific performance analysis.

Empirical Findings: The Role of Textual Supervision

A central result is the clear, quantitative superiority of vision-LLMs over vision-only models in terms of linearly decodable geospatial signal across all localizable clusters.

Figure 2: Heatmap of probe $R^2$ on diverse image clusters for all models; VLMs consistently outperform vision-only models, especially on localizable clusters.

Average $R^2$ for vision-only models rarely exceeds 0.3, even at large scale (e.g., DINOv2-giant, Web-SSL DINO-7B), while smaller CLIP-base models match or outperform them, and foundation VLMs such as LLaVA and Qwen2.5-VL-3B achieve $R^2$ values up to 0.8 on clusters containing landmarks and structured scenes. Notably, model scale alone within vision-only training is insufficient to close the gap to VLMs; for instance, the Web-SSL DINO-7B (7B parameters) underperforms MetaCLIP-huge (600M parameters) when both are trained on the same image corpus, attributing the difference to the presence of language supervision.

Analysis of cluster-localizability reveals that scenes with strong correlation to geographic context—such as landmarks, streets, and architectural features—are the most amenable to geospatial decoding, regardless of model. VLMs, however, show the additional capability to extract signal from text-rich clusters (signs, billboards), implicating the joint image-text training objective as an amplifier for multi-modal geospatial grounding.

Figure 3: Visualization of model-specific localizability across clusters, highlighting VLMs’ ability to localize both landmarks and signage-rich clusters.

Layerwise and Prompt-Driven Analysis

Layerwise probing reveals that vision-only models progressively accumulate geospatial information up to the last layer. In contrast, VLMs exhibit a markedly different pattern: absent a geospatial prompt, geospatial signal increases initially, then either stagnates or decays at higher language-model layers (Gemma, Qwen2.5, LLaVA), indicating that geospatial evidence is likely deprioritized unless explicitly solicited.

The addition of explicit geospatial prompts (e.g., “Guess the latitude and longitude of this image.”) reverses this decay. Under prompting, $R^2$ remains stable or increases through the latter VLM layers, reaching as high as 0.88 for Qwen2.5-VL-3B on the Landmarks dataset.

Figure 4: Layerwise probe $R^2$: (a) Unprompted, VLMs’ late layers lose geospatial signal. (b) Prompting with geospatial tasks stabilizes or improves late-layer performance.

This finding directly supports the hypothesis that geospatial and linguistic representations are entangled in the latent space and that prompt-driven activation is essential for robust propagation of spatial information to the output head.

Compactness and Manipulability of Spatial Subspaces

Dimensional ablation studies show that geospatial information is highly concentrated: for all models, top 40% of embedding dimensions suffice to recover nearly all available geospatial $R^2$ signal, and for Qwen2.5-VL, top 10% achieve more than 90% of maximal performance.

Figure 5: Predictive $R^2$ as a function of retained embedding dimensions, demonstrating compact spatial subspaces.

Taking this further, the paper demonstrates that targeted feature editing in the model’s latent space offers controllability. Explicitly swapping the top geospatial dimensions from one image to another within a VLM (Qwen2.5-VL-3B) leads to consistent changes in generated text indicating the new location (e.g., moving a pyramid from Egypt to “Rome, Italy” in the output), while non-geospatial edits have little effect or destabilize generation.

Figure 6: Editing geospatial latent dimensions causes altered location prediction in VLM-generated text for a given image.

Implications and Theoretical Considerations

The study underscores that language—not model scale—is the key accelerator for geospatial structure emergence in current architectures, confirming that multimodal grounding imposes strong biases favoring world-structure encoding even without explicit location supervision. The finding that compact, semantically manipulable subspaces exist for geospatial reasoning bears on both interpretability and model steering applications.

Importantly, the presence of robust geospatial representations in VLMs, acquired without location annotation, implies both opportunities and risks:

• Sample-Efficient Transfer: Downstream applications, such as image-based country classification, benefit from these pre-existing representations, as shown by the alignment between probing scores and fine-tuning results.
• Fairness and Privacy: Geospatial performance is concentrated in data-rich (Western, urban) regions and is weaker elsewhere, mirroring training-set geography and raising equity concerns. The capacity to extract detailed location from images, including via latent editing, introduces new dimensions to privacy and surveillance risk, motivating regulatory scrutiny.
• Model Steering: The ability to identify and manipulate geospatial features within the latent space signals potential for controlled generation, adversarial robustness, and targeted debiasing.

Conclusion

Textual supervision in vision-LLMs creates robust, linearly decodable geospatial representations, concentrated in compact latent subspaces and manipulable via prompt- or feature-level intervention. This multimodal grounding effect is not fully replicable by scaling vision-only models in isolation, underscoring the unique value of text in world-model construction. As model capabilities advance, the theoretical and practical intersection of geospatial AI, privacy, and model interpretability will become increasingly central. Future directions should include extending analysis to satellite/remote-sensing imagery, understanding the transferability of geospatial features to new tasks, and developing mechanisms for privacy-preserving VLM deployment.