Global Cross-Modal Geo-Localization: A Million-Scale Dataset and a Physical Consistency Learning Framework

Abstract: Cross-modal Geo-localization (CMGL) matches ground-level text descriptions with geo-tagged aerial imagery, which is crucial for pedestrian navigation and emergency response. However, existing researches are constrained by narrow geographic coverage and simplistic scene diversity, failing to reflect the immense spatial heterogeneity of global architectural styles and topographic features. To bridge this gap and facilitate universal positioning, we introduce CORE, the first million-scale dataset dedicated to global CMGL. CORE comprises 1,034,786 cross-view images sampled from 225 distinct geographic regions across all continents, offering an unprecedented variety of perspectives in varying environmental conditions and urban layouts. We leverage the zero-shot reasoning of Large Vision-LLMs (LVLMs) to synthesize high-quality scene descriptions rich in discriminative cues. Furthermore, we propose a physical-law-aware network (PLANET) for cross-modal geo-localization. PLANET introduces a novel contrastive learning paradigm to guide textual representations in capturing the intrinsic physical signatures of satellite imagery. Extensive experiments across varied geographic regions demonstrate that PLANet significantly outperforms state-of-the-art methods, establishing a new benchmark for robust, global-scale geo-localization. The dataset and source code will be released at https://github.com/YtH0823/CORE.

• The paper introduces CORE, a dataset of 1,034,786 street–satellite image pairs with fine-grained text descriptions across 225 regions on six continents.
• The paper presents PLANET, which aligns text-based color, structure, and texture descriptors with parameter-free satellite-image statistics to capture physical attributes beyond global contrastive similarity.
• The paper reports 55.84% World-level R@1 and 59.66% L@150 on CORE, outperforming CrossText2Loc and improving cross-continent transfer by up to 6.21 percentage points.

Overview

This paper addresses two coupled deficiencies in cross-modal geo-localization (CMGL): the absence of a dataset with global geographic coverage and text annotations at scale, and the reliance of existing retrieval methods on coarse global-contrastive alignment that fails to capture fine-grained physical attributes. The authors contribute CORE, a dataset of 1,034,786 cross-view image pairs with fine-grained textual descriptions spanning 225 regions on six continents, and PLANET, a physical-law-aware network that aligns projected textual physical descriptors with parameter-free statistical signatures mined from satellite imagery. Experiments on CORE, on the CVG-Text benchmark, and in cross-continent transfer settings show state-of-the-art retrieval and localization accuracy.

The CORE dataset

CORE is constructed from 517,393 street-level panoramas (2,048 × 1,024 px) drawn from existing datasets, Google Street View, and Baidu Maps, each paired via the Google Maps API with a satellite tile covering 500 m² at 0.1–0.6 m resolution. The data are partitioned into four continent-level subsets—North America, Europe, Asia–Oceania, and Africa–South America—designed both to mitigate regional bias during training and to enable cross-continent generalization testing.

Textual annotations are generated with Qwen3-VL-Plus under a structured prompting framework that enforces a first-person pedestrian perspective and a fixed narrative order (architecture, road geometry, environmental elements), with emphasis on intrinsic physical attributes such as building height, façade material, chromatic properties, and vegetation density. Quality control combines manual expert review of roughly 10% of pairs (~50,000 samples, 10 evaluators, ~300 hours) with heuristic filtering rules derived from observed error patterns (hallucinated landmarks, visual–textual contradictions) applied to the remainder. Descriptions average 125–175 tokens (peaking at 140), and the reported text-similarity matrix shows low cross-sample correlation, indicating that individual descriptions remain discriminative. The authors claim CORE exceeds Geotext-1652 and CVG-Text in both average token length and entropy; this is a plausible consequence of the annotation protocol but rests partly on LVLM-generated text, so the annotations inherit any systematic biases of the generating model.

Relative to prior CMGL resources—Geotext-1652 (campus-scale), CVG-Text (three cities), and GeoLoc (36 regions, ~158k images)—CORE is an order of magnitude larger and roughly six times broader in regional coverage, making it, by the paper's account, the first million-scale dataset dedicated to CMGL.

The PLANET method

PLANET is built on a dual-stream CLIP-ViT-L/14@336px backbone and adds three components. First, an intrinsic physical signature mining module extracts three deterministic, non-parametric statistics from each satellite image: normalized per-channel color histograms (spectral chromaticity), histograms of gradient orientation over an adaptively masked set of salient pixels ($\tau_{rel}=0.15$) capturing geometric structure, and log-transformed Laplacian energy histograms capturing surface texture. The key methodological distinction from prior uses of hand-crafted features is that these statistics serve as supervision targets, not input features. Second, a physical semantic projection module applies learnable query vectors ($Q_{color}$, $Q_{struc}$, $Q_{tex}$) via scaled dot-product attention over the full text-encoder token sequence, producing attribute-specific descriptors $\hat{t}_k$ rather than compressing all information into a single global vector. Third, a consistency contrastive loss $\mathcal{L}_{phy}$ aligns each projected descriptor with the corresponding mined image signature across the batch, using a dedicated temperature $\tau_p$. Training jointly optimizes the standard InfoNCE global alignment $\mathcal{L}_{itc}$ and $\mathcal{L}_{phy}$ with a single hyper-parameter $\lambda$; the three physical branches are deliberately weighted equally ($Q_{color}$0 each), on the argument that color, structure, and texture are orthogonal and equally indispensable.

Benchmark results

On the CORE benchmark (R@1 for retrieval; L@150 for localization accuracy, chosen over the conventional L@50 due to the dataset's spatial breadth), PLANET achieves 55.84% R@1 and 59.66% L@150 at the World level, exceeding the strongest baseline CrossText2Loc by 3.92 and 3.78 percentage points respectively, and leading all four intercontinental subsets by 2.84–4.89% R@1 and 2.86–5.38% L@150. On CVG-Text, PLANET again ranks first in all three cities, with R@1 gains of 2.50% (New York), 2.25% (Brisbane), and 1.67% (Tokyo) over CrossText2Loc, indicating that the physical consistency mechanism transfers to conventional dense-urban settings.

Ablations and generalization

Ablations support three claims. First, each individual physical constraint (color, structure, or texture) improves R@1 by an average of 1.87–2.00% over the physical-free baseline across subsets, and the full model adds a further 1.40–2.16% over the best single-branch variant, consistent with the orthogonality assumption. Second, weight ablations confirm the design choices: a total physical-loss weight of 1.0 is optimal (both 0.3 and 3.0 degrade performance), and the uniform 1:1:1 internal ratio outperforms any configuration biased toward a single attribute. Third, heatmap comparisons show PLANET attending to physically meaningful regions ("red-tiled roofs", "tall, leafy trees") where the baseline attention is diffuse, and UMAP visualizations show tighter intra-class cohesion and cross-modal coupling.

The cross-subset generalization experiments address the paper's own stated concern: whether PLANET's compact latent structure reflects domain-invariant physical laws or overfitting to domain statistics. Training on Subsets 1–3 and testing on Subset 4 (African and South American regions with unpaved roads and informal architecture, a substantial distribution shift), all baselines degrade sharply, but PLANET improves R@1 by absolute margins of 5.19%, 5.74%, and 6.21% over CrossText2Loc for transfers from Subsets 1, 2, and 3 respectively. This is the strongest evidence in the paper that the gains derive from physically grounded rather than domain-specific representations.

Limitations and open questions

The authors identify one explicit limitation: the lack of temporal synchronization between street-view and satellite acquisition means transient elements (vehicles, construction) inject noise into cross-modal alignment, and localization in extreme conditions such as conflict zones or disaster areas remains unresolved despite the robustness conferred by physical signature mining. Two further caveats are implicit in the design. The annotations are LVLM-generated and validated on a 10% sample, so annotation quality for the unchecked remainder depends on the transferability of heuristic filtering rules. And the physical signatures—color histograms, gradient-orientation histograms, Laplacian energy—are heuristic proxies for "physical laws"; whether this specific triplet exhausts the physically informative statistics of satellite imagery, or whether richer spectral and geometric descriptors would yield further gains, is not examined. The paper also leaves open whether the physical consistency paradigm generalizes beyond geo-localization to broader vision-language tasks, which the authors note as future work.

Conclusion

The paper pairs a dataset contribution—CORE, the first million-scale, globally distributed, text-annotated CMGL benchmark—with a methodological contribution, PLANET, which uses parameter-free physical statistics of satellite imagery as supervision anchors for query-based textual attribute projection. The consistent state-of-the-art results on CORE, CVG-Text, and cross-continent transfer settings, together with ablations confirming the orthogonality and balanced weighting of the three physical cues, establish a credible new benchmark and a reproducible training paradigm for globally robust cross-modal geo-localization. The principal open issues are temporal robustness under imagery asynchrony, the completeness of the chosen physical signature set, and the scalability of LVLM-based annotation quality control.