VG-SSL: Geo-localization with Self-supervision

• The paper presents a unified self-supervised framework that replaces hard negative mining with geo-aware pair sampling, achieving competitive visual retrieval performance.
• The methodology integrates six SSL methods (e.g., SimCLR, MoCov2, BYOL) within a retrieval pipeline using a ResNet-50 backbone with NetVLAD aggregation to optimize Recall@1 on varied datasets.
• The study demonstrates efficiency improvements and robust cross-domain performance, particularly on challenging benchmarks like Tokyo 24/7, by leveraging a geo-related pair construction strategy.

VG-SSL, whose correct name is Visual Geo-localization with Self-supervised Learning rather than “VGG-SSL,” is a unified self-supervised framework for visual geo-localization (VG) that benchmarks multiple SSL paradigms within a retrieval-oriented training pipeline (Xiao et al., 2023). In this setting, VG, also known as Visual Place Recognition, is the task of estimating the geolocation of a query image by matching it to a georeferenced database via image retrieval. The framework was introduced to address two linked objectives: eliminating the need for Hard Negative Mining (HNM) while maintaining or surpassing its performance, and providing a standardized, scalable benchmark for studying how diverse SSL objectives behave on VG across multiple datasets. Its central mechanism is a geo-related pair construction strategy, GeoPair, which supplies informative positive pairs from geographic proximity and includes negatives from the database without HNM (Xiao et al., 2023).

1. Problem setting and conceptual scope

Visual geo-localization is critical for robotics and autonomous vehicles because it enables robust localization under changes in viewpoint, illumination, weather, and seasonal conditions (Xiao et al., 2023). Contemporary VG systems typically rely on learned global descriptors produced by a feature extractor and an aggregation module such as NetVLAD, followed by retrieval over a large geo-referenced image database. In supervised VG pipelines, training commonly uses triplet losses with HNM, but HNM incurs significant computational and memory costs on large-scale datasets (Xiao et al., 2023).

VG-SSL was formulated as a direct response to this bottleneck. Rather than mining hard negatives in feature space, it integrates SSL objectives directly into the VG pipeline and replaces mining with a geographically structured sampling mechanism. The framework instantiates and systematically benchmarks six SSL methods for VG: SimCLR, MoCov2, BYOL, SimSiam, Barlow Twins, and VICReg (Xiao et al., 2023). The paper also reports the standard Triplet Margin Loss as a supervised VG baseline.

The paper characterizes the framework as the first benchmarking study of SSL in VG and emphasizes that the benchmark is intended both as a methodological unification and as a computationally lighter alternative to HNM-based training (Xiao et al., 2023). A plausible implication is that VG-SSL should be understood not merely as a new loss or model, but as an experimental protocol for comparing SSL objectives under a common retrieval architecture and geo-aware sampling regime.

2. GeoPair and the elimination of hard negative mining

The key design element in VG-SSL is GeoPair, a geo-related pair construction and sampling mechanism defined from image geolocations. Each image $I$ is associated with a geolocation $g$ given by latitude and longitude, and geographic distance is denoted by $d(g_i, g_j)$ (Xiao et al., 2023). Positives and negatives are defined by distance thresholds:

• Positive radius: $r_{pos} = 10 \text{ m}$, with

$$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$

• Negative radius: $r_{neg} = 25 \text{ m}$, with

$$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$

For each epoch, the framework samples $m_q$ query images $\{I_q\}$ and, for each query, a positive $I_{k^p} \in P(q)$ uniformly. It then samples $g$0 negatives from a global negative pool, where $g$1 is the Database Negative Ratio. Two types of pairs are constructed: geo-positive pairs $g$2 and identical “database negative” pairs $g$3 (Xiao et al., 2023). The resulting pair set is

$g$4

Sampling is uniform within each set; no distance weighting $g$5 is applied, and no HNM is used (Xiao et al., 2023).

This construction serves two functions. First, it gives SSL objectives a geo-semantic notion of positive correspondence derived from geographic proximity rather than synthetic augmentation alone. Second, it ensures that many database images that would otherwise be unused due to missing corresponding queries still participate in training via identical database negative pairs (Xiao et al., 2023). Importantly, negatives are never selected by nearest-neighbor mining in feature space.

The paper gives an explicit complexity comparison. With full-database HNM, extraction is $g$6 and matching is $g$7; even partial HNM requires $g$8. By contrast, VG-SSL uses only positives and identical database negatives, with extraction cost $g$9 and matching cost $d(g_i, g_j)$0 (Xiao et al., 2023). This is the principal efficiency argument for the framework.

3. Loss functions and objective-specific behavior

VG-SSL applies SSL objectives to embeddings produced by a feature extractor $d(g_i, g_j)$1, where $d(g_i, g_j)$2 is a local backbone, $d(g_i, g_j)$3 a global aggregation module, and $d(g_i, g_j)$4 a projection head (Xiao et al., 2023). For a query, positive, and sampled database negative, the framework computes

$d(g_i, g_j)$5

Depending on the objective, embeddings may be $d(g_i, g_j)$6-normalized, denoted with tildes.

The supervised reference is the Triplet Margin Loss,

$d(g_i, g_j)$7

which requires HNM to select hard $d(g_i, g_j)$8 (Xiao et al., 2023). It functions as the baseline against which the SSL variants are compared.

For SimCLR and MoCov2, VG-SSL uses the InfoNCE objective,

$d(g_i, g_j)$9

A symmetric form averages the loss after swapping the two views. In VG-SSL, positives are geo-related pairs, while negatives arise implicitly from other positives in the batch and explicitly through the presence of identical database negative pairs. MoCov2 additionally uses a momentum target encoder to produce stable keys (Xiao et al., 2023).

For BYOL and SimSiam, the framework uses an embedding prediction loss,

$r_{pos} = 10 \text{ m}$0

Again, a symmetric version can be formed by swapping inputs. BYOL uses a momentum target encoder, whereas SimSiam shares weights and relies on stop-grad and batch norm to avoid collapse (Xiao et al., 2023). The paper notes a specific failure mode in VG-SSL: identical negative pairs can induce collapse for self-distillation methods when $r_{pos} = 10 \text{ m}$1 is large, so $r_{pos} = 10 \text{ m}$2 must be tuned carefully.

For Barlow Twins, the framework computes the $r_{pos} = 10 \text{ m}$3 cross-correlation matrix

$r_{pos} = 10 \text{ m}$4

with loss

$r_{pos} = 10 \text{ m}$5

For VICReg, the framework uses

$r_{pos} = 10 \text{ m}$6

For both BT and VICReg, VG-SSL avoids $r_{pos} = 10 \text{ m}$7 normalization because the objectives operate on batchwise correlation or covariance (Xiao et al., 2023).

A central empirical distinction reported by the paper is that contrastive methods and information-maximization methods tolerate $r_{pos} = 10 \text{ m}$8 well, whereas BYOL and SimSiam are sensitive to large $r_{pos} = 10 \text{ m}$9 and may collapse (Xiao et al., 2023). This suggests that GeoPair does not interact uniformly with all SSL paradigms: its identical database negative pairs are benign for some objectives and destabilizing for others.

4. Architecture, implementation, datasets, and evaluation

The feature extractor in VG-SSL consists of a local feature extractor $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$0, a global aggregation module $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$1, and a projection head $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$2 (Xiao et al., 2023). The local feature extractor is an ImageNet-pretrained ResNet-50, with features taken from conv4_x; the global aggregation module is NetVLAD; and the projection head is a fully connected MLP with $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$3 layers, where hidden dimensionality equals output dimension $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$4 (Xiao et al., 2023).

The framework evaluates embedding dimensionalities $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$5. BT and VICReg benefit markedly from larger $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$6, whereas SimCLR, MoCov2, and BYOL are less sensitive and remain competitive at $$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$7–$$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$8 (Xiao et al., 2023). With respect to head depth, linear projection ($$P(q) = \{k \in \text{database} : d(g_q, g_k) \le r_{pos}\}.$$9) works best for SimCLR and MoCov2, while non-linear projection ($r_{neg} = 25 \text{ m}$0) is best for BYOL, SimSiam, BT, and VICReg. The paper further reports that with $r_{neg} = 25 \text{ m}$1, SimSiam collapses and BYOL, BT, and VICReg degrade, consistent with their original designs (Xiao et al., 2023).

Training and evaluation use images resized to $r_{neg} = 25 \text{ m}$2; Tokyo 24/7 queries are resized to the same resolution for consistency (Xiao et al., 2023). Default batch size is 64 for most experiments and ablations, while optimal settings use batch size 256 for SimCLR, BT, and VICReg, and 64 for MoCov2, BYOL, and SimSiam. The optimizer is Adam, with learning rate $r_{neg} = 25 \text{ m}$3 for SimCLR and MoCov2, $r_{neg} = 25 \text{ m}$4 for BYOL, SimSiam, BT, and VICReg, and weight decay $r_{neg} = 25 \text{ m}$5. Approximately 24 hours of training yields approximately 250 epochs, with 5000 queries per epoch, on one NVIDIA A100 80GB GPU. No HNM is used at any time (Xiao et al., 2023).

The benchmark uses five public VG datasets: Pitts30k, MSLS, Tokyo 24/7, Eynsham, and St. Lucia (Xiao et al., 2023). Pitts30k, MSLS, Eynsham, and St. Lucia are car-mounted datasets; Tokyo 24/7 is cross-domain, with a database from Google Street View and smartphone queries. MSLS is used for training, and its validation set is used as test because the test ground truth is private (Xiao et al., 2023).

Evaluation uses Recall@N, with the paper emphasizing Recall@1 as the main metric. Retrieval is performed with KNN in the learned embedding space, and a query counts as correct when at least one of the top-$r_{neg} = 25 \text{ m}$6 retrieved database images lies within an evaluation threshold $r_{neg} = 25 \text{ m}$7 of the ground-truth location:

$r_{neg} = 25 \text{ m}$8

At inference, the projected embeddings after $r_{neg} = 25 \text{ m}$9 are used directly without further fine-tuning (Xiao et al., 2023).

5. Quantitative results and ablation findings

The main quantitative results show that several SSL methods match or approach supervised triplet baselines while avoiding HNM (Xiao et al., 2023). At $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$0, BT-FC-2-4096-1 with batch 256 achieves Recall@1 values of 76.8 on Pitts30k, 77.9 on MSLS, 43.8 on Tokyo 24/7, 84.2 on Eynsham, and 91.3 on St. Lucia. Under the same dimensionality, VICReg-FC-2-4096-1 with batch 256 achieves 72.0, 74.2, 35.7, 81.7, and 86.7, respectively. The reproduced Triplet-PCA-4096 baseline attains 76.7 on Pitts30k, 77.1 on MSLS, 43.3 on Tokyo 24/7, 85.2 on Eynsham, and 94.7 on St. Lucia (Xiao et al., 2023).

At $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$1, SimCLR-FC-1-2048-1 with batch 256 achieves 75.2 on Pitts30k, 76.6 on MSLS, 44.3 on Tokyo 24/7, 85.6 on Eynsham, and 91.2 on St. Lucia. MoCov2-FC-1-2048-1 with batch 64 achieves 76.2, 74.2, 47.0, 84.8, and 89.1. BYOL-FC-2-2048-0.25 with batch 64 achieves 76.0, 75.7, 43.9, 84.3, and 90.5. SimSiam-FC-2-2048-0.25 is reported as lower across the board, including 69.2 on Pitts30k. The reproduced Triplet-PCA-2048 baseline reaches 75.8 on Pitts30k, 76.7 on MSLS, 40.7 on Tokyo 24/7, 84.8 on Eynsham, and 94.6 on St. Lucia (Xiao et al., 2023).

The benchmark also reports a strong HNM-based reference, Triplet (65535-dim, HNM; from benchmark), with Recall@1 values of 80.9 on Pitts30k, 76.9 on MSLS, 87.2 on Eynsham, and 93.8 on St. Lucia (Xiao et al., 2023). Against this backdrop, the paper’s main interpretation is that at $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$2, BT is comparable to Triplet-PCA-4096 and close to Triplet with HNM on most datasets, without HNM. At $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$3, SimCLR and MoCov2 are highly competitive and outperform Triplet-PCA-2048 on the cross-domain Tokyo 24/7 benchmark (Xiao et al., 2023).

Tokyo 24/7 receives special attention because it evaluates hand-held queries against a car-mounted database. On this benchmark, MoCov2-FC-1-2048-1 reaches $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$4 and SimCLR-FC-1-2048-1 reaches 44.3, both above the reproduced Triplet-PCA-2048 result of 40.7 under the unified resizing protocol (Xiao et al., 2023). The paper interprets this as evidence that SSL objectives emphasizing invariances and large-batch contrastive signal can produce descriptors with stronger cross-domain robustness.

The ablation studies isolate three main controls. First, for the Database Negative Ratio $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$5, SimCLR, MoCov2, BT, and VICReg perform best at $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$6 on most datasets, whereas BYOL performs best at $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$7 and SimSiam at $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$8 (Xiao et al., 2023). Second, projection head depth matters: $$N(q) = \{k \in \text{database} : d(g_q, g_k) \ge r_{neg}\}.$$9 is best for SimCLR and MoCov2, but $m_q$0 is best for BYOL, SimSiam, BT, and VICReg. Third, BT and VICReg benefit significantly from higher embedding dimension, especially 4096, whereas SimCLR, MoCov2, and BYOL are less sensitive and remain competitive at lower dimensions (Xiao et al., 2023).

6. Practical use, interpretation, and limitations

VG-SSL is released as public code at https://github.com/arplaboratory/VG_SSL, built on PyTorch Lightning and including training pipelines for all integrated SSL variants, the feature extractor with NetVLAD aggregation, and evaluation scripts for Recall@N (Xiao et al., 2023). The paper’s practical defaults are explicit: use an ImageNet-pretrained ResNet-50 and NetVLAD; use image size $m_q$1; set $m_q$2 and $m_q$3; use $m_q$4 for SimCLR and MoCov2 and $m_q$5 for BYOL, SimSiam, BT, and VICReg; and prefer $m_q$6 for SimCLR, MoCov2, BT, and VICReg, $m_q$7–$m_q$8 for BYOL, and $m_q$9 for SimSiam (Xiao et al., 2023).

For adaptation to new regions or datasets, the paper recommends keeping the $\{I_q\}$0 resizing and the same radii unless the dataset has substantially different GPS noise or density, increasing $\{I_q\}$1 for BT and VICReg, and considering larger batches for SimCLR, BT, and VICReg if resources permit (Xiao et al., 2023). If the domain gap is large, as in hand-held versus car-mounted imagery, MoCov2 and SimCLR with $\{I_q\}$2 and $\{I_q\}$3 are reported to have strong cross-domain results (Xiao et al., 2023).

The limitations are also method-specific. Resource constraints meant that the paper often used smaller batch sizes than are typical in SSL pretraining, and it notes that SimCLR, BT, and VICReg improve with larger batches (Xiao et al., 2023). BYOL and SimSiam can collapse when $\{I_q\}$4 is large, BT and VICReg require larger $\{I_q\}$5 and benefit from a two-layer projector, and the best SSL settings for VG differ from those often used in classification, notably the superiority of linear projection for SimCLR and MoCov2 in this retrieval setting (Xiao et al., 2023). The paper also did not exhaustively explore augmentation strategies or domain-specific pretext tasks, and it identifies extension to more SSL methods and analysis of augmentations and image size as future work.

A common misconception concerns the name itself: the framework is VG-SSL, not “VGG-SSL” (Xiao et al., 2023). More substantively, another possible misconception is that removing HNM necessarily weakens retrieval-oriented training. The reported results do not support that generalization. Instead, VG-SSL shows that geo-positive sampling combined with appropriately chosen SSL objectives can match or closely approach HNM-based supervised baselines on multiple datasets, while substantially reducing time and memory usage (Xiao et al., 2023).