SupScene: Overlap-Aware SfM Descriptor

• SupScene is a global descriptor framework for image retrieval in unconstrained SfM, aligning descriptor similarity with true 3D co-visibility rather than mere semantic similarity.
• It employs subgraph-based training with overlap-aware supervision and a semantic-modulated DiVLAD aggregator leveraging Vision Transformer attention for refined feature extraction.
• Experimental results on the GL3D dataset demonstrate state-of-the-art performance with improved retrieval recall and efficient computation.

SupScene is a global descriptor framework designed for image retrieval in unconstrained Structure-from-Motion (SfM), with a focus on aligning descriptor similarity with true 3D scene overlap rather than mere semantic similarity. It introduces an overlap-aware supervision regime via subgraph-based training and a semantic-modulated residual aggregation scheme, DiVLAD, leveraging Vision Transformer (ViT) attention mechanisms. SupScene establishes new state-of-the-art performance on SfM candidate retrieval tasks and offers generalizable improvements across aggregation architectures while maintaining computational efficiency (Shi et al., 17 Jan 2026).

1. Reframing SfM Retrieval: Overlap-Aware Descriptor Learning

Traditional global descriptors such as NetVLAD and GeM are trained to maximize semantic similarity, treating all images depicting similar objects or scenes as neighbors in feature space. However, in unconstrained SfM, geometric matchability—the spatial overlap of visible 3D structure—affords the relevant retrieval criterion. Two images might share class semantics yet possess no co-visible 3D regions, or may appear different but have extensive geometric overlap.

Existing approaches typically utilize binary pairwise (overlap vs. non-overlap) or triplet (anchor-positive-negative) training regimes, which collapse the rich spectrum of partial overlaps into coarse supervision. This loses granularity, treating barely and substantially overlapping pairs equivalently, and fails to leverage the many-to-many relationships present in real-world image graphs. SupScene recasts candidate retrieval as an explicit overlap prediction task, wherein cosine similarity between descriptors is calibrated against true overlap ratios derived from SfM reconstructions.

2. Subgraph-Based Training and Soft Contrastive Supervision

SupScene constructs an overlap graph $G = (V, E, w)$ per scene, where $V$ are images and each edge $(i,j)$ is weighted by the mesh-reprojection overlap ratio $w_{ij} \in [0,1]$ extracted from 3D point clouds or COLMAP “common-track” statistics. Training samples subgraphs $G_B$ of size $n$ using two complementary strategies:

• Anchor Expansion (BFS): Starting from an anchor, traversing high-overlap edges ($w_{ij} \ge \tau_{iou}$), and collecting neighbors.
• Balanced Sampling: Greedy batch construction to maintain a target positive pair ratio $\rho$.

For each subgraph, a ground-truth overlap matrix $O \in [0,1]^{n \times n}$ provides the precise overlap ratios for all pairs. After encoding subgraph images as unit-normalized descriptors $g_i$, SupScene computes the similarities $S_{ij} = g_i^\top g_j$. Supervision uses a soft weight matrix $W_{ij}$ modulated by overlap:

$$W_{ij} = \begin{cases} O_{ij}^{\gamma_s}, & O_{ij} \geq \tau_{iou} \ O_{ij}^{1/\gamma_s}, & 0 < O_{ij} < \tau_{iou} \ 0, & \text{otherwise} \end{cases}$$

with $\tau_{iou}\!=\!0.25$, $\gamma_s\!=\!0.7$. The Soft SupConLoss encourages pairs with higher overlap to align descriptors:

$$\mathcal{L} = \frac{1}{|\mathcal{V}_B|}\sum_{i \in \mathcal{V}_B} \left( -\frac{1}{Z_i}\sum_{j=1}^n W_{ij} \log \frac{\exp(S_{ij}/t)}{\sum_{k \ne i} \exp(S_{ik}/t)} \right)$$

where $t=0.1$. This soft formulation enables fine-grained, graded supervision, improving both convergence and generalization compared to hard binary contrastive loss.

3. DiVLAD: DINO-Inspired VLAD Aggregation

SupScene employs a DINOv2-B Vision Transformer as its backbone, with all but the final block frozen. Two outputs are extracted: the patch-token feature map $F \in \mathbb{R}^{C \times H \times W}$ and multi-head attention maps $A_{attn} \in \mathbb{R}^{N_h \times H \times W}$. Patches $x_n$ and attention scores $A_{h,n}$ are processed with cluster assignment $a_{k,n}$ from a lightweight $1 \times 1$ convolution for each of $K=64$ clusters.

The novel DiVLAD (DINO-inspired VLAD) aggregator synthesizes:

• Visual Assignment: Standard spatial cluster assignment from convolutional features.
• Semantic Modulation: Cluster-adaptive, per-token weighting of ViT multi-head attention via a learnable gating mechanism.

For cluster $k$, residual aggregation is:

$$v_k = \sum_{n=1}^N a_{k,n} \left[\sum_{h=1}^{N_h} g_{k,h} w_{h,n}\right](x_n - c_k)$$

where $c_k$ is the learnable cluster center, $w_{h,n}$ is a token-wise quality score, and $g_{k,h}$ are cluster-head gates. Concatenated over $K$ clusters and L2-normalized, this produces the final global descriptor $g \in \mathbb{R}^{KC}$.

4. Gating Mechanism for Semantic-Visual Fusion

The DiVLAD aggregator’s gating mechanism facilitates adaptive integration of visual and semantic cues:

• Token-Wise Quality: Per-head attention normalized over tokens and sharpened via sigmoid, $$w_{h,n} = [\sigma((A_{h,n} - \mu_h)/\sigma_h)]^{\gamma_g}$$, with $\gamma_g=0.5$.
• Head-Level Confidence: Aggregates $w_{h,n}$ across tokens and downweights heads with high entropy.
• Cluster-Head Gating: Learnable matrix $G \in \mathbb{R}^{K \times N_h}$ modulated by head confidence; softplus-activated and softmax-normalized over heads for each cluster.

This structured mechanism allows each VLAD cluster to leverage different attention heads as context dictates, enhancing discriminativeness for overlap prediction.

5. Implementation Details

The SupScene pipeline’s key configuration choices include:

• Backbone: DINOv2-B ViT with input size $322 \times 322$; all but the last transformer block frozen.
• DiVLAD Aggregator: $K=64$ clusters, $N_h=12$, $C=768$ channels.
• Training Data: GL3D dataset (110k images, 503 scenes), with $\tau_{iou}=0.25$.
• Optimization: AdamW, initial LR $2 \times 10^{-4}$, linear warmup for 10% of steps, cosine decay, 50 epochs.
• Batching: Subgraph size $n=32$, two subgraphs per GPU, eight GPUs (batch size 16), balanced sampling per batch.
• Augmentation: Resize to $322 \times 322$, random flip, color jitter.

Additional parameters from DiVLAD’s gating matrix ($\approx768$ weights) and the $1 \times 1$ convolutional head ($\sim$0.1M) introduce negligible overhead relative to NetVLAD ($\sim$0.5M). Inference speed is nearly identical to NetVLAD, with gating and aggregation adding less than 1 ms per image on an A6000 GPU.

6. Experimental Validation and Comparative Analysis

Experiments on GL3D demonstrate SupScene’s empirical advances:

Retrieval Recall@K

Method	Backbone	Dim	Recall@25	Recall@100
SiaMAC	CNN	512	59.4	88.6
NetVLAD	CNN	16,384	58.8	87.8
MIRorR	CNN	256	61.1	90.3
AVG-pool	DINOv2	768	71.5	96.6
DiVLAD	DINOv2	24,576	73.0	97.2

Aggregator Ablation (all under SupScene training)

Aggregator	Dim	@25	@50	@100
AVG	768	71.5	87.0	96.6
GeM	768	71.9	87.1	96.6
NetVLAD	24,576	72.5	87.7	97.0
SALAD	8,448	73.0	87.7	96.5
DiVLAD	24,576	73.0	88.2	97.2

Training Strategy Ablations

Full subgraph-based training converges faster and yields an approximately 1.5 percentage point higher Recall@25 than triplet-based supervised contrastive baselines, even with hard-negative mining. Subgraph sizes up to $n\sim32$ produce most gains; further increase yields diminishing returns. Replacing soft overlap-weighted supervision with a hard binary mask reduces Recall@25 by ~0.8 points, underscoring the benefit of graded overlap supervision.

7. Impact, Limitations, and Future Prospects

SupScene demonstrates that adopting overlap-aware, subgraph-level supervision and semantically modulated aggregation frameworks enables global descriptors that align with true 3D co-visibility, substantially improving candidate-pair recall and reconstruction completeness in SfM pipelines. A plausible implication is that overlap-aware weighting and semantic-visual fusion mechanisms may generalize effectively to related aggregation paradigms such as Fisher vectors or attention pooling. Further directions include integrating richer graph neural fusion for global descriptors, dynamic subgraph sizing based on scene complexity, and extending gating mechanisms to other aggregation models for enhanced generality and robustness (Shi et al., 17 Jan 2026).