What VGGT Knows About Overlap: Probing Geometric Foundation Models for Co-Visibility

Abstract: A fundamental challenge in 3D reconstruction and robotic localization is co-visibility: determining which image pairs share overlapping visible surfaces, particularly in scenarios with minimal overlap. We demonstrate that VGGT implicitly encodes co-visibility as an emergent behavior: without any supervision for this task, its internal representations exhibit a clear hierarchical structure mirroring that of LLMs, i.e. early layers build a 3D-aware scene representation, while late layers act as dedicated co-visibility reasoners. In particular, we identify layer L17 as a negative anchor that consistently routes non-co-visible pairs for this backbone, regardless of the evaluation setting, providing task-grounded evidence of layer specialization in a geometry-grounded foundation model. Building on this, we introduce Co-VGGT, which freezes VGGT and trains only a lightweight layer-wise mixture-of-experts head (less than 7.5M parameters) to classify co-visibility from RGB alone, treating each layer as a specialized expert whose geometric abstraction is adaptively weighted per input pair. On the Co-VisiON benchmark, Co-VGGT surpasses the human annotation baseline and improves over prior work by more than 25% pairwise and 10% multiview. Pairwise predictions are well-calibrated (ECE=0.030), enabling direct use as edge weights in visibility graphs for downstream SfM and SLAM pipelines without post-hoc correction. Code and data are available.

• The paper demonstrates that the VGGT backbone encodes a strong, hierarchical co-visibility signal that benefits 3D reconstruction in sparse-view settings.
• The methodology extracts multi-layer embeddings and employs a lightweight mix-of-experts head, achieving superior performance over classical methods with well-calibrated probability outputs.
• The empirical results show state-of-the-art improvements in IoU and AUC across benchmarks, even surpassing human annotation in challenging scenarios.

Probing Geometric Foundation Models for Co-Visibility: Analysis of Co-VGGT

Introduction and Motivation

Determining co-visibility—whether two or more images of a scene share overlapping visible surfaces—is critical in 3D reconstruction, SLAM, and robotic localization, especially under sparse-view or minimal-overlap regimes. Classical pipelines infer co-visibility implicitly via feature matching and geometric verification, which degrade rapidly with decreased viewpoint overlap or scene texture. The "What VGGT Knows About Overlap" study (2607.09503) advances this domain by empirically demonstrating that geometry-grounded foundation models, specifically the Visual Geometry Grounded Transformer (VGGT), implicitly encode a strong co-visibility signal even in the absence of explicit supervision.

The authors analyze this phenomenon, show its hierarchical structure at the layer level, and introduce Co-VGGT—a lightweight, mix-of-experts (MoE) head atop a frozen VGGT backbone. Co-VGGT infers co-visibility from raw RGB, achieving superior performance over prior methods and even surpassing human annotation baselines in challenging sparse-view settings.

Figure 1: Illustration of the co-visibility prediction task and the Co-VGGT approach, including extraction of layer-wise embeddings from a frozen VGGT and aggregation via an MoE head to produce co-visibility probabilities for all image pairs.

Methodology

Architecture

The method leverages frozen VGGT as an invariant geometric feature extractor, avoiding expensive backbone adaptation. For each scene, layer-wise patch token features are extracted from selected layers ($L$) of VGGT. These are projected, summarized via learned queries in a cross-attention summarizer, and flattened into per-view, per-layer embeddings.

Pair features are constructed for each possible pair of images as a symmetric aggregation of embeddings:

• concatenation of embeddings,
• absolute difference,
• and elementwise product, resulting in a 4$D$-dimensional vector per layer per pair.

A trainable, lightweight MoE head ($\sim$7.5M parameters) treats each VGGT layer as an 'expert,' applying a per-layer MLP classifier and adaptively weighting logits via a learned gating network (softmax over layers). The final co-visibility probability for a pair is a mixture of the layer-expert outputs, reflecting the input-dependent contribution of each level of abstraction.

Figure 2: Block diagram of the Co-VGGT pipeline, detailing extraction, projection, summarization, and MoE-based co-visibility prediction.

Figure 3: Depiction of the MoE co-visibility prediction head and resulting scene-level visibility graph.

Training and Calibration

The MoE head is trained solely for binary co-visibility classification using a balanced binary cross-entropy loss. The backbone weights remain fixed, focusing model capacity and data requirements on the downstream head. The outputs are calibrated probabilities (ECE=0.030), enabling direct use as edge weights in visibility graphs for downstream pipelines, including SfM and SLAM.

Additionally, a zero-shot baseline is defined by removing the trainable head and using pooled frozen VGGT features with cosine similarity, establishing lower bounds for model probing.

Layer Specialization and Hierarchical Analysis

One of the core findings is that co-visibility reasoning emerges predominantly in late VGGT layers (notably L17), which serve as a negative anchor for non-co-visible pairs, while earlier layers encode generic geometric information but lack discriminative overlap cues. Gating analysis shows that, for negative pairs, the MoE distributes most weight to L17, while positive pairs leverage a broader combination of late layers (L15-L24), varying between pairwise and multiview regimes.

Figure 4: MoE gating weights per layer; strong specialization of late layers for decisive co-visibility prediction.

Cross-view attention visualizations corroborate the specialization: for non-overlapping pairs, attention maps in late layers become diffuse, indicating a 'failure' to correspond structure between images, while overlapping pairs yield focused mutual attention.

Figure 5: Exemplary cross-view token similarity matrix from layer 17, showing discriminative attention patterns for co-visible versus non-co-visible pairs.

Empirical Results

Benchmarks and Baselines

The Co-VGGT framework is evaluated on the Co-VisiON benchmark (covering HM3D and Gibson indoor environments) in both pairwise and full-scene (multiview) settings. Compared to classical feature matchers (SuperGlue, SIFT+RANSAC), learned encoders (DUSt3R, CroCo v2, NetVLAD), and prompted VLMs (GPT-4o, Gemini-2.0-Flash), Co-VGGT establishes a new state of the art.

Highlighted results:

• Pairwise (Gibson): IoU* 0.85, AUC 0.78
• Multiview (HM3D): IoU* 0.76, AUC 0.74
• Surpasses Covis baseline by >25% (pairwise) and ~10% (multiview)
• Outperforms human annotation on Gibson multiview
• Achieves strong generalization: zero-shot variants, and cross-domain transfer (train on Gibson, test on HM3D), with only minimal IoU/AUC loss

Calibration, Difficulty Analysis, and Downstream Utility

Co-VGGT not only excels in discriminative power but also exhibits strong calibration, with ECE 0.030 on pairwise evaluation (see Fig. 7). Extreme thresholds correspond to minimal false positives/negatives, making these scores directly usable for pair selection or pruning in scene graph construction.

Figure 6: Calibration analysis for predicted co-visibility probabilities.

Difficulty stratification demonstrates robust performance in low-overlap and globally sparse cases, where prior models and VLMs often collapse (co-visibility for <10% overlap: Co-VGGT IoU* 0.84 vs GPT-4o IoU* 0.34).

Furthermore, using Co-VGGT as a pair selector in SfM (COLMAP) achieves ~83% image registration and ~70% sparse point recovery at 2.6x reduced runtime, with improved geometric quality compared to comprehensive matching.

Limitations and Future Directions

The methodology retains certain limitations:

• Co-visibility estimation is formulated as binary; extension to continuous overlap or area prediction is left as future work.
• Multiview setting is currently computed as exhaustive pair loops, with no explicit global consistency or context aggregation.
• While layer-specialization is empirically demonstrated, mechanistic explanations for why specific depth cues emerge in certain layers remain unexplored.

Future research will entail: (i) continuous overlap estimation for richer geometric constraint modeling, (ii) globally consistent multiview aggregation (set transformers or relational modules), (iii) application in embodied and real-time perception systems for robust loop closure/active vision, and (iv) principled model interpretability drawing on analogies from LLMs.

Figure 7: IoU versus threshold plots highlighting robust performance across operating points on multiple datasets and settings.

Conclusion

This work empirically demonstrates that geometric foundation models such as VGGT encode emergent, hierarchically-structured co-visibility signals exploitable for robust 3D vision under sparse observations. The Co-VGGT mix-of-experts head efficiently extracts and calibrates this latent signal without any backbone adaptation, achieving SOTA across standard benchmarks. Analysis of layer specialization reveals architectural analogues to LLMs, with clear implications for interpretability and modularity in geometric perception models. These results suggest that large-scale geometry-pretraining imparts far richer spatial priors than conventional techniques, fundamentally improving reliability and calibration in downstream geometric vision tasks (2607.09503).