Co-VGGT: Co-visibility with Frozen VGGT Backbone

• The paper demonstrates that Co-VGGT effectively repurposes a frozen 24-layer, 1.2B-parameter VGGT with a layer-wise mixture-of-experts head to predict binary co-visibility.
• It achieves significant performance gains, reporting over 25% absolute improvement in IoU* and robust AUC metrics on Co-VisiON benchmarks for both pairwise and multiview setups.
• The calibrated co-visibility predictions facilitate improved feature matching and geometric reconstruction in SfM and SLAM, reducing runtime and enhancing loop closure reliability.

Co-VGGT is a co-visibility prediction model built on top of the Visual Geometry Grounded Transformer (VGGT). In its explicit formulation, it freezes a 24-layer, 1.2B-parameter VGGT backbone and trains only a lightweight layer-wise mixture-of-experts head to classify whether two RGB views share any visible physical surface, treating co-visibility as a first-class, calibrated signal for view-graph construction in SfM and SLAM. The model was introduced in “What VGGT Knows About Overlap: Probing Geometric Foundation Models for Co-Visibility” and is motivated by the observation that VGGT, although not trained with explicit co-visibility supervision, already contains a structured internal representation of overlap reasoning (Ziliotto et al., 10 Jul 2026).

1. Problem formulation and role in geometric vision

Co-visibility asks whether two or more images of a scene see any of the same physical surface. Formally, for views $I_i, I_j$ of the same scene, they are co-visible if there exists at least one 3D point on a surface that projects into both images and is not occluded in either. In SfM and SLAM, this determines which image pairs should be matched, which edges should be included in the view graph for bundle adjustment or pose-graph optimization, and which constraints should be trusted in loop closure and localization (Ziliotto et al., 10 Jul 2026).

The problem is especially acute in sparse-view regimes, strong occlusions, and large viewpoint changes. Traditional pipelines usually infer co-visibility only implicitly, through the success or failure of feature matching followed by geometric verification such as RANSAC. Retrieval methods such as NetVLAD can also be misleading because semantic similarity and geometric overlap are not equivalent; visually similar but spatially disjoint corridors can appear close in feature space. The hardest regime is minimal overlap, where two images share only a sliver of surface or a few jointly visible pixels. The Co-VisiON benchmark isolates this exact reasoning problem by asking for the binary co-visibility graph from sparse RGB views of indoor scenes (Ziliotto et al., 10 Jul 2026).

Within this formulation, Co-VGGT is a frozen-backbone system that predicts binary co-visibility from RGB alone, without depth, explicit geometry, or camera poses at test time. Its intended function is not generic image retrieval but calibrated visibility-graph estimation for downstream geometric pipelines (Ziliotto et al., 10 Jul 2026).

2. Emergent co-visibility inside VGGT

The defining conceptual claim behind Co-VGGT is that VGGT already encodes co-visibility as an emergent behavior. Probing results show a hierarchical structure analogous to LLMs: early layers build a 3D-aware scene representation, while late layers act as dedicated co-visibility reasoners. A zero-shot cosine-similarity probe on layer-wise view embeddings already achieves competitive IoU relative to several supervised baselines, indicating that substantial co-visibility signal is present in the frozen backbone before any task-specific training (Ziliotto et al., 10 Jul 2026).

The layer-wise gating analysis of the trainable probing head sharpens this picture. Layers 0–12 receive near-zero gating weight in both pairwise and multiview settings, suggesting that they capture generic appearance and low-level geometry but are not directly discriminative for overlap versus non-overlap. Late layers, approximately 14–24, dominate the decision process. For positive pairs, the mixture spreads across several late layers; in multiview mode, gate mass for positives concentrates around L15, whereas in pairwise mode positives draw from a broader range, roughly L18–L23. For negative pairs, there is a sharp and stable peak at layer L17 (Ziliotto et al., 10 Jul 2026).

L17 is identified as a negative anchor. For negative pairs, average gating weight $\alpha^{(17)}$ is approximately 0.37–0.41, far higher than for any other layer, and this behavior is stable across pairwise and multiview setups. Qualitative attention evidence is aligned with this interpretation: for non-co-visible pairs, L17 attends in a structured, object-centric manner in the first image but shows sparse and diffuse attention in the second image, consistent with a failure of cross-image correspondence; for positives, attention remains coherent in both images and the gating weights are more distributed across late layers (Ziliotto et al., 10 Jul 2026).

Layer ablations reinforce the same conclusion. Restricting the expert pool to late layers [14,24] barely hurts performance, with Gibson multiview IoU* remaining 0.74 versus 0.74 for all layers [1,24]. Restricting the model to early layers [1,12] degrades to IoU* 0.53 in multiview and 0.30 in pairwise mode. This is consistent with later mechanistic evidence that VGGT implicitly performs correspondence matching within its global attention layers and encodes epipolar geometry despite lacking explicit geometric constraints in training (Bratulić et al., 12 Dec 2025).

3. Architecture and inference regimes

Co-VGGT keeps the VGGT backbone completely frozen and adds a layer-wise mixture-of-experts head with approximately 7.5M trainable parameters. The backbone itself is the 24-layer Visual Geometry Grounded Transformer introduced by Wang et al. as a feed-forward multi-view geometry model that predicts camera parameters, depth maps, point maps, and tracking features from one to hundreds of views (Wang et al., 14 Mar 2025). In Co-VGGT, only its internal activations are reused; no backbone weights are updated (Ziliotto et al., 10 Jul 2026).

Given a batch of $S$ RGB views from the same scene, VGGT produces patch-token activations for selected layers. Co-VGGT first projects token channels with a shared linear projection and LayerNorm, then summarizes tokens via learned queries through cross-attention into a fixed number of summary tokens per view, which are flattened into compact per-view embeddings. For each labeled pair $(i,j)$, it constructs a symmetric pair representation from the two embeddings, their absolute difference, and their element-wise product. This follows the standard sentence-pair pattern of concat, absolute difference, and element-wise product, and is reported to be more effective than simple concatenation or using only differences or products (Ziliotto et al., 10 Jul 2026).

Each layer is then treated as an expert. A per-layer expert MLP outputs a co-visibility logit, while a gating MLP outputs a softmax-normalized layer weight. The final logit is the weighted sum of the layer logits, and the probability is obtained with a sigmoid. The head therefore plays two roles simultaneously: it is the task model used at inference time, and it is also a probe that exposes which layers dominate positive and negative decisions (Ziliotto et al., 10 Jul 2026).

The model operates in two regimes. In pairwise mode, each sample contains exactly two views. In multiview mode, each sample is an entire scene with $S>2$ views; VGGT sees all views jointly, and pair features are then built for all labeled pairs in that scene. The architecture is otherwise identical. Pairwise inference is thus a special case of multiview inference with $S=2$. In both regimes, the test-time input modality is RGB only (Ziliotto et al., 10 Jul 2026).

4. Supervision, benchmark, and empirical performance

Co-VGGT is trained as a binary classifier with binary cross-entropy on co-visibility labels. Supervision comes from the Co-VisiON benchmark, where an edge $(i,j)$ is positive if the two views share non-zero co-visible surface area, computed from ground-truth 3D data in Habitat for Gibson and HM3D. The supervision is therefore synthetic-from-geometry rather than human-generated, except for a separate human baseline on Gibson multiview (Ziliotto et al., 10 Jul 2026).

The reported training configuration is AdamW with learning rate $10^{-4}$, weight decay $10^{-4}$, batch size 32, and 50 epochs, although performance saturates early: around 30 epochs in multiview and around 10 epochs in pairwise mode. Model selection uses the best validation AUC (Ziliotto et al., 10 Jul 2026).

Co-VisiON provides sparse-view indoor scenarios rendered in Habitat. Gibson contains 85 train and 755 validation scenes with 33,849 labeled pairs in total under an 80/20 split. HM3D uses a 90/10 split with 210,008 labeled pairs. The main metrics are best-threshold graph IoU, denoted IoU*, and AUC defined as area under the IoU-threshold curve (Ziliotto et al., 10 Jul 2026).

The headline results are unusually large for a frozen-backbone add-on. In pairwise evaluation on Gibson, Covis scores 0.56 IoU* and 0.54 AUC, DUSt3R 0.54 and 0.54, GPT-4o 0.58 and 0.58, SpatialRGPT 0.49 and 0.49, Co-VGGT zero-shot 0.50 and 0.31, and Co-VGGT 0.85 IoU* and 0.78 AUC. On HM3D, Covis scores 0.53 and 0.51, GPT-4o 0.54 and 0.54, Co-VGGT zero-shot 0.46 and 0.31, and Co-VGGT 0.84 and 0.78. The paper characterizes this as a gain greater than 25% absolute in IoU* over prior state of the art (Ziliotto et al., 10 Jul 2026).

In multiview evaluation on Gibson, the human baseline is 0.72 IoU* and 0.72 AUC, Covis 0.59 and 0.57, Covis-freeze 0.61 and 0.57, MV-DUSt3R+ 0.56 and 0.56, GPT-4o 0.63 and 0.63, Co-VGGT zero-shot 0.37 and 0.30, and Co-VGGT 0.74 and 0.73. On HM3D, Covis scores 0.57 and 0.56, Covis-freeze 0.58 and 0.56, GPT-4o 0.59 and 0.59, Co-VGGT zero-shot 0.36 and 0.30, and Co-VGGT 0.76 and 0.74. On this benchmark, Co-VGGT therefore surpasses the human annotation baseline on Gibson multiview and improves over prior work by about 10% or more in multiview IoU* and AUC (Ziliotto et al., 10 Jul 2026).

Difficulty breakdowns show that the gain is concentrated in the regime the task was designed to expose. For hard pairs with overlap below 10%, GPT-4o reaches 0.34 Graph IoU, Covis-freeze 0.30, Co-VGGT pairwise 0.84, and Co-VGGT multiview 0.70. For hard scenes with average overlap below 4%, GPT-4o reaches 0.57, Covis-freeze 0.54, Co-VGGT pairwise 0.84, and Co-VGGT multiview 0.73. Interestingly, pairwise Co-VGGT exceeds multiview Co-VGGT, which the paper attributes to compression noise in multiview embeddings and the lack of explicit global graph consistency (Ziliotto et al., 10 Jul 2026).

5. Calibration and downstream geometric use

A distinctive feature of Co-VGGT is that its probabilities are calibrated well enough to be used directly as edge weights. On Gibson validation in pairwise mode, the model reports ECE = 0.030, Brier = 0.043, and MCE approximately 0.223. In multiview mode, ECE = 0.074, Brier = 0.085, and MCE approximately 0.347. The paper notes that the pairwise model is well calibrated and that more than 90% of predictions are near 0 or near 1, with the higher MCE concentrated in rarely populated mid-confidence bins (Ziliotto et al., 10 Jul 2026).

This calibration enables direct integration into SfM and SLAM. The intended use is to treat $p_{ij}$ as a visibility-graph edge weight, to set thresholds such as matching features only for pairs with $\alpha^{(17)}$0, or to discard pairs with $\alpha^{(17)}$1 before RANSAC. No temperature scaling or post-hoc calibration is required (Ziliotto et al., 10 Jul 2026).

A concrete downstream experiment uses COLMAP on 20 Gibson scenes. Co-VGGT scores are used to select the top 30% of image pairs, and COLMAP is run only on that subset. Relative to exhaustive pairwise matching, Co-VGGT recovers approximately 83% of images, approximately 70% of the sparse points of exhaustive matching, improves verification ratio from 0.910 to 0.966, improves reprojection error from 0.323 to 0.270, and reduces runtime from 97.0 s to 37.8 s, approximately 2.6 times faster for pair selection and matching. At the same 30% budget, it also outperforms random and NetVLAD-based selection in both coverage and geometry quality (Ziliotto et al., 10 Jul 2026).

Implementation measurements reported for an NVIDIA H100 are approximately 50 ms per pair and approximately 150 ms for 10 images in multiview embedding extraction. VGGT dominates inference time; the MoE head overhead is negligible, and retaining all 24 layers does not significantly increase runtime relative to using only late layers (Ziliotto et al., 10 Jul 2026).

6. Interpretation, limitations, and broader usage of the term

As a model class, Co-VGGT is narrow by design. It predicts binary co-visibility, not continuous overlap, although an auxiliary experiment adds a regression branch predicting overlap strength in $\alpha^{(17)}$2 with MSE alongside BCE. The paper does not pursue that direction in depth, arguing that overlap ratio is inherently ambiguous from sparse RGB and that non-co-visible pairs dominate with trivial zero targets. The stated future directions are continuous overlap estimation, unified multiview architectures such as set transformers or token-level graph reasoning, integration into embodied mapping systems for uncertainty-aware loop closure and next-best-view planning, and interpretability transfer from NLP to geometry-grounded foundation models (Ziliotto et al., 10 Jul 2026).

The main failure modes are minimal overlap, very small common regions, heavy occlusions, and medium-overlap cases without enough contextual views. Domain coverage is also limited: training and evaluation are on Habitat-rendered indoor scenes from Gibson and HM3D. Cross-domain transfer between Gibson and HM3D is reported as good, with IoU* loss no greater than 0.04 and performance still above in-domain state of the art, but generalization to real-world images, outdoor scenes, or dynamic environments is untested. The layer-wise interpretation is also correlational rather than causal: L17 is inferred to be a negative anchor from gating dominance and attention patterns, not from direct internal ablations (Ziliotto et al., 10 Jul 2026).

In later VGGT-based literature, the name “Co-VGGT” is used more loosely than in the original co-visibility paper. “VGGT-Segmentor” explicitly states that it does not introduce a method named Co-VGGT, but sketches a “Conceptual Co‑VGGT” aimed at co-visible regions and dense correspondences (Gao et al., 15 Apr 2026). “VGGT-MPR” similarly discusses “Co-VGGT, or any future VGGT-based model that seeks to coordinate modalities or views” as a design extrapolation for multimodal place recognition (Xu et al., 23 Feb 2026). “Diversity-aware View Partitioning for Scalable VGGT” describes a training-free scalable inference framework in terms of cooperative view organization, effectively extending the label toward chunked, diversity-aware VGGT inference (Park et al., 2 Jul 2026). This suggests that the term has developed a broader, informal meaning in the VGGT ecosystem, whereas its precise and explicit technical definition remains the frozen-backbone co-visibility classifier introduced in (Ziliotto et al., 10 Jul 2026).