Co-VisiON Benchmark

• Co-VisiON is a benchmark that evaluates whether models can determine co-visibility by constructing binary graphs over sparse indoor image sets.
• It leverages photorealistic simulation environments like Gibson and HM3D with both automatic and human annotations to provide robust ground-truth labels.
• The proposed Covis method uses multi-view masking to enhance spatial reasoning, outperforming traditional vision methods while still trailing human performance.

Searching arXiv for the Co-VisiON benchmark paper and closely related context. Co-VisiON, short for Co-Visibility reasONing, is a benchmark for evaluating whether a model can infer which images in a sparse set of indoor views overlap in what they see and thereby construct a binary co-visibility graph over the image set (Chen et al., 20 Jun 2025). It targets a capability that the benchmark frames as central to indoor perception and robotics: determining whether two images share non-zero co-visible surface area even when the views are sparsely distributed, wide-baseline, and only weakly overlapping. In this formulation, co-visibility is treated not as ordinary image retrieval or place recognition, but as spatial reasoning across multiple views under sparse indoor-view conditions.

1. Conceptual scope and task definition

The benchmark is designed to measure three related abilities: determine co-visibility between image pairs in a sparse indoor scene, construct a co-visibility graph whose nodes are images and whose edges indicate that two images share visible physical surfaces, and do so under sparse, wide-baseline, indoor-view conditions, where local feature overlap may be small, noisy, or misleading (Chen et al., 20 Jun 2025). The motivating claim is that humans can often identify which images “go together” because they depict parts of the same physical space, even when the views are far apart or have very little apparent pixel overlap.

Formally, given a sparse image set

$\mathcal{I} = \{I_1, I_2, \ldots, I_n\},$

the goal is to build a co-visibility graph $\mathcal{G}$. For each pair $(I_i, I_j)$, a model outputs a co-visibility score $d_{ij}$. Thresholding with $\tau$ yields the binary adjacency matrix $\mathbf{A}$: $a_{ij} = 1 \iff d_{ij} > \tau.$ The benchmark therefore reduces co-visibility reasoning to binary graph prediction over sparse image sets.

A central misconception addressed by the benchmark is that co-visibility can be reduced to near-duplicate detection or simple local correspondence. The benchmark explicitly distinguishes it from ordinary retrieval: two images may be physically connected through the same room or object even if they are not near-identical, while two geographically close images may not be co-visible if they look at disjoint parts of a room. This suggests that the task probes scene topology rather than only pairwise visual similarity.

2. Construction of the benchmark

Co-VisiON is built from photorealistic indoor simulation environments, specifically Gibson and HM3D, selected because they provide textured 3D meshes, controllable camera viewpoints, accurate pixel-level visibility via rendering, and reliable ground-truth co-visibility annotations that are difficult to obtain from real-world depth or point-cloud data (Chen et al., 20 Jun 2025). The paper reports over 1,000 indoor scenarios overall, with 85 Gibson scenes and 755 HM3D scenes.

A scenario is defined as a structured collection of images with camera poses within a scene. Gibson uses 5 scenarios per scene on average, while HM3D uses 1 scenario per scene. The dataset includes 1-floor scenes, 2-floor scenes, 3-floor scenes, and 4+ floor scenes, thereby varying scene complexity.

The benchmark’s sparse setup is a deliberate design choice. Candidate camera placements are selected progressively so that scene coverage increases while the image set remains sparse and the graph is neither too dense nor disconnected. The selected views are intended to have pairwise IoU below about 30% to preserve sparsity and above about 5% to avoid disconnected graphs. The iterative process continues until the appendix’s target of more than 80% scene coverage is reached. Cameras are placed near walls or furniture to mimic realistic human photography behavior, and the camera is generally oriented away from walls to cover broader spatial context.

The dataset statistics reported for the two sources are as follows.

Dataset	Scenes	Scenarios / Images / Image Pairs
Gibson	85	425 scenarios, 5,954 images, 33,849 image pairs
HM3D	755	755 scenarios, 16,245 images, 210,008 image pairs

These design choices position the benchmark around sparse indoor reasoning rather than dense-view reconstruction. A plausible implication is that performance on Co-VisiON depends not only on geometric matching quality, but also on whether a model can integrate partial evidence across the entire image set.

3. Annotation and ground-truth generation

The benchmark contains two annotation regimes: automatic co-visibility annotation and human annotation (Chen et al., 20 Jun 2025). For automatic annotation, the authors use depth and 3D simulation to convert depth images into global point clouds and estimate overlap between views, yielding dense pairwise co-visibility labels suitable for large-scale benchmark construction.

Candidate-view selection is guided by the score

$S=\alpha \cdot O_u+\beta \cdot O_p,$

where $O_u$ denotes newly explored region covered by the candidate, $O_p$ denotes previously explored region, and $\mathcal{G}$0, $\mathcal{G}$1. This weighting favors views that expand scene coverage. Candidate positions near an already selected camera are pruned using

$\mathcal{G}$2

where $\mathcal{G}$3 is Euclidean distance between candidate location $\mathcal{G}$4 and selected position $\mathcal{G}$5.

The human annotation subset is manually labeled by experienced annotators and is used as a human baseline. The labeling rules include shared objects, object continuity, sub-scene relationship, and the rejection of plain featureless overlaps such as blank walls. Each scene is annotated at least twice, with cross-validation and discussion until agreement. The paper treats this subset as a human reasoning upper bound for the benchmark.

The coexistence of simulation-derived labels and human annotation is methodologically significant. The former enables scale and dense pairwise supervision, while the latter constrains evaluation to a notion of co-visibility that is not reducible to trivial overlaps. This suggests that the benchmark is intended to capture semantically meaningful overlap in physical space rather than arbitrary shared pixels.

4. Evaluation protocol and metrics

Predicted graphs are evaluated against ground truth using Graph IoU and AUC (Chen et al., 20 Jun 2025). Graph Intersection over Union measures overlap between the predicted graph $\mathcal{G}$6 and the ground-truth graph $\mathcal{G}$7: $\mathcal{G}$8 The appendix also provides an explicit binary-matrix form using element-wise AND and OR, with $\mathcal{G}$9 added to avoid division by zero.

Because IoU depends on the threshold used to binarize pairwise scores, the benchmark also reports AUC over thresholds sampled from $(I_i, I_j)$0. The stated rationale is that a good method should maintain consistently high IoU across thresholds rather than perform well only at a single hand-picked operating point.

The evaluation protocol reflects the benchmark’s graph-prediction framing. Methods are not judged only on isolated pairwise decisions; they are judged on the induced adjacency structure over the sparse image set. This matters because an algorithm can be locally plausible yet globally inconsistent. A plausible implication is that Co-VisiON penalizes methods that produce unstable threshold behavior or exploit dataset-specific score calibration without truly recovering co-visibility structure.

5. Baselines, evaluated models, and quantitative results

The benchmark evaluates a broad range of approaches, including pure vision baselines—SIFT + RANSAC, SuperGlue, ResNet18, VGG, ViT, a SimCLR-style contrastive model, and NetVLAD—along with 3D reconstruction methods (DUSt3R and MV-DUSt3R+), vision-LLMs (GPT-4o, Qwen2.5-VL 72B, with additional analysis including Gemini-2.0-Flash and SpatialRGPT), the proposed Covis baseline, and human performance via manually annotated co-visibility graphs (Chen et al., 20 Jun 2025).

The core empirical result is that all conventional pure-vision methods fall substantially behind humans, GPT-4o is the best-performing model overall, Covis is the best pure-vision method, and even Covis remains below the human upper bound. The main numbers reported for Gibson and HM3D are summarized below.

Method	Gibson	HM3D
SuperGlue	0.47 IoU*, 0.11 AUC	0.38 IoU*, 0.10 AUC
SIFT+RANSAC	0.35 IoU*, 0.05 AUC	0.34 IoU*, 0.05 AUC
ViT / NetVLAD	0.47 IoU*, 0.43 AUC (ViT)	0.42 IoU*, 0.39 AUC (NetVLAD)
DUSt3R	0.54 IoU*, 0.54 AUC	0.40 IoU*, 0.40 AUC
MV-DUSt3R+	0.56 IoU*, 0.56 AUC	0.45 IoU*, 0.45 AUC
Covis (pairwise)	0.56 IoU*, 0.54 AUC	0.53 IoU*, 0.51 AUC
Covis (multi-view)	0.59 IoU*, 0.57 AUC	0.57 IoU*, 0.56 AUC
GPT-4o	0.63 IoU*, 0.63 AUC	0.59 IoU*, 0.59 AUC
Qwen2.5-VL 72B	—	0.39 IoU*, 0.39 AUC
Human annotation	0.72 IoU	—

These results support the paper’s interpretation that co-visibility reasoning is hard for current models, especially those trained primarily for local correspondence or retrieval. Traditional methods often remain effective at local correspondence, but the reported gap indicates that sparse indoor co-visibility requires broader scene-level spatial reasoning and multi-image evidence integration.

6. Covis and the role of multi-view reasoning

Covis is the benchmark paper’s proposed baseline for co-visibility reasoning (Chen et al., 20 Jun 2025). It is described as a supervised binary classification and mask segmentation model that operates either in a pairwise or a multi-view setting. Its stated intuition is that multi-view input provides more context, but that context must be filtered so that irrelevant content does not corrupt the prediction.

Architecturally, Covis uses CroCo v2 or MV-DUSt3R-style encoder-decoder components and predicts both a topology/co-visibility score and a pixel-wise co-visible mask. The mask head is described by

$(I_i, I_j)$1

where $(I_i, I_j)$2 is the decoder feature map. The mask is then used to filter features: $(I_i, I_j)$3 where $(I_i, I_j)$4 denotes pixel-wise multiplication. The model uses binary cross-entropy (BCE) for both topology prediction and mask supervision.

The paper attributes Covis’s gains to masking away non-co-visible regions, irrelevant background, and distracting negative content, particularly in the multi-view regime. Empirically, multi-view Covis consistently outperforms pairwise Covis by about 3% IoU across datasets, and the learnable mask yields roughly 10% improvement. On Gibson, the mask ablation reports 0.52 IoU for No Mask, 0.59 IoU for Supervised Mask, and 0.74 IoU for Applied GT Mask. The appendix further reports that multi-view Covis is more robust than pairwise Covis in zero-shot cross-dataset transfer and gives an average predicted-mask IoU of 67.3% against binarized ground-truth masks.

The hardest cases remain those with very low overlap, sparse scenes, and weak visual cues. In these cases, GPT-4o and Covis gain the most relative advantage over other baselines, but all methods still experience sharp performance drops. This indicates that multi-view reasoning and masking help, but do not eliminate the underlying difficulty of inferring scene topology from sparse observations.

7. Significance, downstream uses, and limitations

The benchmark is presented as significant because it exposes a gap between capabilities that many current vision systems handle well—local matching, retrieval, and pairwise similarity—and what indoor scene understanding often requires: multi-view topological reasoning (Chen et al., 20 Jun 2025). The paper specifically identifies co-visibility as foundational for Structure-from-Motion (SfM), SLAM, visual localization, mapping, robotics navigation and pose estimation, and 3D reconstruction and scene graph construction. A reliable co-visibility graph can reduce the number of pairwise matches required in reconstruction and localization, improving both efficiency and robustness.

Two downstream uses are demonstrated. First, in 3D reconstruction, co-visibility graphs improve efficiency for DUSt3R-like pipelines by selecting only relevant frame pairs. The paper compares random, star, co-visibility, complete, and ground truth graph structures and reports that the co-visibility graph offers a strong tradeoff between reconstruction accuracy, memory cost, and computation. Second, co-visibility graphs can be used for automated training set labeling, generating useful training pairs for view completion, VPR training, and stereo learning. On CroCo completion, the sparse co-visibility graph performs comparably to a stricter high-overlap graph while being much easier to construct.

The principal limitation exposed by the benchmark is the remaining performance gap to humans. Even the best reported model, GPT-4o, is well below human performance, while purely vision-based methods lag further behind. The paper interprets this as evidence that current systems still lack robust scene-level spatial reasoning, the ability to integrate evidence across multiple images, and a human-like understanding of what it means for two views to belong to the same physical subspace. This suggests that future progress may require more explicit spatial structure modeling rather than reliance on basic pairwise matching or retrieval objectives alone.