ConViS-Bench: Video Similarity & Co-Visibility

• ConViS-Bench is defined as a disambiguation case that distinguishes a benchmark for concept-based video similarity (using semantic concepts like main action, subjects, objects, location, and order) from a separate co-visibility reasoning benchmark for indoor scenes.
• In the video similarity domain, the benchmark employs fine-grained annotations on 610 video pairs from YouTube categories, using methods such as DINOv2 and Sentence-BERT with Spearman’s ρ and Kendall’s τ to evaluate multi-dimensional similarity.
• In the co-visibility reasoning domain, the benchmark uses graph-based evaluations on sparse indoor image sets, leveraging methods like Covis and GPT-4o to assess pairwise co-visible surface areas and revealing a notable machine–human performance gap.

Searching arXiv for the cited papers and related benchmark names to ground the article. ConViS-Bench is used in recent arXiv literature in more than one sense. The available usage suggests two distinct benchmark lineages: a benchmark for concept-based video similarity estimation associated with ConViS, which scores similarity with respect to semantic concepts such as action, subjects, objects, location, and action order (Liberatori et al., 23 Sep 2025), and a benchmark name used in the discussion of Co-VisiON, which evaluates co-visibility reasoning on sparse indoor image sets by asking whether image pairs share any non-zero co-visible surface area (Chen et al., 20 Jun 2025). By contrast, the earlier paper "ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal LLMs" introduces a hallucination-mitigation method rather than a benchmark, and explicitly states that there is no separate dataset called “ConViS-Bench” in that work (Park et al., 2024).

1. Nomenclature and scope

The current naming is best treated as a disambiguation problem rather than a single settled benchmark identity. In one usage, ConViS-Bench denotes the benchmark introduced with Concept-based Video Similarity estimation (ConViS). In another, ConViS-Bench appears in the benchmark discussion of Co-VisiON as a benchmark for co-visibility reasoning. The earlier ConVis paper on MLLM hallucination mitigation is related only by name and should not be conflated with either benchmark line (Liberatori et al., 23 Sep 2025).

Name usage	Technical focus	Source
ConViS-Bench	Concept-based video similarity estimation	(Liberatori et al., 23 Sep 2025)
ConViS-Bench / Co-VisiON benchmark discussion	Co-visibility reasoning on sparse indoor image sets	(Chen et al., 20 Jun 2025)
ConVis	Training-free contrastive decoding for MLLM hallucination mitigation; no separate ConViS-Bench dataset	(Park et al., 2024)

This distinction matters because the two benchmarks probe different capabilities. The video benchmark measures semantic, concept-conditioned similarity judgments across clips. The co-visibility benchmark measures scene-level spatial reasoning across sparse views. A plausible implication is that the shared name reflects convergent interest in structured multimodal comparison rather than a common dataset family.

2. ConViS-Bench as concept-based video similarity estimation

In the video literature, ConViS-Bench is defined around the claim that video similarity is not a single fixed quantity. Instead of requesting one opaque overall score, the task asks for similarity with respect to specific semantic concepts such as main action, main subjects, main objects, location, and order of actions (Liberatori et al., 23 Sep 2025).

The formalization introduces a predefined concept set

$\mathcal{C}=\{C_1,\dots,C_K\},$

where each concept is expressed in natural language. Given two videos $V_1$ and $V_2$, the concept-based similarity for concept $C_i$ is

$s(V_1,V_2 \mid C_i) \in \mathbb{R},$

and the paper also gives an aggregate formulation

$$s(V_1,V_2) = \sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i) \quad \text{with }\sum_{i=1}^K\lambda_i=1,$$

where $\lambda_i \ge 0$ are user-chosen weights (Liberatori et al., 23 Sep 2025).

This compositional view is central to the benchmark. It replaces a fixed global notion of similarity with a controllable one, allowing a system to weight whichever semantic dimensions matter for the application. The paper positions this as a bridge between traditional global embedding-based similarity systems and purely textual difference-description methods: the former provide a single score without explicit semantic decomposition, while the latter can explain differences in words but do not yield explicit per-concept quantitative scores (Liberatori et al., 23 Sep 2025).

The concept set used throughout the benchmark consists of five general-purpose semantic concepts: main action, main subjects, main objects, location, and order of actions. The paper argues that these concepts reflect how people naturally interpret events while remaining domain-agnostic across diverse video content (Liberatori et al., 23 Sep 2025). This suggests that the benchmark is intended less as a narrow task-specific dataset than as an evaluation scaffold for human-like, semantically factorized video comparison.

3. Dataset construction and annotation in the video benchmark

ConViS-Bench for video similarity is built from FineVideo, a large YouTube video collection. The authors select a subset of FineVideo categories with rich visual and temporal dynamics and avoid categories that are mostly static talking heads. They trim long source videos into event-level clips using timestamp annotations and form candidate pairs using either visual similarity or textual similarity, but not both at once. For pairing, they compute visual embeddings with DINOv2 and textual embeddings with Sentence-BERT, then select pairs with high cosine similarity in a single modality (Liberatori et al., 23 Sep 2025).

After manual filtering and verification, the final benchmark contains 610 video pairs drawn from 543 videos. The dataset spans 16 domains, and the appendix groups these into top-level themes such as Sports, Lifestyle, Education, and Hobbies & Interests, with examples including athlete workouts, recipe videos, gardening tips, documentaries, DIY crafts, knitting, hiking, camping, and fishing. The videos are also longer on average than in prior pairwise benchmarks: 28.2 seconds average duration for ConViS-Bench, versus 8.8 s for VidDiffBench and 12.0 s for StepDiff (Liberatori et al., 23 Sep 2025).

Annotation is performed by 150 annotators recruited through Prolific, all college-educated English speakers. For each pair, annotators rate similarity on a 5-point Likert scale for each of the five concepts and must provide at least one similarity tag and one difference tag in free-form text. Annotators may propose additional concepts if needed, but only 5.28% of annotations introduce a new concept. After curation, 7.75% of annotations are discarded, and the final dataset includes annotations from 147 annotators, with an average of 6.2 annotations per pair and, on average, 15 textual tags per video pair (Liberatori et al., 23 Sep 2025).

The benchmark also reports concept-wise agreement using Krippendorff’s $\alpha$: 0.244 for main subjects, 0.322 for main action, 0.361 for location, 0.288 for main objects, and 0.319 for order of actions. These values indicate that concept-conditioned video similarity is nontrivial even for humans, which is consistent with the paper’s broader argument that similarity judgments are multi-dimensional rather than reducible to a single scalar (Liberatori et al., 23 Sep 2025).

4. Evaluation protocol and empirical findings for video similarity

For concept-conditioned scoring, the paper evaluates nine open-source LMMs plus Gemini-2.0-Flash, including mPLUG-Owl3, LLaVA-OV (0.5B and 7B), LLaVA-Video-7B, Qwen2.5-VL (3B and 7B), InternVL2.5 (4B and 8B), InternVL3-8B, and Gemini-2.0-Flash. Models are prompted with concatenated frames from both videos and asked to output a single similarity score from 1 to 5 “in terms of <concept>.” Evaluation uses Spearman’s $\rho$ and Kendall’s $\tau$ correlation with human judgments, reported as percentages (Liberatori et al., 23 Sep 2025).

The strongest overall performer is LLaVA-OV-7B, which achieves the best or near-best correlations across concepts, with especially strong results on main objects and location. The reported values are about 58.64/48.17 on main objects and 58.94/47.72 on location in $V_1$0. LLaVA-Video-7B is also strong, and Gemini-2.0-Flash is competitive. The paper emphasizes, however, that alignment remains only moderate and is far from human consistency (Liberatori et al., 23 Sep 2025).

A recurring result is that order of actions is the most difficult concept for essentially all models, whereas main action, main objects, and location are somewhat easier, though still nontrivial. The frame-ablation experiments show that reducing temporal input hurts performance, especially for models not pretrained on FineVideo, and the authors note that the main action concept benefits the most from additional frames. This supports the claim that the benchmark is testing temporal reasoning rather than static visual matching (Liberatori et al., 23 Sep 2025).

The paper also analyzes methods that produce global similarity scores, grouping them into video-to-video, text-to-text, and cross-modal families. Examples include VideoMAE and DINOv2 for video-to-video, captioning plus Sentence-BERT for text-to-text, and CLIPScore or VQAScore for cross-modal alignment. The main finding is that global similarity methods are biased toward different concepts depending on their representation space: video encoders like DINOv2 are strongest on location, while text-based approaches tend to do better on action-related concepts. Across all global methods, location and order of actions tend to be weakest, and VQAScore has the best average global alignment overall in that section (Liberatori et al., 23 Sep 2025).

A second evaluation setting is concept-conditioned video-to-video retrieval, measured with R@1, P@1, and F$V_1$1@1. The retrieval sets use videos that appear in multiple annotated pairs, with candidate sets of four videos per anchor. The authors exclude order of actions from retrieval because annotators could not reliably establish consistent partial orders across sets. Retrieval is easiest for location and harder for main action. All evaluated LMMs beat random chance, but precision remains imperfect, indicating persistent false positives (Liberatori et al., 23 Sep 2025).

The human-analysis results reinforce the benchmark’s central premise. Model judgments only partially track human judgments, and the appendix notes that most visual concepts have low inter-correlation, except that main action and order of actions are strongly correlated, above 80. This is direct evidence for the paper’s claim that similarity is intrinsically multi-dimensional (Liberatori et al., 23 Sep 2025).

5. ConViS-Bench as co-visibility reasoning on sparse indoor image sets

In the co-visibility literature, the benchmark discussed as ConViS-Bench is the benchmark introduced with Co-VisiON: Co-Visibility ReasONing on Sparse Image Sets of Indoor Scenes. Here the task is not video similarity but co-visibility reasoning: given a sparse image set $V_1$2 from the same indoor scene, the model must infer which image pairs share any non-zero co-visible surface area and therefore should be connected in a binary graph (Chen et al., 20 Jun 2025).

The benchmark formalizes a co-visibility graph $V_1$3 through pairwise scores $V_1$4 and a threshold $V_1$5:

$V_1$6

The binary adjacency matrix $V_1$7 is the final graph representation. Evaluation is performed against a ground-truth adjacency matrix using Graph IoU and AUC. The main-paper definition is

$V_1$8

and the appendix gives the expanded version

$V_1$9

with $V_2$0 and $V_2$1 as elementwise AND/OR and $V_2$2 added for numerical stability. To avoid brittleness from a single threshold, the paper also computes AUC by sweeping thresholds $V_2$3 and integrating IoU over the threshold range (Chen et al., 20 Jun 2025).

The benchmark is built from photorealistic indoor simulation in Habitat-sim using Gibson and HM3D. The final benchmark contains over 1,000 indoor scenarios in total: 425 scenarios from Gibson and 755 scenarios from HM3D. The paper also reports 85 Gibson scenes and 755 HM3D scenes, with scene-level splits to avoid leakage. Gibson contributes 5,954 images and 33,849 image pairs; HM3D contributes 16,245 images and 210,008 image pairs. Scene composition spans 1-floor, 2-floor, 3-floor, and 4+ floor scenes (Chen et al., 20 Jun 2025).

The sampling policy is explicitly sparse but representative. Camera candidates are selected progressively to maximize scene coverage while avoiding overly dense overlap and overly disconnected graphs. The appendix gives the candidate score

$V_2$4

where $V_2$5 is newly explored area, $V_2$6 is already explored area, with $V_2$7 and $V_2$8. After choosing the best candidate, positions within radius $V_2$9 are pruned, and the process repeats until scene coverage exceeds 80%. The selection strategy prefers viewpoints near walls or furniture, mimicking human photography behavior while covering the environment broadly (Chen et al., 20 Jun 2025).

Annotation is generated automatically from simulation by leveraging camera poses and depth to compute exact co-visible regions and pairwise overlap. In addition, the authors collect a human-annotated subset of six Gibson scenes through a custom web GUI. Each scene is annotated at least twice, and disagreements are resolved by discussion until full agreement. The human-labeled subset serves both as a human-performance baseline and as a check on the automatic ground truth (Chen et al., 20 Jun 2025).

6. Baselines, Covis, and benchmark findings in co-visibility reasoning

The co-visibility benchmark evaluates a broad range of methods that all predict pairwise co-visibility probabilities before thresholding into a graph. Baselines include pairwise supervised classification with ViT, VGG, ResNet18, and CroCo v2 backbones; contrastive learning baselines such as supervised SimCLR-style training and NetVLAD; feature matching methods including SIFT+RANSAC and SuperGlue; 3D reconstruction methods such as DUSt3R and MV-DUSt3R+; and vision-LLMs including GPT-4o and Qwen2.5-VL 72B. Appendix results additionally report Gemini-2.0-Flash, SpatialRGPT, SigLIP, and a Covis-freeze variant (Chen et al., 20 Jun 2025).

The key new baseline is Covis, a supervised multi-view co-visibility predictor. The pairwise variant uses a CroCo v2 backbone and binary classification loss on image pairs. The full multi-view version uses the MV-DUSt3R encoder-decoder backbone to exchange information among multiple views simultaneously. Covis adds a learnable co-visible mask to suppress irrelevant features from non-co-visible regions. After decoding, pixel-wise features $C_i$0 are projected and a mask is predicted as

$C_i$1

then applied via

$C_i$2

The paper motivates this design by noting that multi-view attention alone can be contaminated by irrelevant views when the proportion of negative or non-overlapping regions increases (Chen et al., 20 Jun 2025).

Quantitatively, the benchmark exposes a large machine–human gap. Human annotation achieves 0.72 Graph IoU on Gibson. Among pure vision models, feature matching is weak: SuperGlue reaches 0.47 IoU / 0.11 AUC on Gibson and 0.38 / 0.10 on HM3D, while SIFT+RANSAC is 0.35 / 0.05 on both-ish ranges. Standard binary classification with ViT reaches 0.47 IoU / 0.43 AUC on Gibson and 0.48 / 0.46 on HM3D. DUSt3R reaches 0.54 / 0.54 on Gibson and 0.40 / 0.40 on HM3D, and MV-DUSt3R+ improves to 0.56 / 0.56 on Gibson and 0.45 / 0.45 on HM3D. Among pure vision models, Covis is best: the pairwise variant gives 0.48 IoU / 0.41 AUC on Gibson and 0.41 / 0.37 on HM3D, while the multi-view masked Covis reaches 0.59 IoU / 0.57 AUC on Gibson and 0.57 / 0.56 on HM3D (Chen et al., 20 Jun 2025).

On the VLM side, GPT-4o is the strongest overall tested model, with 0.63 IoU / 0.63 AUC on Gibson and 0.59 / 0.59 on HM3D, still below the human 0.72 upper bound. Qwen2.5-VL 72B performs much worse, at 0.41 / 0.41 on Gibson and 0.39 / 0.39 on HM3D. The headline conclusion is that all models lag far behind humans, proprietary VLMs outperform purely visual baselines, and Covis is the strongest open or pure vision method (Chen et al., 20 Jun 2025).

The ablations identify the central failure mode as feature contamination in multi-view reasoning. Without masking, multi-view Covis degrades sharply as the fraction of negative views or non-co-visible content increases. With supervised masks, Gibson/HM3D performance is around 0.59/0.57 and 0.57/0.56, whereas applying the ideal GT mask boosts to 0.74/0.72 and 0.71/0.69. The appendix also reports zero-shot transfer across Gibson and HM3D, where multi-view Covis consistently beats pairwise Covis in both in-domain and cross-dataset settings (Chen et al., 20 Jun 2025).

The paper further demonstrates two downstream uses. Co-visibility graphs can guide efficient 3D reconstruction by selecting only relevant frames for DUSt3R-like methods, and they can be used for automatic training-set labeling in tasks like CroCo-style cross-view completion. In DUSt3R, the co-visibility graph performs close to the complete graph and GT graph while using less memory and time; for cross-view completion, a sparse co-visibility graph achieves PSNR comparable to a high co-visibility graph while being easier to construct (Chen et al., 20 Jun 2025). This suggests that the benchmark is probing a capability that lies between local correspondence and full reconstruction: inferring scene topology from sparse views.

7. Relation to the earlier ConVis method and recurrent misconceptions

A common source of confusion is the similarity between ConViS-Bench, Co-VisiON, and ConVis. The earlier paper "ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal LLMs" introduces a training-free decoding method for reducing hallucinations in MLLMs by using a T2I model to reconstruct images from model-generated captions and then contrasting token distributions from the original and reconstructed images during decoding (Park et al., 2024).

That paper is benchmarked on CHAIR, HallusionBench, POPE, MME, and LLaVA-Bench, but it explicitly states that there is no separate dataset called “ConViS-Bench” in that work. Its contribution is a decoding strategy rather than a benchmark definition. The method is related only nominally to the later benchmarks, not through task formulation or evaluation protocol (Park et al., 2024).

The available literature therefore supports a three-way distinction. ConVis refers to hallucination mitigation in MLLMs. ConViS-Bench in the video paper refers to concept-based video similarity estimation. ConViS-Bench in the co-visibility discussion refers to the Co-VisiON benchmark for sparse indoor scene reasoning. A plausible implication is that any technical discussion of “ConViS-Bench” should specify the domain—video similarity or co-visibility reasoning—to avoid category errors in citation, implementation, or benchmark comparison.