---
title: 'ConViS-Bench: Video Similarity & Co-Visibility'
url: https://www.emergentmind.com/topics/convis-bench
type: topic
---

# ConViS-Bench: Video Similarity & Co-Visibility

Searching arXiv for the cited papers and related benchmark names to ground the article.
ConViS-Bench is used in recent arXiv literature in more than one sense. The available usage suggests two distinct benchmark lineages: a benchmark for **concept-based video similarity estimation** associated with ConViS, which scores similarity with respect to semantic concepts such as action, subjects, objects, location, and action order [2509.19245], and a benchmark name used in the discussion of **Co-VisiON**, which evaluates **co-visibility reasoning** on sparse indoor image sets by asking whether image pairs share any non-zero co-visible surface area [2506.16805]. By contrast, the earlier paper "ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models" introduces a hallucination-mitigation method rather than a benchmark, and explicitly states that there is no separate dataset called “ConViS-Bench” in that work [2408.13906].

## 1. Nomenclature and scope

The current naming is best treated as a disambiguation problem rather than a single settled benchmark identity. In one usage, **ConViS-Bench** denotes the benchmark introduced with **Concept-based Video Similarity estimation (ConViS)**. In another, **ConViS-Bench** appears in the benchmark discussion of **Co-VisiON** as a benchmark for co-visibility reasoning. The earlier **ConVis** paper on MLLM hallucination mitigation is related only by name and should not be conflated with either benchmark line [2509.19245].

| Name usage | Technical focus | Source |
|---|---|---|
| ConViS-Bench | Concept-based video similarity estimation | [2509.19245] |
| ConViS-Bench / Co-VisiON benchmark discussion | Co-visibility reasoning on sparse indoor image sets | [2506.16805] |
| ConVis | Training-free contrastive decoding for MLLM hallucination mitigation; no separate ConViS-Bench dataset | [2408.13906] |

This distinction matters because the two benchmarks probe different capabilities. The video benchmark measures **semantic, concept-conditioned similarity judgments** across clips. The co-visibility benchmark measures **scene-level spatial reasoning** across sparse views. A plausible implication is that the shared name reflects convergent interest in structured multimodal comparison rather than a common dataset family.

## 2. ConViS-Bench as concept-based video similarity estimation

In the video literature, ConViS-Bench is defined around the claim that video similarity is not a single fixed quantity. Instead of requesting one opaque overall score, the task asks for similarity **with respect to specific semantic concepts** such as **main action, main subjects, main objects, location, and order of actions** [2509.19245].

The formalization introduces a predefined concept set
$$
\mathcal{C}=\{C_1,\dots,C_K\},
$$
where each concept is expressed in natural language. Given two videos \(V_1\) and \(V_2\), the concept-based similarity for concept \(C_i\) is
$$
s(V_1,V_2 \mid C_i) \in \mathbb{R},
$$
and the paper also gives an aggregate formulation
$$
s(V_1,V_2) = \sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i) \quad \text{with }\sum_{i=1}^K\lambda_i=1,
$$
where \(\lambda_i \ge 0\) are user-chosen weights [2509.19245].

This compositional view is central to the benchmark. It replaces a fixed global notion of similarity with a controllable one, allowing a system to weight whichever semantic dimensions matter for the application. The paper positions this as a bridge between traditional global embedding-based similarity systems and purely textual difference-description methods: the former provide a single score without explicit semantic decomposition, while the latter can explain differences in words but do not yield explicit per-concept quantitative scores [2509.19245].

The concept set used throughout the benchmark consists of five general-purpose semantic concepts: **main action, main subjects, main objects, location, and order of actions**. The paper argues that these concepts reflect how people naturally interpret events while remaining domain-agnostic across diverse video content [2509.19245]. This suggests that the benchmark is intended less as a narrow task-specific dataset than as an evaluation scaffold for **human-like, semantically factorized video comparison**.

## 3. Dataset construction and annotation in the video benchmark

ConViS-Bench for video similarity is built from **FineVideo**, a large YouTube video collection. The authors select a subset of FineVideo categories with rich visual and temporal dynamics and avoid categories that are mostly static talking heads. They trim long source videos into event-level clips using timestamp annotations and form candidate pairs using either visual similarity or textual similarity, but not both at once. For pairing, they compute visual embeddings with **DINOv2** and textual embeddings with **Sentence-BERT**, then select pairs with high cosine similarity in a single modality [2509.19245].

After manual filtering and verification, the final benchmark contains **610 video pairs drawn from 543 videos**. The dataset spans **16 domains**, and the appendix groups these into top-level themes such as Sports, Lifestyle, Education, and Hobbies & Interests, with examples including athlete workouts, recipe videos, gardening tips, documentaries, DIY crafts, knitting, hiking, camping, and fishing. The videos are also longer on average than in prior pairwise benchmarks: **28.2 seconds average duration** for ConViS-Bench, versus **8.8 s** for VidDiffBench and **12.0 s** for StepDiff [2509.19245].

Annotation is performed by **150 annotators** recruited through **Prolific**, all college-educated English speakers. For each pair, annotators rate similarity on a **5-point Likert scale** for each of the five concepts and must provide at least **one similarity tag and one difference tag** in free-form text. Annotators may propose additional concepts if needed, but only **5.28%** of annotations introduce a new concept. After curation, **7.75%** of annotations are discarded, and the final dataset includes annotations from **147 annotators**, with an average of **6.2 annotations per pair** and, on average, **15 textual tags per video pair** [2509.19245].

The benchmark also reports concept-wise agreement using Krippendorff’s \(\alpha\): **0.244** for main subjects, **0.322** for main action, **0.361** for location, **0.288** for main objects, and **0.319** for order of actions. These values indicate that concept-conditioned video similarity is nontrivial even for humans, which is consistent with the paper’s broader argument that similarity judgments are multi-dimensional rather than reducible to a single scalar [2509.19245].

## 4. Evaluation protocol and empirical findings for video similarity

For concept-conditioned scoring, the paper evaluates **nine open-source LMMs plus Gemini-2.0-Flash**, including **mPLUG-Owl3**, **LLaVA-OV** (0.5B and 7B), **LLaVA-Video-7B**, **Qwen2.5-VL** (3B and 7B), **InternVL2.5** (4B and 8B), **InternVL3-8B**, and **Gemini-2.0-Flash**. Models are prompted with concatenated frames from both videos and asked to output a **single similarity score from 1 to 5** “in terms of <concept>.” Evaluation uses **Spearman’s \(\rho\)** and **Kendall’s \(\tau\)** correlation with human judgments, reported as percentages [2509.19245].

The strongest overall performer is **LLaVA-OV-7B**, which achieves the best or near-best correlations across concepts, with especially strong results on **main objects** and **location**. The reported values are about **58.64/48.17** on main objects and **58.94/47.72** on location in \(\rho/\tau\). **LLaVA-Video-7B** is also strong, and **Gemini-2.0-Flash** is competitive. The paper emphasizes, however, that alignment remains only moderate and is far from human consistency [2509.19245].

A recurring result is that **order of actions** is the most difficult concept for essentially all models, whereas **main action**, **main objects**, and **location** are somewhat easier, though still nontrivial. The frame-ablation experiments show that reducing temporal input hurts performance, especially for models not pretrained on FineVideo, and the authors note that the **main action** concept benefits the most from additional frames. This supports the claim that the benchmark is testing **temporal reasoning** rather than static visual matching [2509.19245].

The paper also analyzes methods that produce **global similarity scores**, grouping them into **video-to-video**, **text-to-text**, and **cross-modal** families. Examples include **VideoMAE** and **DINOv2** for video-to-video, captioning plus **Sentence-BERT** for text-to-text, and **CLIPScore** or **VQAScore** for cross-modal alignment. The main finding is that global similarity methods are biased toward different concepts depending on their representation space: video encoders like **DINOv2** are strongest on **location**, while text-based approaches tend to do better on action-related concepts. Across all global methods, **location** and **order of actions** tend to be weakest, and **VQAScore** has the best average global alignment overall in that section [2509.19245].

A second evaluation setting is **concept-conditioned video-to-video retrieval**, measured with **R@1**, **P@1**, and **F\(_1\)@1**. The retrieval sets use videos that appear in multiple annotated pairs, with candidate sets of four videos per anchor. The authors exclude **order of actions** from retrieval because annotators could not reliably establish consistent partial orders across sets. Retrieval is easiest for **location** and harder for **main action**. All evaluated LMMs beat random chance, but precision remains imperfect, indicating persistent false positives [2509.19245].

The human-analysis results reinforce the benchmark’s central premise. Model judgments only partially track human judgments, and the appendix notes that most visual concepts have low inter-correlation, except that **main action** and **order of actions** are strongly correlated, above 80. This is direct evidence for the paper’s claim that similarity is intrinsically multi-dimensional [2509.19245].

## 5. ConViS-Bench as co-visibility reasoning on sparse indoor image sets

In the co-visibility literature, the benchmark discussed as **ConViS-Bench** is the benchmark introduced with **Co-VisiON: Co-Visibility ReasONing on Sparse Image Sets of Indoor Scenes**. Here the task is not video similarity but **co-visibility reasoning**: given a sparse image set \(\mathcal{I}=\{I_1,\dots,I_n\}\) from the same indoor scene, the model must infer which image pairs share any **non-zero co-visible surface area** and therefore should be connected in a binary graph [2506.16805].

The benchmark formalizes a co-visibility graph \(\mathcal{G}\) through pairwise scores \(d_{ij}\) and a threshold \(\tau\):
$$
a_{ij}=1 \iff d_{ij}>\tau.
$$
The binary adjacency matrix \(\mathbf{A}\) is the final graph representation. Evaluation is performed against a ground-truth adjacency matrix using **Graph IoU** and **AUC**. The main-paper definition is
$$
\text{Graph IoU}=\frac{|\hat{\mathbf{A}\cap\mathbf{A}}|}{|\hat{\mathbf{A}\cup\mathbf{A}}|},
$$
and the appendix gives the expanded version
$$
\text{IOU}(\mathbf{A}, \hat{\mathbf{A}})=\frac{\sum_i\sum_j \mathbf{A}_{ij}\land \hat{\mathbf{A}}_{ij}}{\sum_i\sum_j (\mathbf{A}_{ij}\lor \hat{\mathbf{A}}_{ij})+\epsilon},
$$
with \(\land\) and \(\lor\) as elementwise AND/OR and \(\epsilon\) added for numerical stability. To avoid brittleness from a single threshold, the paper also computes **AUC** by sweeping thresholds \(t_1,\dots,t_n \in [0,1]\) and integrating IoU over the threshold range [2506.16805].

The benchmark is built from **photorealistic indoor simulation** in **Habitat-sim** using **Gibson** and **HM3D**. The final benchmark contains over **1,000 indoor scenarios** in total: **425 scenarios** from Gibson and **755 scenarios** from HM3D. The paper also reports **85 Gibson scenes** and **755 HM3D scenes**, with scene-level splits to avoid leakage. Gibson contributes **5,954 images** and **33,849 image pairs**; HM3D contributes **16,245 images** and **210,008 image pairs**. Scene composition spans **1-floor, 2-floor, 3-floor, and 4+ floor scenes** [2506.16805].

The sampling policy is explicitly sparse but representative. Camera candidates are selected progressively to maximize scene coverage while avoiding overly dense overlap and overly disconnected graphs. The appendix gives the candidate score
$$
S=\alpha O_u+\beta O_p,
$$
where \(O_u\) is newly explored area, \(O_p\) is already explored area, with \(\alpha=0.9\) and \(\beta=0.1\). After choosing the best candidate, positions within radius \(r\) are pruned, and the process repeats until scene coverage exceeds **80%**. The selection strategy prefers viewpoints near walls or furniture, mimicking human photography behavior while covering the environment broadly [2506.16805].

Annotation is generated automatically from simulation by leveraging camera poses and depth to compute exact co-visible regions and pairwise overlap. In addition, the authors collect a **human-annotated subset** of six Gibson scenes through a custom web GUI. Each scene is annotated at least twice, and disagreements are resolved by discussion until full agreement. The human-labeled subset serves both as a human-performance baseline and as a check on the automatic ground truth [2506.16805].

## 6. Baselines, Covis, and benchmark findings in co-visibility reasoning

The co-visibility benchmark evaluates a broad range of methods that all predict pairwise co-visibility probabilities before thresholding into a graph. Baselines include **pairwise supervised classification** with **ViT, VGG, ResNet18, and CroCo v2** backbones; **contrastive learning** baselines such as supervised SimCLR-style training and **NetVLAD**; **feature matching** methods including **SIFT+RANSAC** and **SuperGlue**; **3D reconstruction methods** such as **DUSt3R** and **MV-DUSt3R+**; and **vision-language models** including **GPT-4o** and **Qwen2.5-VL 72B**. Appendix results additionally report **Gemini-2.0-Flash**, **SpatialRGPT**, **SigLIP**, and a **Covis-freeze** variant [2506.16805].

The key new baseline is **Covis**, a supervised multi-view co-visibility predictor. The pairwise variant uses a **CroCo v2** backbone and binary classification loss on image pairs. The full multi-view version uses the **MV-DUSt3R** encoder-decoder backbone to exchange information among multiple views simultaneously. Covis adds a **learnable co-visible mask** to suppress irrelevant features from non-co-visible regions. After decoding, pixel-wise features \(F\in\mathbb{R}^{H\times W\times C}\) are projected and a mask is predicted as
$$
\mathbf{M}=\operatorname{conv}(\mathbf{F}),
$$
then applied via
$$
\mathbf{F}'=\mathbf{F}\odot \mathbf{M}.
$$
The paper motivates this design by noting that multi-view attention alone can be contaminated by irrelevant views when the proportion of negative or non-overlapping regions increases [2506.16805].

Quantitatively, the benchmark exposes a large machine–human gap. Human annotation achieves **0.72 Graph IoU** on Gibson. Among pure vision models, feature matching is weak: **SuperGlue** reaches **0.47 IoU / 0.11 AUC** on Gibson and **0.38 / 0.10** on HM3D, while **SIFT+RANSAC** is **0.35 / 0.05** on both-ish ranges. Standard binary classification with **ViT** reaches **0.47 IoU / 0.43 AUC** on Gibson and **0.48 / 0.46** on HM3D. **DUSt3R** reaches **0.54 / 0.54** on Gibson and **0.40 / 0.40** on HM3D, and **MV-DUSt3R+** improves to **0.56 / 0.56** on Gibson and **0.45 / 0.45** on HM3D. Among pure vision models, **Covis** is best: the pairwise variant gives **0.48 IoU / 0.41 AUC** on Gibson and **0.41 / 0.37** on HM3D, while the **multi-view masked Covis** reaches **0.59 IoU / 0.57 AUC** on Gibson and **0.57 / 0.56** on HM3D [2506.16805].

On the VLM side, **GPT-4o** is the strongest overall tested model, with **0.63 IoU / 0.63 AUC** on Gibson and **0.59 / 0.59** on HM3D, still below the human **0.72** upper bound. **Qwen2.5-VL 72B** performs much worse, at **0.41 / 0.41** on Gibson and **0.39 / 0.39** on HM3D. The headline conclusion is that **all models lag far behind humans**, proprietary VLMs outperform purely visual baselines, and **Covis** is the strongest open or pure vision method [2506.16805].

The ablations identify the central failure mode as **feature contamination** in multi-view reasoning. Without masking, multi-view Covis degrades sharply as the fraction of negative views or non-co-visible content increases. With supervised masks, Gibson/HM3D performance is around **0.59/0.57** and **0.57/0.56**, whereas applying the ideal GT mask boosts to **0.74/0.72** and **0.71/0.69**. The appendix also reports zero-shot transfer across Gibson and HM3D, where multi-view Covis consistently beats pairwise Covis in both in-domain and cross-dataset settings [2506.16805].

The paper further demonstrates two downstream uses. Co-visibility graphs can guide **efficient 3D reconstruction** by selecting only relevant frames for DUSt3R-like methods, and they can be used for **automatic training-set labeling** in tasks like CroCo-style cross-view completion. In DUSt3R, the co-visibility graph performs close to the complete graph and GT graph while using less memory and time; for cross-view completion, a sparse co-visibility graph achieves PSNR comparable to a high co-visibility graph while being easier to construct [2506.16805]. This suggests that the benchmark is probing a capability that lies between local correspondence and full reconstruction: **inferring scene topology from sparse views**.

## 7. Relation to the earlier ConVis method and recurrent misconceptions

A common source of confusion is the similarity between **ConViS-Bench**, **Co-VisiON**, and **ConVis**. The earlier paper "ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models" introduces a **training-free decoding method** for reducing hallucinations in MLLMs by using a T2I model to reconstruct images from model-generated captions and then contrasting token distributions from the original and reconstructed images during decoding [2408.13906].

That paper is benchmarked on **CHAIR, HallusionBench, POPE, MME, and LLaVA-Bench**, but it explicitly states that **there is no separate dataset called “ConViS-Bench”** in that work. Its contribution is a decoding strategy rather than a benchmark definition. The method is related only nominally to the later benchmarks, not through task formulation or evaluation protocol [2408.13906].

The available literature therefore supports a three-way distinction. **ConVis** refers to hallucination mitigation in MLLMs. **ConViS-Bench** in the video paper refers to **concept-based video similarity estimation**. **ConViS-Bench** in the co-visibility discussion refers to the **Co-VisiON benchmark** for sparse indoor scene reasoning. A plausible implication is that any technical discussion of “ConViS-Bench” should specify the domain—video similarity or co-visibility reasoning—to avoid category errors in citation, implementation, or benchmark comparison.

Source: https://www.emergentmind.com/topics/convis-bench