ConViS: Concept-Based Video Similarity

• The paper introduces a paradigm shift by conditioning video similarity assessments on semantic concepts instead of a single scalar score.
• It uses ConViS-Bench with five predefined, domain-agnostic concepts to capture key differences in video content such as main action, subjects, objects, location, and order of actions.
• Empirical findings reveal challenges in temporal reasoning and score calibration, highlighting the need for improved concept-aware and interpretable video representations.

Searching arXiv for the cited ConViS-related papers to ground the article in recent literature. arxiv_search(query="Concept-based Video Similarity estimation ConViS-Bench (Liberatori et al., 23 Sep 2025)", max_results=5) Concept-based Video Similarity Estimation (ConViS) is a video comparison paradigm in which similarity is estimated with respect to specified semantic concepts rather than collapsed into a single undifferentiated score. In the formulation introduced with ConViS-Bench, a predefined concept set $\mathcal{C}=\{C_1,\dots,C_K\}$ conditions pairwise comparison through concept-specific scores $s(V_1,V_2 \mid C_i)\in\mathbb{R}$, while an aggregate global similarity can be expressed as $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$ with non-negative weights summing to one. This design treats video similarity as inherently concept-dependent: two clips may be highly similar in main action yet dissimilar in location, subjects, objects, or temporal ordering. ConViS therefore belongs to a broader shift from instance-level retrieval toward structured, language-conditioned, and interpretable video comparison (Liberatori et al., 23 Sep 2025).

1. Formal definition and scope

ConViS defines similarity as a conditioned relation between two videos and a semantic concept. The conditioning variable is not an auxiliary explanation layer added after retrieval; it is the object of evaluation itself. In the benchmark formulation, concepts are expressed in natural language, and the comparison must attend only to information relevant to the queried concept while ignoring irrelevant aspects. This differs from conventional global video similarity systems, which typically compare pooled spatio-temporal embeddings and return a single scalar that does not identify which semantic dimensions drove the judgment (Liberatori et al., 23 Sep 2025).

A central consequence of this formulation is that similarity is multi-dimensional rather than monolithic. The same video pair can legitimately receive different scores depending on whether the comparison is conditioned on action, subjects, objects, location, or order of actions. This directly opposes the assumption, common in earlier retrieval protocols, that there is a single correct ranking induced by a single notion of relevance. The semantic-similarity retrieval literature had already argued against one-to-one relevance assumptions and proposed graded many-to-many relevance, but ConViS makes the conditioning variable explicit and operational at the concept level (Wray et al., 2021).

The benchmark uses discrete ordinal judgments in practice. Although the formal definition allows $s(V_1,V_2 \mid C_i)\in\mathbb{R}$, annotations are collected on a 5-point Likert scale ranging from 1, meaning completely different, to 5, meaning perfectly or extremely similar. ConViS is therefore both a representation problem and a judgment-calibration problem: a system must infer which evidence in the videos is relevant to the chosen concept and map that evidence to an ordinal human similarity scale (Liberatori et al., 23 Sep 2025).

2. Semantic dimensions and annotation space

ConViS-Bench instantiates ConViS using five predefined domain-agnostic concepts: main action, main subjects, main objects, location, and order of actions. The benchmark describes main action as the principal action or activity shown in the video; main subjects as the main actors or entities involved, including differences such as number of people, gender, and age; main objects as the main objects manipulated or present as central visual entities; location as the environment or place where the video occurs; and order of actions as the temporal ordering of sub-actions or steps within the event (Liberatori et al., 23 Sep 2025).

The rubric is concept-specific rather than generic. For main action, a score of 1 denotes completely different types of action, whereas a score of 5 denotes no difference in the main action. For main subjects, the rubric depends on cardinality, gender, and age. For location, the scale distinguishes completely different places from extremely similar locations. For order of actions, the scale ranges from completely different or no sub-actions in common to same order with no missing steps. For main objects, the rubric ranges from entirely different objects to same or extremely similar objects. These rubrics operationalize similarity as a structured semantic judgment rather than as visual resemblance alone (Liberatori et al., 23 Sep 2025).

The benchmark design also indicates that the five concepts are not merely redundant restatements of a single latent factor. Human concept inter-correlation is generally low, except for main action and order of actions, which have Spearman correlation above 80 according to human labels. This supports the claim that ConViS measures separable dimensions of similarity. At the same time, annotators were allowed to introduce additional concepts, and only 5.28% of annotations did so. The most frequent added concepts were point of view, skill level, and lighting condition, which indicates that the chosen concept set captures much of human comparison behavior while still leaving room for extension (Liberatori et al., 23 Sep 2025).

A frequent misconception is that concept-conditioned similarity is equivalent to multi-label tagging. The benchmark design argues against that simplification. The concept “order of actions,” in particular, is not reducible to the presence of action labels; it concerns temporal structure over sub-actions. Likewise, main subjects and location are not simply object or scene recognition problems, because the comparison is ordinal and pair-relative rather than absolute. ConViS therefore sits between recognition, retrieval, and comparative reasoning.

3. Benchmark construction and evaluation protocol

ConViS-Bench consists of 610 video pairs drawn from 543 videos across 16 domains, with average duration 28.2 seconds, and is released as a test-only benchmark. The source videos come from FineVideo, a large-scale YouTube-derived dataset of human activities. The benchmark creators exclude categories with low visual diversity and limited temporal dynamics, then automatically trim original videos using FineVideo timestamp annotations so that the units being compared correspond more closely to individual events than to long multi-event recordings (Liberatori et al., 23 Sep 2025).

Pair construction is intentionally asymmetric in modality. Candidate pairs are formed by high cosine similarity in a single modality rather than both: either video-to-video similarity using DINOv2 visual embeddings, or text-to-text similarity using Sentence-BERT embeddings of ground-truth descriptions. The candidates are then manually filtered and verified. This procedure is designed to produce pairs with both commonalities and meaningful differences, rather than near-duplicates that are simultaneously similar in every obvious dimension (Liberatori et al., 23 Sep 2025).

Human annotation is central to the benchmark. Annotators were recruited from Prolific; 150 participated initially and 147 were retained after curation. All were college-educated English speakers and were compensated at £8 per hour. For each pair, annotators rated all five concepts on the 1–5 scale and also provided at least one similarity tag and one difference tag in free form. The final pool averages 6.2 annotations per pair. During curation, 7.75% of annotations were discarded, and this process improved average inter-annotator agreement by 17.5% (Liberatori et al., 23 Sep 2025).

Agreement is measured with Krippendorff’s $\alpha$, with reported post-curation values of 0.322 for main action, 0.244 for main subjects, 0.288 for main objects, 0.361 for location, and 0.319 for order of actions. These values are modest rather than high, which is consistent with the subjectivity of pairwise semantic comparison and with the benchmark’s intention to model genuinely ambiguous human judgments rather than deterministic labels. The free-form tags complement the ordinal scores by exposing qualitative reasons for similarity and difference, and the appendix reports that there are on average 15 textual tags per pair (Liberatori et al., 23 Sep 2025).

Evaluation for direct concept-based similarity uses Spearman’s rank correlation $\rho$ and Kendall’s $\tau$ against human judgments, reported per concept and per model. Retrieval is defined separately. For concept-conditioned retrieval, an anchor video $V$, a concept $C_j$, and a candidate set $T_V=\{V_i\}_{i=1}^N$ are given; the model scores each candidate by $s(V_1,V_2 \mid C_i)\in\mathbb{R}$0 and induces a ranking. The retrieval set contains 532 concept-based partial orderings, each anchor has a candidate set of four videos, ties are allowed, and order of actions is excluded because human agreement becomes unstable for ranking larger candidate sets (Liberatori et al., 23 Sep 2025).

4. Empirical findings and model behavior

Among the concept-conditioned large multimodal models evaluated in zero-shot prompting, LLaVA-OV-7B is the best overall model on the main ConViS task. Its reported correlations are $s(V_1,V_2 \mid C_i)\in\mathbb{R}$1 for main action, $s(V_1,V_2 \mid C_i)\in\mathbb{R}$2 for main subjects, $s(V_1,V_2 \mid C_i)\in\mathbb{R}$3 for main objects, $s(V_1,V_2 \mid C_i)\in\mathbb{R}$4 for location, and $s(V_1,V_2 \mid C_i)\in\mathbb{R}$5 for order of actions. LLaVA-Video-7B slightly exceeds it on order of actions, reaching $s(V_1,V_2 \mid C_i)\in\mathbb{R}$6. Smaller models can fail badly: LLaVA-OV-0.5B has near-zero or negative correlations on several concepts. Even the best models remain only moderately aligned with human judgments, which indicates that concept-aware comparative video understanding is substantially unsolved (Liberatori et al., 23 Sep 2025).

The benchmark reveals a clear concept-difficulty hierarchy. Static or quasi-static concepts, especially location and main objects, are easier. Temporal relational structure is harder: order of actions is explicitly identified as the most challenging concept for all evaluated models. Frame-reduction ablations reinforce this interpretation. Models not pretrained on FineVideo, especially LLaVA-OV, LLaVA-Video, and Qwen2.5-VL, degrade as the number of frames per video is reduced; the reported average drops are 16.36% for LLaVA-OV, 10.61% for LLaVA-Video, and 21.74% for Qwen2.5-VL. Main action benefits more from additional temporal context than main subjects, which shows that the task is not simply image comparison under a video wrapper (Liberatori et al., 23 Sep 2025).

Global-similarity baselines expose systematic representational biases. DINOv2 video-to-video cosine similarity is especially strong on location, with reported $s(V_1,V_2 \mid C_i)\in\mathbb{R}$7 in $s(V_1,V_2 \mid C_i)\in\mathbb{R}$8, suggesting that visual embeddings emphasize scene and layout. Text-to-text pipelines based on caption generation followed by Sentence-BERT comparison perform better on action-related semantics than on location. Among non-concept-conditioned alternatives, VQAScore achieves the highest average correlation overall; for example, LLaVA-OV + VQAScore reaches $s(V_1,V_2 \mid C_i)\in\mathbb{R}$9 on main action, $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$0 on main subjects, $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$1 on main objects, $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$2 on location, and $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$3 on order of actions. These results show that global methods can accidentally align with some concepts, but they do so without explicit concept control (Liberatori et al., 23 Sep 2025).

In concept-conditioned retrieval, all tested large multimodal models outperform random. The strongest reported $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$4 scores are 61.9 for main action with LLaVA-OV-7B, 68.7 for main subjects with LLaVA-OV-7B, 65.1 for main objects with LLaVA-Video-7B, and 69.3 for location with LLaVA-OV-7B. The benchmark also notes a recurring pattern in which $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$5: models ხშირად retrieve a relevant top result but also generate many false positives. This suggests that concept-conditioned ranking is possible with current systems, but score calibration and concept selectivity remain weak (Liberatori et al., 23 Sep 2025).

5. Methodological lineages and adjacent research

ConViS did not emerge in isolation. One immediate lineage is semantic-similarity retrieval. “On Semantic Similarity in Video Retrieval” argues that instance-based retrieval wrongly assumes one relevant caption or video per query and proposes graded many-to-many relevance with $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$6, caption-derived semantic proxies, and nDCG evaluation. ConViS inherits that rejection of one-to-one relevance, but replaces generic semantic similarity with explicitly named comparison axes (Wray et al., 2021).

A second lineage is explicit concept representation. CoDEx discovers phrase-level concepts from natural language explanations and uses them as bottleneck variables for video classification, producing binary and probabilistic concept vectors that are human-readable and label-informative. TDCMN studies temporal concept receptive fields over clip-wise concept trajectories and introduces temporal dynamic convolution to adaptively mix kernel widths $$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$7 conditioned on the input video. These works do not define ConViS, but they supply two core ingredients: explicit concept spaces and temporally adaptive concept aggregation (Jeyakumar et al., 2022, Qi et al., 2021).

A third lineage is fine-grained similarity learning without explicit concepts. ViSiL computes video-to-video similarity from region-level frame relations and a CNN over frame-to-frame similarity matrices; S$$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$8VS learns a universal self-supervised video similarity function robust across copy, incident, and event retrieval; 3D-CSL uses clip-level 3D transformer representations with TopK Chamfer Similarity; and VSAL decomposes pairwise similarity into spatial similarity, temporal similarity, and partial alignments for copy localization. These methods demonstrate that video similarity often requires localized correspondences, temporal structure, and partial matching rather than a single pooled embedding, even when the representation is not explicitly semantic (Kordopatis-Zilos et al., 2019, Kordopatis-Zilos et al., 2023, Deng et al., 2022, Han et al., 2021).

A fourth lineage is open-vocabulary and relational representation learning. “Concept Visualization” introduces ConVis, an image-focused CLIP explanation method that grounds arbitrary WordNet concepts through synset definitions, rank-normalized similarity, descendant aggregation, and multiscale patch saliency. VideoSAUR learns object-centric slots from temporal feature similarity targets in self-supervised ViT feature space. Similarity Contrastive Estimation replaces the hard “all negatives are noise” assumption with a soft target distribution over semantically related instances. These works are not themselves ConViS systems, but they provide ingredients for open-vocabulary concepts, object-centric decomposition, and graded semantic neighborhoods. The acronym should therefore be distinguished carefully: ConViS denotes concept-based video similarity estimation, whereas ConVis names a WordNet-grounded CLIP explanation method for images (Giulivi et al., 2024, Zadaianchuk et al., 2023, Denize et al., 2022).

6. Limitations, ambiguities, and research directions

The current benchmark formulation has explicit constraints. ConViS-Bench uses a small set of five general-purpose concepts, contains 610 pairs, is primarily test-only, and evaluates prompted large multimodal models rather than a dedicated ConViS-specific architecture. The authors also acknowledge possible pretraining contamination for some evaluated systems, including known FineVideo exposure for InternVL and private-data uncertainty for Gemini. In retrieval, order of actions had to be removed because pairwise temporal-order judgments did not transfer cleanly to ranking over candidate sets (Liberatori et al., 23 Sep 2025).

These limitations interact with broader unresolved issues in adjacent literatures. CoDEx yields human-readable concept vectors but does not timestamp-align concepts within a video (Jeyakumar et al., 2022). TDCMN models temporal concept behavior but is trained for classification rather than metric learning (Qi et al., 2021). ConVis provides WordNet-grounded open-vocabulary concept probing, but the original method has no temporal modeling and is restricted to physical-object synsets (Giulivi et al., 2024). VideoSAUR captures object-centric temporal structure but not explicit action or event concepts (Zadaianchuk et al., 2023). ViSiL, S$$s(V_1,V_2)=\sum_{i=1}^K \lambda_i\,s(V_1,V_2 \mid C_i)$$9VS, and 3D-CSL learn strong similarity functions, yet their internal representations remain largely latent rather than concept-named (Kordopatis-Zilos et al., 2019, Kordopatis-Zilos et al., 2023, Deng et al., 2022).

This suggests that a mature ConViS system would likely need to combine several research threads rather than extend only one. A plausible implication is that future systems will require explicit concept vocabularies or discovered concept spaces, hierarchy-aware or soft semantic similarity rather than binary relevance, temporally resolved concept trajectories, and pairwise alignment mechanisms that can explain where and when two videos are similar. The benchmark’s added free-form concepts—especially point of view, skill level, and lighting condition—also suggest that concept coverage will need to expand beyond the current five-axis schema (Liberatori et al., 23 Sep 2025).

In that sense, ConViS is best understood not as a single model family but as a research program. It redefines video similarity as a conditioned, interpretable, and multi-dimensional judgment problem. Its benchmark supplies the first broad evaluation resource for this formulation, and the surrounding literature already provides many of the necessary components—graded relevance, concept bottlenecks, dynamic temporal concept modeling, fine-grained pairwise matching, open-vocabulary concept grounding, and soft relational self-supervision—but not yet their full integration into a unified concept-based video similarity engine (Liberatori et al., 23 Sep 2025).