---
title: 'WebVid-CoVR: Composed Video Retrieval Dataset'
url: https://www.emergentmind.com/topics/webvid-covr-data-set
type: topic
---

# WebVid-CoVR: Composed Video Retrieval Dataset

WebVid-CoVR is a large-scale benchmark and data set specifically developed for the composed video retrieval (CoVR) task—a paradigm in computer vision where the system is required to retrieve a target video given a reference video and an associated natural-language modification string detailing compositional change requirements. The WebVid-CoVR data set, derived from the WebVid10M and WebVid2M collections, enables quantitative evaluation and training of retrieval models that can integrate visual semantics with human-like modification instructions. Extensions such as Dense-WebVid-CoVR further augment this resource with fine-grained, richly annotated modification texts, enabling research into models with enhanced contextual fidelity and compositional reasoning [2403.16997], [2508.14039], [2308.14746].

## 1. Formal Structure and Definition

Each sample in WebVid-CoVR is represented as a triplet $S = (v_{\mathrm{base}}, t_{\mathrm{mod}}, v_{\mathrm{target}}) \in \mathcal{V} \times \mathcal{T} \times \mathcal{V}$, where:
- $v_{\mathrm{base}}$: Reference/query video serving as the retrieval anchor,
- $t_{\mathrm{mod}}$: Free-form natural-language text describing the required transformation from the reference to the target video,
- $v_{\mathrm{target}}$: Target video expected to reflect the changes described in $t_{\mathrm{mod}}$.

This structure directly reflects the core CoVR task: mapping an input pair (video, modification text) to a corresponding output video exhibiting the compositional changes specified.

## 2. Dataset Construction, Annotation Protocols, and Splits

The collection of WebVid-CoVR leverages large-scale mining and LLM-driven text generation, further extended in Dense-WebVid-CoVR for fine-grained annotation:

- **Base Corpus**: WebVid2M [2308.14746] and WebVid10M supply millions of video-caption pairs.
- **Sampling**: Caption pairs differing by a single token are identified via CLIP embeddings and FAISS nearest-neighbor search; video pairs are selected based on CLIP-visual similarity of the middle frames.
- **Modification Text Generation**: LLMs (LLaMA-7B, GPT-3/3.5, MiniGPT-4, Gemini-Pro, GPT-4o) are prompted to produce human-like modification instructions.
- **Post-processing**: Outputs are filtered for grammaticality, semantic correctness, and contextual consistency. In manual evaluation and test sets, human annotators vet triplet validity and select best modification texts.
- **Scale and Splits**: Standard WebVid-CoVR contains ~1.65M triplets from 131K videos (average duration ≈ 16.8 s), with ∼467K unique modification texts (mean length 4.8 words) [2308.14746], [2403.16997]. Dense-WebVid-CoVR extends this with 1.6M triplets annotated for dense modification texts (mean length ≈31.2 words) and video descriptions averaging 81.3 words [2508.14039]. Official validation and test splits comprise 7K and ~3K manually verified triplets, respectively.

## 3. Linguistic Content and Modification Text Diversity

Modification texts within WebVid-CoVR are characterized by the following:
- **Format**: Un-templated, concise, human-like English (“Make the sky orange instead of blue.”, “Change the white tulip to yellow.”).
- **Vocabulary**: ~15K unique tokens in training, after BPE tokenization, lowercasing, and non-ASCII filtration.
- **Categories** (synthetic train set):
  - Color changes: ~35%
  - Object addition/removal: ~25%
  - Action/scene changes: ~20%
  - Others (e.g. lighting, orientation): ~20%
- **Dense Annotation**: Dense-WebVid-CoVR modification texts are ×7 longer than WebVid-CoVR, emphasize compositional, contextual, and temporally precise information, and are validated with a seven-step annotation protocol including visual checks and cosine similarity screenings [2508.14039].

The variety and linguistic richness of modification text enable learning of discriminative multimodal embeddings tuned to nuanced user queries.

## 4. Multimodal Integration and Retrieval Objectives

WebVid-CoVR supports joint modeling of visual and textual cues:
- **Modeling**: Query (video, text) pairs are encoded via BLIP-2 with cross-attention. Detailed video descriptions, generated via MiniGPT-4 or Gemini-Pro, are concatenated or fused with modification text for grounding.
- **Video Representation**: Each video is represented by 15 uniformly sampled frames; frame features are weighted and aggregated for robust retrieval [2403.16997], [2308.14746].
- **Retrieval Criterion**: Target video maximizes similarity to the joint (query, modification) embedding under the scoring function:
  $$
  v^* = \underset{v \in \mathcal{V}}{\arg\max} \;\mathrm{sim}(f(v_{\mathrm{base}}, t_{\mathrm{mod}}), g(v))
  $$
- **Training Objectives**: Hard-negative InfoNCE contrastive loss (HN-NCE), optionally symmetric across query/target, and triplet ranking loss formulations are employed.

## 5. Benchmark Metrics and Ablation Results

Performance is reported using recall@K (R@1, R@5, ...), the fraction of queries where the target appears within the top-K retrieved videos.

| Model/Setting                      | R@1   | R@5   | R@10  | R@50  |
|-------------------------------------|-------|-------|-------|-------|
| Zero-shot CLIP/BLIP (avg)           | ~44–45|  —    |  —    |   —   |
| CoVR-BLIP (HN-NCE, CA, N=15)        | 53.1  | 79.9  | 86.9  | 97.7  |
| Dense-WebVid-CoVR (Visual+Text+CA)  | 71.3  | 89.1  | 94.6  | 98.9  |

Models trained on WebVid-CoVR yield strong transfer to zero-shot composed image retrieval (CoIR), outperforming prior approaches on CIRR (R@1 from 35.4 to 38.5) and FashionIQ (R@10 from 24.7 to 27.7) [2308.14746], [2403.16997].

Ablation studies show that:
- Data scale increases recall (tripling seed videos from 500K→2.5M raises CIRR R@1 by 6 points).
- Transitioning from rule-based to LLM-generated modification texts improves retrieval performance by 8–11 points.
- Dense modification texts, as in Dense-WebVid-CoVR, further enhance fine-grained retrieval (+2.4 points R@1).

## 6. Applications, Limitations, and Future Prospects

WebVid-CoVR and its dense variant support research into vision-language retrieval, context enrichment, and compositional modeling, with dual applicability to video (CoVR) and image (CoIR) retrieval tasks [2403.16997]. Models trained on these data sets generalize from video to image retrieval benchmarks in zero-shot fashion, suggesting broad utility.

Key limitations include residual noise in automatically generated modification texts (2–3% for Dense-WebVid-CoVR), incomplete noise-free annotation at scale, and the absence of formal action class labeling. Prospective directions include support for multilingual modification text, scaling to longer clips via hierarchical indexing, and user-interactive feedback mechanisms [2508.14039].

A plausible implication is that unified fusion strategies (joint video+description followed by cross-attention with modification text) yield more coherent mappings and enhanced retrieval, as compared to pairwise or late fusion schemes. Rich, descriptive modification texts are critical for fine-grained compositional video search.

## 7. Context and Benchmark Role

WebVid-CoVR represents the largest natural-domain composed retrieval data set for video [2308.14746]. By enabling automatic triplet construction at scale and supporting rigorous manual evaluation protocols, it provides the foundational benchmark for recent advances in context-aware, discriminative video retrieval models. With the introduction of Dense-WebVid-CoVR establishing a new state of the art in Recall@1, the benchmark is poised to catalyze further development of models tackling complex spatio-temporal and semantic compositionality in large video databases [2508.14039].

*References*: [2403.16997], [2308.14746], [2508.14039]

Source: https://www.emergentmind.com/topics/webvid-covr-data-set