---
title: 'CLIP2Video: Video Retrieval via Frame Aggregation'
url: https://www.emergentmind.com/topics/clip2video-model
type: topic
---

# CLIP2Video: Video Retrieval via Frame Aggregation

CLIP2Video Model

The CLIP2Video model paradigm denotes a family of methods that adapt pretrained image–text contrastive models, specifically CLIP, to the video-text domain. These models extract framewise visual representations using CLIP’s image encoder, aggregate these representations over time, and match the resulting video-level embedding with a text embedding derived from CLIP’s text encoder. The central technical challenge addressed by CLIP2Video is efficient and effective temporal aggregation: how to summarize variable-length, information-sparse sequences of frame embeddings into a compact video descriptor suitable for retrieval or classification, particularly in settings involving long-form or multi-minute videos [2205.08508].

## 1. CLIP2Video Core Mechanism and Temporal Aggregation

At the foundation of CLIP2Video is the extraction and aggregation process. A video $V$ is first sampled into a sequence of $T$ RGB frames $\{I^{(1)}, ..., I^{(T)}\}$, each of which is processed by the frozen or fine-tuned CLIP vision encoder $f_v$, yielding per-frame embeddings $x_i = f_v(I^{(i)}) \in \mathbb{R}^d$. The associated text query is mapped into a CLIP text embedding $q = f_t(T) \in \mathbb{R}^d$.

The model then collapses $\{x_1,\dots,x_T\}$ into a single video vector $v$:

\[
v = \sum_{i=1}^T w_i x_i
\]

where $w_i \ge 0$ and $\sum_i w_i = 1$. Frame weights $\{w_i\}$ are typically determined via softmax normalization of query-conditioned or learned frame relevance scores:

\[
w_i = \frac{\exp(\mathrm{Score}(q, x_i)/\tau)}{\sum_{j=1}^T \exp(\mathrm{Score}(q, x_j)/\tau)}
\]

Here, $\tau$ is a temperature hyperparameter controlling sparsity. The score function admits several instantiations:

- **Query-scoring** (dot-product): $\mathrm{Score}(q, x_i) = q \cdot x_i$, inducing query-adaptive focus.
- **Self-attention scoring:** A single-layer Transformer contextualizes frame embeddings, followed by a linear head; scores are query-independent.
- **Joint-attention scoring:** A cross-modal Transformer processes frames concatenated with $q$, yielding query-dependent weights.

Empirically, simple query-scoring—without any learnable parameters—outperforms all prior temporal modeling attempts that utilize mean-pooling or stack temporal Transformers with millions of learnable parameters, particularly for long videos. The weighted mean allows the model to up-weight the few informative frames (e.g., salient visual events) and suppress "dead air" or redundant content [2205.08508].

## 2. Training Objectives and Contrastive Learning

CLIP2Video models are generally trained end-to-end using a standard symmetric InfoNCE (contrastive) loss over batched pairs of videos and captions:

\[
\ell_{v2t} = -\frac{1}{N} \sum_{k=1}^N \log \frac{\exp(v_k \cdot q_k / \tau)}{\sum_{j=1}^N \exp(v_k \cdot q_j / \tau)}
\]

with a corresponding $\ell_{t2v}$. The total objective is $\ell = (\ell_{v2t} + \ell_{t2v})/2$.

In practice:

- Finetuning is performed on video–text pairs, optimizing both vision and text encoders or just the aggregation module.
- In-batch negatives (other video–text pairs in minibatch) are sufficient; explicit hard negative mining is typically not necessary.
- The learning rate is substantially lower than standard CLIP (e.g., $5\times10^{-7}$), and only a few epochs are required for convergence [2205.08508].

## 3. Implementation Details and Computational Aspects

Key implementation specifics:

- **Backbone:** CLIP ViT-B/16 is standard for both image and text encoders; inherits massive pretraining from 400M image–text pairs.
- **Training:** Typically 16 frames/video (randomly or uniformly sampled), with dropout regularization on text tokens (~10%).
- **Inference:** Longer videos are subsampled at up to $T=120$ frames for aggregation. The weighted mean is computed per video.
- **Retrieval Operations:** 
  - Mean pooling and self-attention variants yield retrieval cost $O(1)$ per video (weights can be pre-computed).
  - Query-scoring and joint-attention scoring are $O(T)$ per video per query (a negligible handful of dot-products since $T=120$).
- **Memory:** Storing $T=120$, $d=512$ embeddings per video yields $\approx$240KB per video.
- **Temperature hyperparameter:** $\tau=0.1$ is robust across long-video retrieval tasks [2205.08508].
- The computational overhead of online score computation is trivial relative to large-scale nearest-neighbor search typical in million-scale video collections.

## 4. Retrieval Benchmarks and Empirical Performance

The CLIP2Video weighted-mean approach has been rigorously evaluated on standard long-video text-to-video retrieval benchmarks:

| Benchmark            | Typical Video Length | R@1 Gain over Prior SOTA       |
|----------------------|---------------------|-------------------------------|
| MSR-VTT 1k-A         | $\sim$15s           | +2–5 points                   |
| ActivityNet Captions | $\sim$180s avg      | Consistent improvements       |
| Condensed Movies     | $\sim$132s avg      | Robust gains                  |

Evaluation metrics include Recall at K (R@1, R@5, R@10), Median Rank, and Mean Rank. Notably, query-scoring weighted mean achieves R@1 gains of 2–5 points on MSR-VTT 1k-A relative to prior CLIP-based methods (CLIP4Clip, CAMoE, previous CLIP2Video, etc.), with no additional scoring parameters. Both self-attention and joint-attention variants perform well above naïve mean pooling and large temporal Transformer baselines, at a fraction of added parameter cost [2205.08508].

## 5. Broader CLIP2Video Variants and Related Approaches

The weighted mean aggregation paradigm defined by CLIP2Video underlies a spectrum of other retrieval and recognition methods:

- **Zero-shot/mean-pooling baselines:** Simple averaging of frame embeddings achieves competitive results in zero-shot settings, as established in early CLIP-based video retrieval protocols [2102.12443].
- **Fine-tuned CLIP models:** Fine-tuning all CLIP layers, or employing prompt tuning, further closes the domain gap and implicitly models temporal cues (scene dynamics, object motion) while keeping aggregation via mean [2212.03640].
- **Video-to-text discretization and codebook fusion:** Modern variants introduce frame-level quantization to text codebooks and confidence-aware fusion with attention modeling, but still adhere to the essential practice of framewise CLIP encoding and relevance-weighted pooling [2503.18407].
- **Temporal Transformer/external alignment:** More involved models insert explicit temporal modules (e.g., temporal difference blocks, temporal alignment blocks) or multi-stage alignment/matching heads, in an effort to capture motion patterns or phrase-frame alignment; however, gains over CLIP2Video weighted mean remain modest unless very large computational resources or data are used [2106.11097][2111.05610].

## 6. Limitations, Insights, and Future Directions

CLIP2Video approaches possess several notable properties and limitations:

- **Saliency-driven weighting:** The ability to upweight semantically informative frames is critical for long videos, as mean-pooling over hundreds of redundant or irrelevant frames dilutes the signal [2205.08508].
- **Simplicity vs. expressivity:** Query-scoring weighted pooling is both simple and effective, outperforming deeper temporal modeling methods for retrieval, yet it is not optimized for recognition or generative tasks.
- **Parameter efficiency:** The best-performing CLIP2Video variants achieve SOTA retrieval with little to no increase in parameter count per video.
- **Limitations:** The method is chiefly retrieval-focused; its direct applicability to generative video captioning or precise temporal localization has not been validated [2205.08508].
- **Potential extensions:** Adaptive sampling, more sophisticated attention-based scoring, or hierarchical/video-scale pooling strategies may further improve performance on longer or more complex video domains. Expanding weighted mean pooling to incorporate cross-frame context or multimodal priors remains an active area of research.

## 7. Summary Table of Scoring Function Variants

| Scoring Function      | Query-Dependent | Requires Extra Params | Online/Offline Weights | Empirical Gain |
|----------------------|-----------------|----------------------|-----------------------|---------------|
| Query-scoring        | Yes             | No                   | Online                | Highest       |
| Self-attention       | No              | Single-layer Transf. | Offline               | High          |
| Joint-attention      | Yes             | Single-layer Transf. | Online                | High          |
| Mean-pooling         | No              | None                 | Offline               | Baseline      |

The query-scoring CLIP2Video model represents an efficient, robust, and easily deployable solution for long-form video retrieval, analytically grounded in weighted frame aggregation and validated against state-of-the-art alternatives on diverse datasets [2205.08508].

Source: https://www.emergentmind.com/topics/clip2video-model