CLIP2Video: Video Retrieval via Frame Aggregation

• The paper introduces a novel method that leverages query-adaptive weighted mean pooling to aggregate frame embeddings for improved video retrieval.
• It employs a symmetric InfoNCE contrastive loss to align video and text representations while requiring minimal parameter increases over the pretrained CLIP model.
• Empirical evaluations show consistent R@1 gains on benchmarks like MSR-VTT and ActivityNet, demonstrating both the efficiency and robustness of the temporal aggregation strategy.

CLIP2Video Model

The CLIP2Video model paradigm denotes a family of methods that adapt pretrained image–text contrastive models, specifically CLIP, to the video-text domain. These models extract framewise visual representations using CLIP’s image encoder, aggregate these representations over time, and match the resulting video-level embedding with a text embedding derived from CLIP’s text encoder. The central technical challenge addressed by CLIP2Video is efficient and effective temporal aggregation: how to summarize variable-length, information-sparse sequences of frame embeddings into a compact video descriptor suitable for retrieval or classification, particularly in settings involving long-form or multi-minute videos (Bain et al., 2022).

1. CLIP2Video Core Mechanism and Temporal Aggregation

At the foundation of CLIP2Video is the extraction and aggregation process. A video $V$ is first sampled into a sequence of $T$ RGB frames $\{I^{(1)}, ..., I^{(T)}\}$, each of which is processed by the frozen or fine-tuned CLIP vision encoder $f_v$, yielding per-frame embeddings $x_i = f_v(I^{(i)}) \in \mathbb{R}^d$. The associated text query is mapped into a CLIP text embedding $q = f_t(T) \in \mathbb{R}^d$.

The model then collapses $\{x_1,\dots,x_T\}$ into a single video vector $v$:

$v = \sum_{i=1}^T w_i x_i$

where $w_i \ge 0$ and $\sum_i w_i = 1$. Frame weights $\{w_i\}$ are typically determined via softmax normalization of query-conditioned or learned frame relevance scores:

$$w_i = \frac{\exp(\mathrm{Score}(q, x_i)/\tau)}{\sum_{j=1}^T \exp(\mathrm{Score}(q, x_j)/\tau)}$$

Here, $\tau$ is a temperature hyperparameter controlling sparsity. The score function admits several instantiations:

• Query-scoring (dot-product): $\mathrm{Score}(q, x_i) = q \cdot x_i$, inducing query-adaptive focus.
• Self-attention scoring: A single-layer Transformer contextualizes frame embeddings, followed by a linear head; scores are query-independent.
• Joint-attention scoring: A cross-modal Transformer processes frames concatenated with $q$, yielding query-dependent weights.

Empirically, simple query-scoring—without any learnable parameters—outperforms all prior temporal modeling attempts that utilize mean-pooling or stack temporal Transformers with millions of learnable parameters, particularly for long videos. The weighted mean allows the model to up-weight the few informative frames (e.g., salient visual events) and suppress "dead air" or redundant content (Bain et al., 2022).

2. Training Objectives and Contrastive Learning

CLIP2Video models are generally trained end-to-end using a standard symmetric InfoNCE (contrastive) loss over batched pairs of videos and captions:

$$\ell_{v2t} = -\frac{1}{N} \sum_{k=1}^N \log \frac{\exp(v_k \cdot q_k / \tau)}{\sum_{j=1}^N \exp(v_k \cdot q_j / \tau)}$$

with a corresponding $\ell_{t2v}$. The total objective is $\ell = (\ell_{v2t} + \ell_{t2v})/2$.

In practice:

• Finetuning is performed on video–text pairs, optimizing both vision and text encoders or just the aggregation module.
• In-batch negatives (other video–text pairs in minibatch) are sufficient; explicit hard negative mining is typically not necessary.
• The learning rate is substantially lower than standard CLIP (e.g., $5\times10^{-7}$), and only a few epochs are required for convergence (Bain et al., 2022).

3. Implementation Details and Computational Aspects

Key implementation specifics:

• Backbone: CLIP ViT-B/16 is standard for both image and text encoders; inherits massive pretraining from 400M image–text pairs.
• Training: Typically 16 frames/video (randomly or uniformly sampled), with dropout regularization on text tokens (~10%).
• Inference: Longer videos are subsampled at up to $T=120$ frames for aggregation. The weighted mean is computed per video.
• Retrieval Operations:
  • Mean pooling and self-attention variants yield retrieval cost $O(1)$ per video (weights can be pre-computed).
  • Query-scoring and joint-attention scoring are $O(T)$ per video per query (a negligible handful of dot-products since $T=120$).
• Memory: Storing $T=120$, $d=512$ embeddings per video yields $\approx$240KB per video.
• Temperature hyperparameter: $\tau=0.1$ is robust across long-video retrieval tasks (Bain et al., 2022).
• The computational overhead of online score computation is trivial relative to large-scale nearest-neighbor search typical in million-scale video collections.

4. Retrieval Benchmarks and Empirical Performance

The CLIP2Video weighted-mean approach has been rigorously evaluated on standard long-video text-to-video retrieval benchmarks:

Benchmark	Typical Video Length	R@1 Gain over Prior SOTA
MSR-VTT 1k-A	$\sim$15s	+2–5 points
ActivityNet Captions	$\sim$180s avg	Consistent improvements
Condensed Movies	$\sim$132s avg	Robust gains

Evaluation metrics include Recall at K (R@1, R@5, R@10), Median Rank, and Mean Rank. Notably, query-scoring weighted mean achieves R@1 gains of 2–5 points on MSR-VTT 1k-A relative to prior CLIP-based methods (CLIP4Clip, CAMoE, previous CLIP2Video, etc.), with no additional scoring parameters. Both self-attention and joint-attention variants perform well above naïve mean pooling and large temporal Transformer baselines, at a fraction of added parameter cost (Bain et al., 2022).

5. Broader CLIP2Video Variants and Related Approaches

The weighted mean aggregation paradigm defined by CLIP2Video underlies a spectrum of other retrieval and recognition methods:

• Zero-shot/mean-pooling baselines: Simple averaging of frame embeddings achieves competitive results in zero-shot settings, as established in early CLIP-based video retrieval protocols (Portillo-Quintero et al., 2021).
• Fine-tuned CLIP models: Fine-tuning all CLIP layers, or employing prompt tuning, further closes the domain gap and implicitly models temporal cues (scene dynamics, object motion) while keeping aggregation via mean (Rasheed et al., 2022).
• Video-to-text discretization and codebook fusion: Modern variants introduce frame-level quantization to text codebooks and confidence-aware fusion with attention modeling, but still adhere to the essential practice of framewise CLIP encoding and relevance-weighted pooling (Zhu et al., 24 Mar 2025).
• Temporal Transformer/external alignment: More involved models insert explicit temporal modules (e.g., temporal difference blocks, temporal alignment blocks) or multi-stage alignment/matching heads, in an effort to capture motion patterns or phrase-frame alignment; however, gains over CLIP2Video weighted mean remain modest unless very large computational resources or data are used (Fang et al., 2021, Gao et al., 2021).

6. Limitations, Insights, and Future Directions

CLIP2Video approaches possess several notable properties and limitations:

• Saliency-driven weighting: The ability to upweight semantically informative frames is critical for long videos, as mean-pooling over hundreds of redundant or irrelevant frames dilutes the signal (Bain et al., 2022).
• Simplicity vs. expressivity: Query-scoring weighted pooling is both simple and effective, outperforming deeper temporal modeling methods for retrieval, yet it is not optimized for recognition or generative tasks.
• Parameter efficiency: The best-performing CLIP2Video variants achieve SOTA retrieval with little to no increase in parameter count per video.
• Limitations: The method is chiefly retrieval-focused; its direct applicability to generative video captioning or precise temporal localization has not been validated (Bain et al., 2022).
• Potential extensions: Adaptive sampling, more sophisticated attention-based scoring, or hierarchical/video-scale pooling strategies may further improve performance on longer or more complex video domains. Expanding weighted mean pooling to incorporate cross-frame context or multimodal priors remains an active area of research.

7. Summary Table of Scoring Function Variants

Scoring Function	Query-Dependent	Requires Extra Params	Online/Offline Weights	Empirical Gain
Query-scoring	Yes	No	Online	Highest
Self-attention	No	Single-layer Transf.	Offline	High
Joint-attention	Yes	Single-layer Transf.	Online	High
Mean-pooling	No	None	Offline	Baseline

The query-scoring CLIP2Video model represents an efficient, robust, and easily deployable solution for long-form video retrieval, analytically grounded in weighted frame aggregation and validated against state-of-the-art alternatives on diverse datasets (Bain et al., 2022).