Video CLIP: Temporal & Fusion Strategies

• Video CLIP is a model that extends the original CLIP framework to handle video data by aggregating per-frame embeddings to capture temporal dynamics.
• It employs methods like weighted mean query scoring, learned temporal modules, and prompt-based adaptations to fuse spatial and motion information effectively.
• Empirical results demonstrate that advanced Video CLIP variants achieve robust zero-shot video retrieval and action recognition while balancing computational efficiency.

Video CLIP refers to a family of models and methodologies that adapt the CLIP (Contrastive Language–Image Pre-training) paradigm—originally designed for image–text alignment—to video-language tasks such as video retrieval, action recognition, and video-text alignment. These adaptations encompass aggregation strategies for per-frame image embeddings, spatio-temporal modeling, query-adaptive fusion, and zero-shot transfer, all built atop the core CLIP architecture.

1. Foundational Principles: From CLIP to Video CLIP

CLIP is a dual-encoder framework trained on massive web-scale image–text pairs to embed visual and textual inputs into a joint space, facilitating high-transfer zero-shot recognition for images. Video CLIP models extend this principle to the video domain, where additional temporal complexity arises due to sequential frames. The central question is how to aggregate CLIP per-frame features to yield informative, temporally-aware video embeddings compatible with CLIP’s text encoder outputs.

This adaptation faces unique challenges:

• Redundancy and dilution: Naively averaging frame features (mean-pooling) can dilute semantically important frames, especially for long-form videos or ones with varying scene content (Bain et al., 2022).
• Temporal relevance: Modeling motion, capturing inter-frame dependencies, and ensuring alignment with text queries are non-trivial (Ahmad et al., 2023).
• Computational efficiency: Scalable large-scale retrieval requires text-agnostic video embeddings to enable offline indexing (Deng et al., 2023).

2. Aggregation and Temporal Modeling Techniques

Mean Pooling Baseline and Its Limitations

Mean-pooling computes a uniform average of frame-level CLIP embeddings: $$\bar{V}_{\mathrm{mean}} = \frac{1}{K} \sum_{k=1}^K f_k$$ where $f_k$ is the embedding for frame $k$ and $K$ is the total number of frames. This baseline enables strong zero-shot transfer, especially on short clips, as seen in "A Straightforward Framework For Video Retrieval Using CLIP" (Portillo-Quintero et al., 2021) and "CLIP4Clip" (Luo et al., 2021). However, for lengthy videos, mean pooling fails to emphasize query-relevant content and is vulnerable to redundancy, leading to suboptimal retrieval (Bain et al., 2022).

Weighted-Mean Query-Scoring

To address mean pooling’s shortcomings, query-dependent weighted aggregation is proposed (Bain et al., 2022): $$w_k = \frac{\exp(s_k/\tau)}{\sum_{j=1}^K \exp(s_j/\tau)}, \;\; s_k = f_k \cdot q$$

$\bar{V}_{\mathrm{weighted}} = \sum_{k=1}^K w_k f_k$

Here, $q$ is the text query embedding. The temperature parameter $\tau$ controls the sharpness—interpolating between uniform mean and max-pooling. This “query-scoring” mechanism outperforms all prior temporal models on long-form video retrieval tasks, with minimal parameter overhead and strong empirical performance on benchmarks such as MSR-VTT, CMD, ActivityNet, and Charades (Bain et al., 2022).

Learned Temporal Modules

Additional strategies involve adding sequential models (LSTM/Transformers) over per-frame CLIP embeddings, or hybrid fusion with query input (cross-attention, joint attention). While such learned temporal aggregators (e.g., CLIP4Clip seqTransf, CAMoE) can achieve marginal improvements, particularly with sufficient data, they often underperform on long or highly redundant videos due to overfitting or under-training, especially when compared to simple soft query scoring (Luo et al., 2021, Bain et al., 2022).

3. Advanced Architectures and Extensions

Parameter-Efficient Prompting and Discretization

EZ-CLIP employs lightweight learnable “temporal visual prompts” injected into each CLIP ViT layer, biasing the backbone to encode motion cues without modifying pretrained weights. An explicit motion-focused loss is used to prevent feature collapse across frames. This yields efficient, robust zero-shot recognition that outperforms previous video CLIP variants in both speed and generalization (Ahmad et al., 2023).

VTD-CLIP introduces a video-to-text discretization pipeline: encoded frames are quantized to their nearest class-aligned text prototypes (via CLIP’s text encoder), followed by confidence-aware fusion (weighted aggregation by semantic similarity), and prompt-tuned codebook adaptation. This approach injects interpretability and maintains CLIP’s zero-shot capabilities, demonstrating gains in both few-shot and base-to-novel splits on HMDB-51, UCF-101, SSv2, and K400 (Zhu et al., 24 Mar 2025).

Open-Vocabulary and Continual-Learning Based Video CLIP

Open-VCLIP and its successor, Open-VCLIP++, extend CLIP’s ViT with local temporal attention at each transformer layer—allowing tokens to attend to the same spatial token in temporal neighbors (t–1, t, t+1)—while text encoders remain unmodified. The key optimization is Interpolated Weight Optimization (IWO): fine-tuned weights are interpolated with original CLIP parameters, regularizing for zero-shot generalization. Additionally, Open-VCLIP++ aligns rich video-level text pseudo-captions with video embeddings to further close the vision-language gap, achieving state-of-the-art zero-shot recognition (Weng et al., 2023, Wu et al., 2023).

Large-Scale and Long Description Video CLIP Models

VideoCLIP-XL, built on ViCLIP, scales positional embeddings and pre-training to handle video–long description pairs. It introduces Text-similarity-guided Primary Component Matching (TPCM) to adapt the number of video feature components retained based on long–short text similarity, and auxiliary ranking losses (DDR/HDR) to focus on detail and hallucination resistance during pre-training. This leads to substantial improvements in both short- and long-description video retrieval and ranking (Wang et al., 2024).

4. Fusion Approaches and Hybrid Video CLIP

Multi-Modal Streams: Motion-Vector and Semantic Fusion

MoCLIP-Lite exemplifies a late-fusion multi-stream design: a frozen CLIP image encoder processes static appearance, a supervised EfficientNet processes compressed-domain motion vectors, and a lightweight MLP fuses both representations. The combined features achieve 89.2% top-1 on UCF101, indicating strong complementarity: CLIP encodes semantics, motion vectors encode dynamics (Huang et al., 21 Sep 2025). This paradigm leverages CLIP’s robustness and augments it with cost-efficient, highly dynamic motion representations.

Dual-Branch and Cross-Modal Attention

In long-term action anticipation, the Video+CLIP baseline fuses fixed CLIP video descriptors with a SlowFast video encoder via concat-fusion, feeding the joint descriptor into a transformer decoder (Das et al., 2022). Cross-attention over per-frame CLIP embeddings with learnable prompts outperforms naive mean/feature concatenation, indicating that prompt-conditioned cross-attention helps select temporally salient, action-relevant frames.

Prompt Switch (Deng et al., 2023) introduces a spatial-temporal "prompt cube" injected and peer-to-peer switched across transformer layers, learning to encode global video semantics in video-only embeddings, making large-scale retrieval tractable by avoiding text-dependent video representation at inference.

5. Quantitative Benchmarks and Empirical Insights

State-of-the-art Video CLIP models consistently set competitive baselines in both text-video retrieval and zero-/few-shot recognition regimes:

Method (Backbone)	MSR-VTT R@1	UCF-101 Top-1	HMDB-51 Top-1	Efficiency (GFLOPs)
CLIP4Clip (ViT-B/32, seqTransf)	44.5%	–	–	–
Query-Scoring (ViT-B/16) (Bain et al., 2022)	47.7%	–	–	–
Open-VCLIP++ (ViT-L/14)	39.0%*	88.1%	58.7%	–
EZ-CLIP (ViT-B/16)	–	79.1%	52.9%	102
OmniCLIP (ViT-B/16)	–	96.3%	76.6%	130
MoCLIP-Lite (ViT-B/32 + MV)	–	89.2%	–	16.9
Prompt Switch (ViT-B/32)	46.1%	–	–	–

*MSR-VTT retrieval. R@1 = Recall@1.

Experimental results demonstrate:

• Naive mean-pooling or query-scoring over CLIP frame features is remarkably strong and computationally efficient (Portillo-Quintero et al., 2021, Luo et al., 2021, Bain et al., 2022).
• Fusion or adapters that efficiently encode motion or dynamic objects (e.g. via motion vectors or parallel temporal adapters) outperform single-stream CLIP, particularly in action recognition (Huang et al., 21 Sep 2025, Liu et al., 2024).
• Parameter-efficient prompt-based adaptation and discretization strike a strong balance between interpretability and sample efficiency (Ahmad et al., 2023, Zhu et al., 24 Mar 2025).
• Computational cost remains a factor, with state-of-the-art models like OmniCLIP reducing GFLOPs relative to prior deep temporal transformers, and Prompt Switch drastically lowering online retrieval cost by avoiding cross-modal fusion at test time (Liu et al., 2024, Deng et al., 2023).

6. Analysis, Limitations, and Future Directions

The effectiveness of weighted per-frame aggregation and query-scoring demonstrates that the spatial CLIP backbone encodes semantically meaningful per-frame features, and simple fusion can often substitute for more complex, data-starved temporal architectures (Bain et al., 2022, Rasheed et al., 2022). However, several limitations persist:

• Query-dependent aggregation increases retrieval cost, albeit in a controlled manner (Bain et al., 2022).
• Most architectures, except those employing explicit motion features or temporal adapters, remain fundamentally frame-centric and may miss subtle or long-range motion cues.
• Training expressive temporal models continues to be bottlenecked by limited large-scale video–text datasets (Bain et al., 2022).

Emerging avenues include self-supervised pre-training on extensive video corpora, more adaptive or dynamic temporal modeling, and explicit integration of multimodal cues (audio, subtitles) for richer semantic grounding (Bain et al., 2022, Korolkov et al., 13 Apr 2025). Prompt-based, tokenized discretization (Zhu et al., 24 Mar 2025) and learned adapters (Ahmad et al., 2023) show promise for interpretable, efficient transfer learning. Hybrid fusion paradigms leveraging both visual and compressed motion domains are poised to further bridge the static–dynamic information divide, as exemplified by MoCLIP-Lite (Huang et al., 21 Sep 2025).

7. Broader Implications and Benchmarking Practices

Video CLIP and its numerous variants have recalibrated the baseline for video–language modeling:

• Off-the-shelf CLIP models, when adapted minimally with appropriate pooling or prompt-based adapters, form robust starting points for video retrieval and recognition (Rasheed et al., 2022, Bain et al., 2022).
• Query-scoring aggregation is now a critical baseline future methods must surpass to demonstrate meaningful gains in temporal aggregation (Bain et al., 2022).
• The field benefits from explicit efficiency and interpretability metrics, with leading works quantifying GFLOPs, parameter count, and offline/online retrieval costs.

Evaluating across diverse datasets—from short web videos (MSR-VTT, MSVD) and movie scenes (CMD, LSMDC) to unconstrained actions (Kinetics, UCF-101, HMDB-51)—remains essential for robust benchmarking and generalization assessment.

The “Video CLIP” paradigm unifies a broad range of approaches under the objective of leveraging powerful image–text pre-trained representations with minimal, well-principled adaptation, driving continued progress in video–language understanding (Bain et al., 2022, Luo et al., 2021, Zhu et al., 24 Mar 2025, Ahmad et al., 2023, Wang et al., 2024).