OmniCLIP for Efficient Video Recognition

• The paper introduces a parameter-efficient framework that extends CLIP with Parallel Temporal Adapters and Self-Prompt Generators to capture omni-scale spatial-temporal dynamics.
• It integrates lightweight modules within each ViT block to explicitly model spatial and temporal features, enhancing accuracy across various video recognition tasks.
• Empirical results show substantial improvements on benchmarks like HMDB51 and UCF101, with less than 10% additional parameters compared to existing methods.

OmniCLIP is a parameter-efficient framework that adapts CLIP for video recognition by introducing omni-scale spatial-temporal feature learning. It enhances the frozen CLIP-ViT visual backbone with two lightweight modules—a Parallel Temporal Adapter (PTA) and a Self-Prompt Generator (SPG)—to explicitly model complex spatial-temporal dynamics required for accurate video understanding. OmniCLIP focuses on generating features that are both spatially and temporally comprehensive, enabling robust performance across supervised, few-shot, and zero-shot video recognition tasks (Liu et al., 2024).

1. Model Architecture

OmniCLIP extends the standard CLIP architecture by integrating the PTA and SPG into each transformer block of a frozen CLIP-ViT (ViT-B/16) encoder. The text encoder remains frozen, leveraging class name templates like “A video of [CLASS] action” for textual conditioning. For a video input $\mathbf{V}\in\mathbb{R}^{T\times H\times W\times3}$ of $T$ frames, each frame is patchified into $K$ patches ($K=HW/P^2$ for patch size $P$), projected linearly into a $d$-dimensional space, and prepended with a class token. Inputs are augmented with learnable spatial and temporal positional embeddings.

Within each ViT block $i=1,\ldots,N$:

1. SPG computes spatial self-prompts $R^{i-1}$ at a coarser spatial resolution.
2. $R^{i-1}$ is concatenated to the input token sequence as extra tokens.
3. The transformer block processes the sequence, producing updated representations:
   • $V_s^i = \mathrm{ViTBlock}_i(Z^{i-1})$ (spatial modeling)
   • $T$0 (temporal modeling via cross-frame attention)
   • $T$1 ($T$2 is a learnable scalar)

Feature aggregation is accomplished by fusing the final class tokens with multi-head attention (MHA) and averaging to form the video-level representation $T$3.

The PTA processes sequences by projecting token features to a bottleneck dimension, applying framewise multi-head self-attention, then projecting back, operating in parallel to the CLIP blocks. All weights for PTA and SPG are trained from scratch; CLIP weights remain unchanged.

2. Spatial-Temporal Omni-Scale Feature Learning

OmniCLIP explicitly constructs and fuses features across spatial, temporal, and dynamic spatial-temporal scales. The SPG outputs self-prompts $T$4, obtained by average-pooling token features spatially by a factor of 2 and projecting them through an MLP. These prompts enable attention to object regions at multiple resolutions. PTA leverages these prompts to compute attention along the temporal axis for each spatial region, capturing object motion and scale variation across frames.

Key feature representations include:

• $T$5: spatial feature at block $T$6 ($T$7)
• $T$8: temporal feature at block $T$9 ($K$0)
• $K$1: omni-scale dynamic feature ($K$2)
• Final video feature: $K$3, where pooling is implemented as MHA + averaging and $K$4 (uniform in practice)

This construction enables the model to sense and integrate information from fine-grained object instances and their motion trajectories over time.

3. Training Procedure and Optimization

OmniCLIP employs standard CLIP-style cross-entropy over cosine similarity for supervision: $K$5

$K$6

where $K$7 are class text embeddings and $K$8 is a learnable temperature.

All CLIP weights are frozen; optimization targets only PTA, SPG, the final projection, and $K$9, yielding a parameter-efficient training regime (<10% additional parameters).

Supervised training utilizes Kinetics-400, SSv2, HMDB51, and UCF101 datasets with established batch sizes, AdamW optimizer settings, cosine learning rate schedules, substantial augmentation (RandomFlip, ColorJitter, Gray, Mixup, Cutmix), and label smoothing.

For few-shot learning, a frozen linear classifier on $K=HW/P^2$0 is optionally added. Zero-shot evaluation is performed by pretraining on Kinetics-400 and deploying without fine-tuning.

4. Empirical Evaluation

OmniCLIP demonstrates strong results on standard video recognition benchmarks, often surpassing both supervised and parameter-efficient baselines:

Dataset	Method	Top-1 (%)	GFLOPs	Few-Shot (HMDB,16-shot)
Kinetics-400	OmniCLIP	84.1	130	—
SSv2	OmniCLIP (16f)	67.3	255	—
HMDB51	OmniCLIP	76.6	—	74.4
UCF101	OmniCLIP	96.3	—	95.1

OmniCLIP consistently outperforms recent parameter-efficient methods (e.g., MotionPrompt, X-CLIP) and achieves 74.4% top-1 accuracy on HMDB51 in the 16-shot setting compared to 64.6% for MotionPrompt. Zero-shot transfer is also improved, with 51.3% on HMDB51 (cf. 50.1% for MotionPrompt and 40.8% for vanilla CLIP). GFLOPs per view are lower than principal competitors.

Ablation studies confirm:

• The combined use of PTA and SPG yields optimal accuracy.
• Block-parallel placement of PTA is superior to cascade or attention-parallel schemes.
• Bottleneck ratios of 1/4 for PTA and average-pooling plus projection for SPG maximize performance.

5. Qualitative Analysis and Feature Visualization

Attention map analysis reveals that OmniCLIP's final-layer class token attends intensively to moving objects and relevant regions, such as instrument strings or small interactable items, in contrast to vanilla CLIP, which predominantly focuses on static, background, or large objects. Case studies highlight that for actions such as “pulling something from behind” or “playing harp,” SPG accurately focuses on small or low-resolution regions, while PTA maintains temporal consistency and tracking across frames. This synergy facilitates robust detection of both spatially small and rapidly moving targets.

6. Practical Implications and Parameter Efficiency

OmniCLIP introduces negligible computational overhead (≈130 GFLOPs/view) and does not require any update to CLIP backbone or text encoder, a salient feature for large-scale deployment. It operates as a parameter-efficient fine-tuning (PEFT) solution, with all modifications isolated to PTA, SPG, final projection, and $K=HW/P^2$1. The approach removes the need for additional distillation, momentum encoders, or teacher models, simplifying the training pipeline.

A plausible implication is that OmniCLIP’s design principles could generalize to other frozen backbone–based VLM architectures seeking to adapt to video or temporal tasks. The demonstrated performance gains for fast-moving or small objects (2–9% accuracy improvement) underscore the importance of integrating multi-scale spatial prompts and explicit temporal adapters in vision-LLMs for dynamic scenarios (Liu et al., 2024).