MoCLIP-Lite: Efficient Two-Stream Action Recognition

• The paper demonstrates significant gains by fusing frozen CLIP semantics with motion vectors, achieving 89.2% Top-1 accuracy on UCF101.
• Efficiency is achieved by freezing both CLIP and motion backbones and training only a small MLP head, adding just ~0.05 GFLOPs over baselines.
• MoCLIP-Lite employs a two-stream late fusion design that leverages complementary modalities while maintaining throughput close to motion-only methods.

Searching arXiv for the specified paper and closely related naming variants to ground the article. MoCLIP-Lite is a lightweight two-stream video action recognition framework that combines semantic appearance knowledge from a frozen CLIP image encoder with temporal motion cues from a compressed-domain motion-vector network in order to perform efficient video recognition (Huang et al., 21 Sep 2025). The method was introduced in "MoCLIP-Lite: Efficient Video Recognition by Fusing CLIP with Motion Vectors" and is presented as a late-fusion system in which both backbones are frozen during fusion and only a small Multi-Layer Perceptron head is trained (Huang et al., 21 Sep 2025). In the same broader naming neighborhood, the literature also contains mobile-oriented MobileViCLIP and a lightweight CLIP-guided segmentation method sometimes informally associated with similar terminology; however, the name MoCLIP-Lite is explicitly attached in the provided record to the video recognition model of Zhang and colleagues rather than to those other architectures (Huang et al., 21 Sep 2025, Yang et al., 10 Aug 2025, Jin et al., 2023).

1. Problem setting and motivating trade-off

The model addresses a familiar trade-off in video recognition. Strong video models such as 3D CNNs and Transformers are accurate but expensive and often require large-scale video pretraining, whereas efficient compressed-domain methods using motion vectors are cheap but miss rich semantic context (Huang et al., 21 Sep 2025). CLIP, by contrast, offers powerful semantic understanding and zero-shot generalization on static images, but it is fundamentally an image model and does not directly capture motion (Huang et al., 21 Sep 2025).

MoCLIP-Lite is built on the claim that these modalities are complementary: CLIP helps recognize what is in the scene, while motion vectors help capture how things move (Huang et al., 21 Sep 2025). The framework is therefore designed to bridge the gap between static semantics and temporal dynamics by combining a frozen image encoder with a lightweight supervised motion stream and a minimal trainable fusion stage (Huang et al., 21 Sep 2025).

This suggests that the method is best understood not as a fully end-to-end multimodal video foundation model, but as an efficiency-oriented composition of strong frozen semantic priors and low-cost temporal evidence. A plausible implication is that its design philosophy is closer to modular systems engineering than to large-scale video pretraining.

2. Architectural design

MoCLIP-Lite follows a two-stream late fusion design (Huang et al., 21 Sep 2025). The appearance stream uses a CLIP image encoder to process one representative RGB frame per video, while the motion stream uses a TSN-style motion-vector encoder on compressed-video motion vectors (Huang et al., 21 Sep 2025). The two resulting feature vectors are concatenated and passed to a small MLP classifier (Huang et al., 21 Sep 2025). The fused representation is written as

$\mathbf{f}_{\text{fusion} = [\mathbf{f}_{\text{app};\mathbf{f}_{\text{motion}]$

and the classifier output as

$\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$

Late fusion is chosen because each branch can remain specialized and use its native input format without forcing early alignment between RGB and motion vectors (Huang et al., 21 Sep 2025). This is a notable design decision: unlike architectures that impose cross-modal attention or shared tokenization, MoCLIP-Lite keeps the branches separate until the classifier stage.

The appearance pathway uses the official CLIP ViT-B/32 image encoder (Huang et al., 21 Sep 2025). For each video $V=\{I_1,\dots,I_T\}$, a single representative RGB frame $I_{rep}$ is sampled, and the CLIP image feature is precomputed offline as

$\mathbf{f}_{\text{app} = \mathcal{E}_I(\text{Preprocess}(I_{rep})) ,$

where $\mathcal{E}_I$ is the frozen CLIP image encoder and $d_{\text{clip}=512$ (Huang et al., 21 Sep 2025). The paper also states that a class text library is built using prompt ensembling with the frozen CLIP text encoder, averaging multiple templates per class for the zero-shot baseline (Huang et al., 21 Sep 2025).

The motion pathway extracts motion directly from the compressed video stream using motion vectors (Huang et al., 21 Sep 2025). The input is a 2-channel tensor

$M_t \in \mathbb{R}^{H \times W \times 2}$

with channels $(MV_x, MV_y)$, normalized to $[-0.5, 0.5]$ using a preprocessing pipeline consistent with prior MV recognition work (Huang et al., 21 Sep 2025). A Temporal Segment Network sampling strategy is used, with $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$0 segments and random segment sampling during training, and $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$1 temporal segments with center frame per segment and single center crop at test time (Huang et al., 21 Sep 2025). The motion feature is

$\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$2

where $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$3 is an EfficientNet-B0 backbone adapted for 2-channel input and $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$4 (Huang et al., 21 Sep 2025).

3. Fusion head and training protocol

The fused vector is the concatenation of the 512-D CLIP feature and the 1280-D motion feature, giving

$\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$5

This vector is passed to a small MLP with a 1792-D input, a 512-D hidden layer, ReLU activation, dropout $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$6, and a final linear 101-way classifier (Huang et al., 21 Sep 2025). Only this MLP head is trained during fusion (Huang et al., 21 Sep 2025).

The fusion stage uses a standard cross-entropy loss, $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$7 (Huang et al., 21 Sep 2025). The paper does not introduce a special multimodal loss; the simplicity of the training objective is explicitly part of the efficiency argument (Huang et al., 21 Sep 2025). During late fusion, CLIP is frozen and the motion encoder is also frozen, so the model does not backpropagate through the large encoders during fusion training (Huang et al., 21 Sep 2025).

The implementation details reported for preprocessing and optimization are specific. The CLIP backbone is ViT-B/32 and frozen; the motion backbone is EfficientNet-B0 and ImageNet pre-trained; the code is implemented in PyTorch + CUDA and run on a single NVIDIA RTX 3090 (Huang et al., 21 Sep 2025). MV-only training uses 200 epochs, Adam, learning rate $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$8, weight decay $\mathbf{y} = \mathcal{C}_{\text{fusion}(\mathbf{f}_{\text{fusion}) .$9, and learning-rate drops at epochs 80 and 160 (Huang et al., 21 Sep 2025). Fusion training uses 50 epochs, AdamW, and learning rate $V=\{I_1,\dots,I_T\}$0 (Huang et al., 21 Sep 2025). Data augmentation includes multi-scale cropping and horizontal flip adapted for motion vectors, while motion-vector extraction uses the CoViAR library (Huang et al., 21 Sep 2025).

4. Efficiency profile

Efficiency is central to the method’s definition. The paper identifies three mechanisms: frozen encoders, offline CLIP feature caching, and a very small fusion head (Huang et al., 21 Sep 2025). CLIP appearance features are computed once offline and stored on disk, so inference does not repeatedly run the CLIP encoder (Huang et al., 21 Sep 2025). The fusion head contains only 0.97M trainable parameters, and the paper states that fusion adds only about 0.05 GFLOPs over the CLIP-only + MV-only baseline, making the runtime overhead negligible (Huang et al., 21 Sep 2025).

The reported FLOPs are defined under each method’s own test protocol:

$V=\{I_1,\dots,I_T\}$1

The paper emphasizes that this matters because methods use different numbers of temporal segments and crops (Huang et al., 21 Sep 2025).

The principal efficiency figures reported in the experimental section are summarized below.

Method	Trainable params	GFLOPs
CLIP-only	0	4.4
MV-only	5.3M	12.5
MoCLIP-Lite	0.97M	16.9

In the ablation table, CLIP-only is marked as having N/A throughput because CLIP features are cached offline, while MV-only reaches 286.85 vids/s and fusion reaches 275.31 vids/s (Huang et al., 21 Sep 2025). The proximity of 275.31 videos/sec to the MV-only branch is presented as evidence that the fused system retains very high throughput (Huang et al., 21 Sep 2025).

5. Experimental evaluation on UCF101

The experimental setup is confined in the provided record to UCF101, consisting of 13,320 video clips and 101 action classes, evaluated on Official Split 1 with 9,537 training videos and 3,783 test videos (Huang et al., 21 Sep 2025). The metric is Top-1 accuracy (%) (Huang et al., 21 Sep 2025).

The paper compares MoCLIP-Lite against three groups of baselines: zero-shot VLMs, MV-only methods, and fusion methods (Huang et al., 21 Sep 2025). The listed zero-shot VLMs are CLIP (ViT-B/32), X-CLIP, EPK-CLIP, and the authors’ own CLIP-only baseline (Huang et al., 21 Sep 2025). The MV-only methods are CoViAR (MV branch), EMV-CNN, DTMV-CNN, and the authors’ MV-only baseline (Huang et al., 21 Sep 2025). The fusion methods are Two-Stream, DTMV+RGB-CNN, and MoCLIP-Lite (Huang et al., 21 Sep 2025).

The main reported Top-1 accuracies are as follows.

Method	Top-1 accuracy (%)
CLIP (official zero-shot)	64.5
CLIP-only zero-shot baseline	65.0
X-CLIP	72.0
EPK-CLIP	75.3
CoViAR (MV branch)	63.9
MV-only baseline	66.5
EMV-CNN	79.3
DTMV-CNN	80.3
Two-Stream	86.2
DTMV+RGB-CNN	87.5
MoCLIP-Lite	89.2

MoCLIP-Lite thus achieves 89.2% Top-1 accuracy on UCF101, exceeding the CLIP-only zero-shot baseline at 65.0% and the MV-only baseline at 66.5% (Huang et al., 21 Sep 2025). The ablation table further reports a +22.1% absolute gain over the stronger single-stream baseline (Huang et al., 21 Sep 2025). The paper characterizes this as confirmation that the two modalities are strongly complementary (Huang et al., 21 Sep 2025).

Within the boundaries of the provided evidence, the result positions MoCLIP-Lite as an efficient baseline for video understanding that attains higher accuracy than the listed single-modality and classical fusion comparators on this benchmark (Huang et al., 21 Sep 2025). A plausible implication is that much of the performance gain comes not from increasing backbone scale, but from combining heterogeneous signals with low trainable capacity.

6. Qualitative behavior, limitations, and naming context

The qualitative and per-class analysis indicates that motion-dominant classes such as FloorGymnastics, FrisbeeCatch, JavelinThrow, RopeClimbing, and WallPushups improve further with fusion (Huang et al., 21 Sep 2025). Difficult classes such as Hammering are often misclassified by both single streams but corrected by MoCLIP-Lite, and visually driven actions like PizzaTossing benefit from CLIP semantics while still gaining from motion (Huang et al., 21 Sep 2025). A qualitative example reports that CLIP-only misclassifies Archery as Nunchucks, MV-only misclassifies it as Juggling Balls, and MoCLIP-Lite correctly predicts Archery (Huang et al., 21 Sep 2025).

Two limitations are stated explicitly. First, the appearance stream uses a single RGB frame, which may miss actions where the important object appears only briefly (Huang et al., 21 Sep 2025). Second, the fusion mechanism is simple concatenation, which is efficient but potentially weaker than attention-based fusion or transformer-based multimodal integration (Huang et al., 21 Sep 2025). The suggested future directions are using multiple frames for appearance, exploring attention or transformer fusion, and scaling to larger datasets (Huang et al., 21 Sep 2025).

The naming context merits clarification because adjacent literature contains superficially similar labels. "MobileViCLIP: An Efficient Video-Text Model for Mobile Devices" describes MobileViCLIP, MobileViCLIP-Tiny, and MobileViCLIP-Small as mobile-oriented variants derived from MobileCLIP; it explicitly states that there is no “MoCLIP-Lite” name in that paper and that such a phrase would most plausibly be a mistaken or informal reference to MobileViCLIP (Yang et al., 10 Aug 2025). Likewise, "CLIP for Lightweight Semantic Segmentation" presents a CLIP-guided segmentation framework with a Conv-Former fusion module and notes that “MoCLIP-Lite” is not the paper’s own formal name (Jin et al., 2023). Accordingly, in the current arXiv record the most precise use of MoCLIP-Lite refers to the UCF101 video action recognition framework introduced in 2025 rather than to the mobile video-text model or the lightweight segmentation method (Huang et al., 21 Sep 2025, Yang et al., 10 Aug 2025, Jin et al., 2023).