MobileViCLIP: Efficient Mobile Video-Text Model

• The paper introduces MobileViCLIP, which adapts an image–text backbone with spatiotemporal modules for efficient mobile video–text processing.
• It converts MobileCLIP into a spatiotemporal model via temporal pooling and reparameterization, preserving strong zero-shot retrieval performance.
• MobileViCLIP-Small achieves up to 55.4× speedup over baselines on mobile devices while maintaining competitive accuracy on benchmarks.

Searching arXiv for the specified paper to ground the article. MobileViCLIP is an efficient video–text model for mobile devices that converts an efficient image–text backbone into a spatiotemporal model by introducing temporal structural reparameterization and then training on a large-scale high-quality video–text dataset (Yang et al., 10 Aug 2025). It is positioned against video pre-trained models that still focus on the common ViT architecture with high latency, and is designed to retain strong zero-shot classification and retrieval capabilities while substantially reducing mobile inference latency. In the reported evaluation, MobileViCLIP-Small is 55.4x times faster than InternVideo2-L14 and 6.7x faster than InternVideo2-S14 on mobile devices, while achieving similar zero-shot retrieval performance to InternVideo2-L14 and exceeding InternVideo2-S14 by 6.9\% on MSR-VTT (Yang et al., 10 Aug 2025).

1. Architectural basis and model variants

MobileViCLIP is built on MobileCLIP and is presented in two variants: MobileCLIP-S0, denoted “Tiny,” and MobileCLIP-S2, denoted “Small” (Yang et al., 10 Aug 2025). In the Tiny configuration, the image encoder is MCi$_0$, described as a hybrid CNN–Transformer from FastViT, and the text encoder is MCt, described as RepMixer-based. The parameter counts are reported as Params(video)$=11.4$ M and Params(text)$=42.4$ M. In the Small configuration, the image encoder is MCi$_2$, characterized as wider/deeper than MCi$_0$, while the text encoder is a 12-layer Transformer in the ViT-B/16-CLIP style, with Params(video)$=35.8$ M and Params(text)$=63.4$ M (Yang et al., 10 Aug 2025).

The conversion from image–text to video–text proceeds by fine-tuning MobileCLIP’s image branch into a spatiotemporal video branch, freezing the text branch, and fully fine-tuning the video branch on video–text pairs. Video-level features are obtained by simple temporal pooling, specifically averaging over frame embeddings (Yang et al., 10 Aug 2025). This design concentrates adaptation capacity in the visual branch while preserving the text encoder learned during image–text pre-training.

Two spatiotemporal modules are added to each MCi block: Spatiotemporal RepMixer and Spatiotemporal Attention. The former uses depthwise convolution along time and space; the latter combines addable temporal positional embedding with 2D conditional positional encoding. This modular modification is central to the model’s mobile orientation, because it augments temporal modeling without replacing the efficient base backbone with a heavier video transformer (Yang et al., 10 Aug 2025).

2. Spatiotemporal formulation and reparameterization

The paper specifies Spatiotemporal RepMixer for an input feature tensor $X\in\mathbb R^{T\times H\times W\times C}$. At training time, batch normalization, a 1D depthwise convolution over the time axis, and a 2D depthwise convolution over spatial axes are each used with residual skips: $$X' \;=\; \mathrm{DWConv1D}\bigl(\mathrm{BN}(X)\bigr)\;+\;X, \qquad Y \;=\; \mathrm{DWConv2D}\bigl(\mathrm{BN}(X')\bigr)\;+\;X'.$$ At inference, these two layers are re-parameterized, with BN and skip fused into plain convolutions: $$X' = \mathrm{DWConv1D}(X), \quad Y = \mathrm{DWConv2D}(X').$$ This structural re-parameterization is one of the deployment-oriented mechanisms that reduces runtime overhead on mobile hardware (Yang et al., 10 Aug 2025).

Spatiotemporal Attention is defined for the same tensor flattened to $=11.4$0. A learnable temporal positional encoding, TPE, of shape $=11.4$1 is added, after which the original conditional positional encoding, implemented as a 2D depthwise convolution, and self-attention are applied: $=11.4$2 At inference, the CPE convolution and TPE addition can be folded similarly (Yang et al., 10 Aug 2025). This suggests that MobileViCLIP does not treat temporal modeling as a separate heavy-stage video stack; instead, temporal information is embedded directly into lightweight backbone blocks.

For video–text feature extraction, frame-level embeddings are computed as

$=11.4$3

The video-level representation is then obtained by temporal average pooling: $=11.4$4 This choice is consistent with the stated use of simple temporal pooling over frame embeddings (Yang et al., 10 Aug 2025).

3. Training objective and data pipeline

The training objective is a bi-directional contrastive loss of the InfoNCE type. Given a batch $=11.4$5 of video and text embeddings and a learnable temperature $=11.4$6, cosine similarity is defined as $=11.4$7. The directional losses are

$=11.4$8

and the final objective is

$=11.4$9

The loss construction aligns video and text embeddings symmetrically and matches the model’s zero-shot retrieval and classification use cases (Yang et al., 10 Aug 2025).

Pre-training is organized in two stages. Stage 1 uses DataCompDR-1B, described as 1 billion web images plus captions via reinforcement, for image–text pre-training. Stage 2 performs video–text fine-tuning on InternVid-10M-FLT, a collection of 10 million YouTube clips covering 16 scene categories and approximately 6 000 action concepts, with fine-grained captions generated or filtered by LLMs (Yang et al., 10 Aug 2025).

The preprocessing and augmentation pipeline samples 8 frames per video, resizes to $=42.4$0, and applies random crop plus horizontal flip with $=42.4$1 during training. No trainable temporal augmentations are used (Yang et al., 10 Aug 2025). A plausible implication is that the model’s temporal robustness is expected to emerge primarily from architectural adaptation and large-scale video–text supervision rather than explicit temporal augmentation policies.

4. Scale, latency, and mobile deployment

The model size and complexity are reported separately for video and text branches. For the Tiny variant, the paper reports 14.9 G FLOPs for video and 1.3 G for text, with 11.5 M and 42.4 M parameters respectively. For the Small variant, it reports 49.5 G FLOPs for video and 4.3 G for text, with 36.0 M and 63.4 M parameters respectively (Yang et al., 10 Aug 2025).

Inference speed is measured on an iPad Air 2020 with A14 Bionic, iOS 18, and Core ML Tools v7.1, with batch size 1. InternVideo2-L14 is reported at 2 298.7 ms for video and 20.5 ms for text, for approximately 2 319 ms total. InternVideo2-S14 is reported at 275 ms for video and 7.2 ms for text, totaling 282.2 ms. MobileViCLIP-Small is reported at 195.7 ms for video and 15.4 ms for text, totaling 211.1 ms on GPU, and 37.8 ms for video plus 4.0 ms for text, totaling 41.8 ms on mobile (Yang et al., 10 Aug 2025).

These latency figures underpin the stated speedup: MobileViCLIP-Small is approximately 55.4× faster than InternVideo2-L14 and approximately 6.7× faster than InternVideo2-S14 on mobile (Yang et al., 10 Aug 2025). In deployment terms, the model is exported with Core ML Tools v7.1 to .mlmodel for iOS-18 and A14 Bionic. Structural re-parameterization fuses BN and skip into plain convolutions at inference, and checkpointing is used during training to reduce peak GPU memory (Yang et al., 10 Aug 2025).

Future work mentioned in the source includes quantization, for example 8-bit, and TensorRT/CoreML graph optimizations (Yang et al., 10 Aug 2025). This suggests that the reported mobile latency is not presented as a terminal lower bound but as a baseline that may admit further systems-level gains.

5. Zero-shot retrieval and classification results

The principal retrieval benchmark reported is zero-shot text-to-video retrieval on MSR-VTT using R@1. InternVideo2-L14 reaches T2V$=42.4$2 and V2T$=42.4$3. InternVideo2-S14 reaches T2V$=42.4$4 and V2T$=42.4$5. MobileViCLIP-Small reaches T2V$=42.4$6, which is $=42.4$7 over L14 and $=42.4$8 over S14, and V2T$=42.4$9, which is $_2$0 versus L14 and $_2$1 versus S14 (Yang et al., 10 Aug 2025).

On zero-shot action classification, top-1 video-to-text accuracy is reported as 63.1\% on Kinetics-400 for the Small model, $_2$2 over InternVideo2-S14; 81.0\% on UCF-101, $_2$3; 53.7\% on HMDB-51, $_2$4; and 46.1\% on SSV2-MC, $_2$5 (Yang et al., 10 Aug 2025). The aggregate pattern is not uniformly superior across all benchmarks, since SSV2-MC shows a small deficit relative to InternVideo2-S14, but it is competitive or stronger on the other listed datasets.

A concise comparison is as follows:

Model	Mobile latency	MSR-VTT R@1
InternVideo2-L14	$_2$6 ms	T2V 42.1\%, V2T 44.1\%
InternVideo2-S14	282.2 ms	T2V 35.6\%, V2T 35.9\%
MobileViCLIP-Small	41.8 ms	T2V 42.5\%, V2T 43.5\%

The reported results support a specific efficiency profile: relative to large or standard video CLIP baselines, MobileViCLIP-Small preserves near-parity retrieval quality with the largest baseline while sharply reducing latency on real mobile hardware (Yang et al., 10 Aug 2025).

6. Ablations, design trade-offs, and interpretive context

The ablation studies isolate the effect of the temporal modules on MSR-VTT retrieval. Starting from a baseline R@1 of 38.4, Spatiotemporal RepMixer only yields $_2$7 to 39.5. Spatiotemporal Attention without TPE yields $_2$8 to 39.1. Spatiotemporal Attention with TPE yields $_2$9 to 39.6. Using both modules with TPE yields $_0$0 to 40.1 (Yang et al., 10 Aug 2025). These results indicate that the two temporal components are complementary and that TPE contributes positively within the attention branch.

Another ablation concerns optimization strategy rather than architectural content: freezing the text branch saves approximately 3 GB of GPU memory, from 27 GB to 24 GB, with no drop in R@1 (Yang et al., 10 Aug 2025). This matters because the training recipe explicitly freezes the text branch while fully fine-tuning the video branch. The result suggests that, within the reported setup, performance is limited more by video adaptation capacity than by further text-encoder adjustment.

A common misconception in efficient video–language modeling is that latency improvements necessarily require large retrieval-quality sacrifices. The reported MSR-VTT numbers complicate that view: MobileViCLIP-Small is similar to InternVideo2-L14 on retrieval while being much faster on mobile, yet its advantage is not universal across all downstream settings, as seen in the $_0$1 result on SSV2-MC relative to InternVideo2-S14 (Yang et al., 10 Aug 2025). Another possible misconception is that temporal modeling in mobile architectures must rely on complex trainable temporal augmentation or large temporal transformers; the described system instead uses lightweight spatiotemporal modules, simple frame averaging, and no trainable temporal augmentations.

Taken together, the design suggests a broader pattern in efficient multimodal modeling: strong video–text behavior can be obtained by adapting an efficient image–text backbone with structurally re-parameterizable temporal operators, provided that fine-tuning is performed on a sufficiently large and high-quality video–text corpus (Yang et al., 10 Aug 2025).