Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

Abstract: Video foundation models achieve strong performance across many video understanding tasks, but typically require large-scale pre-training on massive video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive video models be built by reusing these spatial representations and pre-training only for temporal reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of video data and compute required compared to end-to-end video pre-training. In this work, we explore the feasibility of this approach before investing in computing for video pre-training. Our empirical findings across multiple video understanding tasks suggest that strong temporal performance can emerge without large-scale video pre-training, motivating future work on recurrent video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-video-image-frozen .

• The paper demonstrates that a frozen image encoder (e.g., DINOv3) paired with a lightweight temporal module achieves competitive video understanding with significantly reduced training data.
• It employs multi-depth feature extraction from multiple ViT layers, leading to systematic performance improvements across diverse video tasks.
• Empirical findings reveal that spatial features from image pre-training match or outperform video pre-training, thereby cutting down computational costs and data requirements.

Data-Efficient Video Pre-training with Frozen Image Foundation Models

Introduction

This work investigates the degree to which powerful pre-trained image foundation models (IFMs) can obviate the need for large-scale end-to-end video pre-training in video understanding tasks. Contemporary video foundation models (VFMs) typically require extensive pre-training on massive video corpora, inducing prohibitive data and computational costs. By contrast, image foundation models such as DINOv2 and DINOv3 offer advanced spatial representations acquired from billions of images. This provokes a fundamental inquiry: can VFMs be constructed by freezing a high-capacity IFM as the spatial backbone and focusing video data exclusively on a lightweight, modular temporal reasoning component?

Methodology

The proposed framework decouples spatial and temporal modeling. For each frame in a video, a frozen image encoder (e.g., DINOv3) generates multi-layer ViT features without gradient updates. These are aggregated (with multi-depth extraction shown to provide statistically significant gains) and passed to a trainable, recurrent temporal module—variously implemented as RVM\textsubscript{RNN}, Mamba, MambaMix, or the introduced GMMix. Finally, an attentive readout head produces task-specific video predictions under either streaming (causal) or offline (non-causal, all-frame) protocols.

The architecture supports systematic evaluation of:

• The comparative utility of image versus video encoder pre-training for the spatial backbone,
• The impact of temporal module architectural choice,
• The effect of pre-training the temporal module on video data versus random initialization, and
• Data efficiency when downstream adaptation is performed on limited video data.

Empirical Findings

Image vs. Video Pre-training for Spatial Encoders

Experimental results with a frozen DINOv3 encoder, paired with GMMix and trained only on downstream video data (e.g., for action recognition, object detection, depth estimation), demonstrate strong and in many cases superior performance relative to RVM (a state-of-the-art video pre-trained model). Notably, image pre-trained encoders consistently match or outperform their video pre-trained counterparts when only the temporal module and a readout head are trained, as visualized below.

Figure 1: The GMMix temporal module achieves comparable or better results with frozen image pre-trained encoders than with video pre-trained RVM, underscoring the representational strength of image models for spatial features.

Temporal Module Design and Multi-depth Features

Multiple temporal module designs were assessed. While RVM\textsubscript{RNN} provides strong results, the GMMix module achieves competitive performance, especially when supplied with multi-depth DINOv3 features. Across all temporal architectures and all video tasks, multi-depth feature extraction (from several ViT layers) is found to yield systematic performance improvements over final-layer-only token usage.

Figure 2: Multi-depth feature extraction from frozen DINOv3 improves or matches the use of only final-layer features for all benchmarks and temporal architectures.

Necessity and Transferability of Video Pre-training for the Temporal Module

By initializing the temporal module with pre-trained weights from RVM (rather than random weights), consistent improvements are observed, including when transplanting the pre-trained RVM temporal core onto a DINOv3 encoder. These positive transfer results indicate that temporal dynamics learned during video pre-training are at least partially encoder-agnostic and highlight the efficiency of training only lightweight temporal heads with video data atop a shared frozen image encoder.

Data Efficiency

The DINOv3+GMMix configuration not only outperforms frozen RVM on all video tasks but achieves frozen RVM's peak performance using less than 25% of the downstream video data.

Figure 3: DINOv3 + GMMix surpasses frozen RVM’s performance using less than one-quarter of the SSv2 training data, exemplifying pronounced gains in data efficiency.

Comparison to Prior Video Foundation Models

Direct comparison with established video foundation models (VideoMAE, V-JEPA, 4DS, and RVM) under a frozen-backbone regime demonstrates that DINOv3 + GMMix attains superior or competitive results on all video understanding benchmarks. For instance, on action recognition (SSv2), DINOv3 + GMMix (66.4 top-1 accuracy) surpasses all video pre-trained baselines. The empirical evidence robustly addresses both research questions: (1) spatial features from leading IFMs are at least as effective as those from video pre-training for video downstream tasks, and (2) strong video models can be built by training only the temporal module on limited video data.

Theoretical and Practical Implications

These results recalibrate the conventional wisdom regarding end-to-end video pre-training. They validate a modular approach in which an IFM is universally deployed as a frozen spatial backbone; model and application developers need only train compact, shareable temporal heads on modestly sized video corpora. This paradigm shift promises substantial reductions in both compute and data for building VFMs, facilitates reproducibility and model sharing, and potentially democratizes access to high-performance video understanding architectures.

The demonstrated transferability of video pre-trained temporal modules across different image encoders also suggests an underlying generality to temporal dynamics learned from video, opening directions for further research in encoder-agnostic temporal pre-training.

Future Directions

The work motivates large-scale video pre-training of temporal modules atop frozen IFMs, as well as systematic exploration of temporal module design for streaming, non-causal, and latency-sensitive video tasks. Expanding the space of IFM backbones, scaling up to XX-large ViTs, and accounting for more diverse pre-training objectives remains important. The decoupled approach may further drive research into modular, plug-and-play video models.

Conclusion

This study provides compelling evidence that a frozen image foundation model, when paired with a carefully constructed light temporal head, is sufficient for high-capacity video understanding on a wide array of tasks. It shows that large-scale end-to-end video pre-training is not strictly necessary; spatial and temporal modules can be effectively decoupled, vastly enhancing efficiency and flexibility. While future work remains in scaling temporal module pre-training, these findings chart a path toward more practical and adaptable video model development.