VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer

Abstract: With the rapid development of pre-training technologies, adapting large-scale Vision-LLMs (VLMs) for video understanding \emph{\ie} image-to-video transfer learning has become a dominant paradigm. To achieve superior performance, it raises as an effective strategy among recent advances to employ Mixture-of-Experts (MoE) to enhance VLMs' temporal modeling capabilities. However, conventional MoE designs suffer from expert homogenization, where all experts act as identical generalists, inefficiently learning spatio-temporal features from undifferentiated video streams. To overcome this problem, we propose VidPrism, a novel heterogeneous temporal Mixture-of-Experts framework. VidPrism pioneers a division of labor by deploying functionally specialized experts, each assuming a role ranging from spatial understanding to temporal modeling. To feed these specialists appropriately, we introduce a content-aware, multi-rate sampling module that dynamically generates streams ranging from semantically rich to motion-focused representations, providing specialized inputs for experts. Furthermore, a dynamic, bidirectional fusion mechanism enables synergistic information exchange between these pathways, leading to a comprehensive video representation. Extensive experiments on various video recognition benchmarks demonstrate that VidPrism achieves state-of-the-art performance and effectively fosters expert specialization. Our source code is available at \href{https://github.com/Lrrrr549/VidPrism.git}{https://github.com/Lrrrr549/VidPrism.git}.

• The paper introduces a heterogeneous MoE model that dynamically partitions video frames into specialized streams for improved feature extraction.
• It employs content-aware multi-rate sampling and dynamic bidirectional fusion to effectively combine spatial and temporal cues.
• Empirical results show state-of-the-art performance on benchmarks like Kinetics-400 and HMDB-51 with reduced computational cost.

VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer

Motivation and Problem Formulation

Image-to-video transfer leverages pre-trained Vision-LLMs (VLMs), such as CLIP, for video recognition tasks. While image foundation models excel in spatial reasoning, they lack robust mechanisms to capture the temporal dynamics inherent to video data. Conventional Mixture-of-Experts (MoE) architectures, when applied to video, typically assign homogeneous generalist experts, causing redundancy and diminishing specialization. This inefficiency motivates the need for explicit expert heterogeneity and content-adaptive pathways, as articulated by VidPrism.

VidPrism Architecture

VidPrism's pipeline is constructed around three principal modules: content-aware multi-rate input generation, dynamic bidirectional interaction, and a heterogeneous MoE (HMoE) head. Figure 1 provides an overview of the architecture, depicting the end-to-end flow from video input to classification output.

Figure 1: VidPrism end-to-end architecture combining content-aware multi-rate stratification, bidirectional pathway fusion, and heterogeneous expert aggregation.

Content-Aware Multi-Rate Sampling and Aggregation

The core Rate-guided Spatio-Temporal Aggregation Module (RgSTA) dynamically partitions the video into streams at varying temporal rates, enabling distinct spatial-semantic and motion-focused input pathways. A hybrid scoring mechanism fuses learnable importance prediction and intrinsic frame properties (e.g., L2 norm), efficiently identifying and aggregating informative frames while re-integrating dropped frame information via attention-weighted summation. This stratification robustly compresses sequences while maintaining information density.

Figure 2: STAM pipeline stratifies features, predicts importance scores, computes attention between sets, and aggregates rest-set information to the kept set.

Dynamic Bidirectional Fusion

VidPrism's Dynamic Bidirectional Interaction (DBI) module selectively exchanges information between pathways using learned gate scores derived from pathway representations. The exchange occurs along slow-to-fast (global context to motion detail) and fast-to-slow (fine temporal cues to structural context) streams, enhancing synergy and enabling optimal specialization and collaboration among experts.

Heterogeneous Expert Modeling and Combination

Each pathway is assigned a specialized Transformer-based expert exclusive to a specific temporal rate. This specialization enforces functional heterogeneity, as evidenced in the stark activation peaks shown in Figure 3, which contrasts VidPrism's focused attention with the flat distribution of homogeneous MoTE.

Figure 3: Inter-frame attention distribution for Dunk activity illustrates VidPrism’s distinct peaks at critical action moments versus MoTE’s diffuse attention.

A learnable global query vector aggregates outputs from all experts using multi-head cross-attention, effectively synthesizing spatial and temporal features for global classification.

Supervision Strategy

Four loss terms guide training:

• Classification Loss (cross-entropy) for global accuracy.
• Ranking Loss (KL divergence between predicted and target frame importance scores) for robust scoring.
• Diversity Loss (maximizing pairwise feature dissimilarities across experts) for functional differentiation.
• Gating Balancing Loss (regularizing expert contributions) for stable readout and improved generalization.

Empirical Validation

VidPrism achieves consistent state-of-the-art results on Kinetics-400, HMDB-51, UCF-101, and SSv2. Notably, VidPrism outperforms MoTE by 1.0% Top-1 on Kinetics-400 (ViT-B/16, 8 frames), and FocusVideo by achieving equivalent accuracy at lower computational cost (162 GFLOPs vs. 204 GFLOPs). Under four-expert configuration (rates 2, 4, 8, 16), VidPrism scales effectively with larger backbones (ViT-L/14 and VideoMAEv2), delivering improved scores across zero-shot, few-shot, and fully-supervised scenarios.

For $K$-shot tasks, VidPrism-C peaks at 74.1% on HMDB-51 and VidPrism-M reaches unprecedented 96.6% on UCF-101 ($K$=16). Ablation studies show that expert specialization, bidirectional fusion, and global attention are fundamentally necessary for optimal performance; best results are obtained only when all modules and supervision terms are present.

Expert Specialization Analysis

Visualization of expert usage in Figure 4 underscores functional heterogeneity: training and correct test samples show consistent, category-specific expert activation patterns, confirming specialization. Misclassified samples reveal failures in expert selection, providing diagnostic insight.

Figure 4: Expert usage heatmaps for UCF-101 distinguish stable, task-dependent activation (train, correct) from erratic usage in errors, evidencing functional division among experts.

Implications and Theoretical Significance

VidPrism substantively advances the architectural design for image-to-video transfer by enforcing heterogeneity and content-adaptive routing in the MoE paradigm. This approach bridges neuroscience-inspired dual-pathway hypotheses with scalable deep learning models, striking a balance between flexibility, efficiency, and representation power. The modular fusion and specialization mechanisms suggest a path forward for integrative video modeling, particularly as multimodal LLMs and video-language tasks become increasingly prominent.

Practically, VidPrism’s efficiency and adaptability enable deployment in resource-constrained settings while maintaining competitive accuracy. Theoretically, its expert specialization protocols can inform future MoE developments in vision, language, and cross-modal domains, with prospects for even more granular routing, deeper hierarchical organization, and multi-modal temporal reasoning.

Conclusion

VidPrism introduces a heterogeneous temporal Mixture-of-Experts architecture for image-to-video transfer learning. By dynamically generating multi-rate content-aware streams, enforcing expert functional specialization, and synergistically fusing spatial-temporal features, VidPrism achieves superior efficacy and efficiency across a spectrum of video recognition benchmarks. Its design principles and empirical successes provide a robust foundation for next-generation video understanding frameworks and stimulate future research on adaptive MoE architectures in multimodal AI systems (2605.28229).