---
title: 'Video Foundation Models: Unified Video Intelligence'
url: https://www.emergentmind.com/topics/video-foundation-models-vifms
type: topic
---

# Video Foundation Models: Unified Video Intelligence

Video Foundation Models (ViFMs) are large-scale, general-purpose video models intended to support a broad range of video tasks through shared representations rather than task-specific architectures. In formal terms, a ViFM can be written as a parameterized function $f_\theta : \{V, X\} \to Z$ that maps raw video $V\in\mathbb{R}^{T\times H\times W\times C}$ and, in multimodal settings, auxiliary inputs $X$ such as text or audio to a shared embedding space $Z\in\mathbb{R}^d$; pretraining typically combines generative and discriminative objectives over large, weakly curated corpora [2405.03770]. The term now covers not only video understanding but also generation, editing, instruction following, unified perception, and world modeling, reflecting a shift from isolated task solvers toward shared spatiotemporal backbones that can be adapted, prompted, or decoded for multiple modalities and outputs [2602.21835][2603.25892].

## 1. Conceptual scope and taxonomic organization

A standard taxonomy divides ViFMs into three categories. **Image-based ViFMs** adapt existing image models for video tasks, typically by adding temporal adapters, temporal prompt tuning, or post-pretraining. **Video-based ViFMs** use video-specific encoding methods such as encoder-only or encoder-decoder transformers operating on video tubes or spatiotemporal patches. **Universal Foundational Models (UFMs)** combine multiple modalities—image, video, audio, and text—within a single framework, often through joint-, dual-, or mix-encoder designs [2405.03770].

A complementary taxonomy distinguishes **clip-based ViFMs** from **instance-based ViFMs**. Clip-based systems operate on short video clips or frame sequences to produce sequence-level outputs, as in OmniVL, InternVideo, and VATT. Instance-based systems perform dense per-object inference over time and unify tasks such as single object tracking, multiple object tracking, and video object segmentation, as in OmniTracker, Unicorn, and UNINEXT. In this view, ViFMs function as “off-the-shelf” experts that can be composed to extract appearance cues, temporal dynamics, and audio/video semantics for downstream systems such as video question answering, content moderation, recommendation, and AR/VR [2304.14407].

The standard pretraining formulation emphasizes the coexistence of reconstruction and alignment objectives. The survey literature writes pretraining as
$$
\theta^* = \arg\min_\theta \mathbb{E}_{(V,X)\sim D}\big[\alpha L_{\mathrm{gen}}(f_\theta;V,X)+\beta L_{\mathrm{dis}}(f_\theta;V,X)\big],
$$
with masked video modeling as a canonical generative term and video–text contrastive learning as a canonical discriminative term. This formalization is useful because it captures the two dominant intuitions in the field: temporal structure can be learned by reconstruction of masked spatiotemporal content, and semantic structure can be learned by aligning video with auxiliary modalities such as language [2405.03770].

## 2. Architectures, objectives, and scaling regimes

The architectural center of gravity in ViFM research is the transformer. Image-based systems usually begin from a pretrained ViT or CLIP-like encoder and add temporal structure after the fact, whereas video-based systems tokenize space-time directly. Canonical pretraining objectives include masked video modeling
$$
L_{\mathrm{MVM}}=\|M\odot(V-\mathrm{Dec}(\mathrm{Enc}(M\odot V)))\|_2^2
$$
and video–text contrastive loss
$$
L_{\mathrm{VTC}}=-\log\frac{\exp(\mathrm{sim}(z_V,z_X)/\tau)}{\sum_{X'\in B}\exp(\mathrm{sim}(z_V,z_{X'})/\tau)}.
$$
Universal models add further matching, captioning, or next-token objectives, often with audio and language branches [2405.03770].

A representative large-scale design is InternVideo2, which uses a progressive training approach that unifies masked video modeling, crossmodal contrastive learning, and next token prediction, while scaling the video encoder to 6B parameters. Its data pipeline prioritizes spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions, and the reported pretraining corpus comprises approximately 412M unique training entries across modalities. The paper reports state-of-the-art results in video recognition, video-text tasks, video-centric dialogue, and long video understanding, with gains from model scaling and from multimodal caption fusion [2403.15377].

A different line of work argues that video capabilities can be harvested efficiently from image foundation models. “Harvest Video Foundation Models via Efficient Post-Pretraining” starts from CLIP’s pretrained ViT-B/16 and applies two simple modifications during post-pretraining: aggressive **video patch dropping** and **text masking**. With a default drop ratio $p=0.90$, the method keeps only 10% of patches, reduces per-GPU memory for batch size 128 from approximately 37 GB to approximately 16 GB, and completes 50k steps on 8×A100 GPUs in about 192 GPU-hours, using only WebVid-10M. The reported result is state-of-the-art performance on a range of zero-shot and fine-tuned video-language tasks at a fraction of the cost of conventional video-language pretraining [2310.19554].

At the systems level, scaling modern generative VFMs requires re-engineering the full pipeline from curation to inference. The NVIDIA NeMo recipe combines NeMo Curator for Internet-scale clipping, annotation, and WebDataset sharding; Megatron Energon for high-throughput mixed-modal loading and sequence packing; Megatron Core and Transformer Engine for 4D-parallel diffusion-transformer training; and a context-parallel inference engine. The paper describes operation at **100 PB+ of raw video**, cluster-wide GPU utilization of **90 + % even at 1 K GPUs**, MFU of **up to 48 % on H100 clusters**, and strong scaling efficiency **> 95 % when scaling from 8→32 nodes**. For inference, Cosmos-1.0-Diffusion-7B-Text2World is reported at **10 fps** on a single GPU in BF16 and **310 fps total** on 32 GPUs with FP8 plus tensor and context parallelism [2503.12964].

## 3. Adaptation mechanisms and unified task interfaces

Because many ViFMs are pretrained once and reused many times, adaptation is a core design variable. VideoGLUE evaluates four standard strategies: full end-to-end finetuning; a frozen backbone with a single-layer cross-attention pooler; a frozen backbone with multi-layer attention pooling (MLAP); and frozen backbones augmented with low-rank adapters of rank 64. The same study argues that both the task and the adaptation method materially affect downstream performance, especially when comparing video-native and image-native backbones [2307.03166].

Temporal aggregation is a particularly important adaptation point for image-based backbones. In the cross-view study of fine-grained action understanding, per-frame embeddings $\{z_t\}_{t=1}^T$ are combined through average pooling, max pooling, self-attention over full token sets or CLS tokens, cross-attention with a learnable query, LSTM, or TCN. The paper’s key methodological contribution is not a new backbone but a systematic comparison of temporal fusion schemes under viewpoint shift; it recommends modern image FMs, especially DINOv2, combined with lightweight self-attention on all tokens when deployment involves substantial viewpoint variation [2407.15605].

ViFMs are also being recast as promptable unified predictors rather than feature extractors. THFM is derived from a pretrained text-to-video diffusion model and repurposes it as a **single-forward-pass perception model**. It jointly addresses dense tasks—depth, normals, segmentation, dense pose—and sparse tasks—2D/3D keypoint estimation—within a single architecture, modulated by text prompts such as “estimate depth” or “output 3D keypoints.” For dense outputs, the model decodes a single denoiser pass into multi-channel RGB video encoding the target modality; for sparse outputs, it appends learnable query tokens with 3D Rotary Position Embedding and predicts keypoints with an MLP head. The entire model is trained exclusively on synthetic human-centric video rendered in Blender [2603.25892].

A different unification strategy appears in VFMF, which treats a vision foundation model feature space as the state space for world modeling. The method extracts DINOv2-like features, compresses them with a $\beta$-VAE into a compact latent, and then performs autoregressive rectified-flow forecasting in that latent space. The central claim is that stochastic conditional generation in VFM feature space preserves semantics and geometry while avoiding the full cost of pixel-space video generation; the sampled latent can then be decoded into segmentation, depth, normals, and RGB with lightweight heads [2512.11225]. This suggests that ViFM representations are increasingly being used not merely as descriptors of observed video, but as manipulable state variables for prediction and control.

## 4. Benchmarking, evaluation philosophy, and empirical patterns

A recurring theme in the literature is that benchmark design has lagged behind model ambition. VideoGLUE evaluates video understanding through three hallmark tasks—action recognition, temporal action localization, and spatiotemporal action localization—across eight datasets and adaptation-cost measurements. Its headline conclusion is cautionary: task-specialized models still significantly outperform the seven foundation models it studies, in contrast to the situation in NLP and image understanding. At the same time, VideoGLUE reports that video-native FMs are generally better than image-native FMs on motion-rich classification, temporal localization, and videos containing more than one action, especially under light adaptation [2307.03166].

VideoEval makes a related but broader point. It introduces VidTAB for few-shot task adaptation and VidEB for frozen-embedding retrieval, benchmarks **20 open-source vision foundation models**, and reports that current VFMs exhibit **weak generalization across diverse tasks**. The paper further states that increasing video data, whether labeled or weakly labeled video-text pairs, does not necessarily improve task performance, that some pretraining paradigms may not be fully validated by conventional benchmarks, and that combining different pretraining paradigms yields **+3–8 TA-score improvement** in multi-stage settings such as UMT-stage2 and InternVideo2-stage2. It also reports that DINOv2 outperforms most VFMs on VidEB retrieval, suggesting that spatial modeling quality remains a limiting factor for many video backbones [2407.06491].

Viewpoint robustness exposes another fault line. The cross-view study on DriveAct and IKEA-ASM argues that high accuracies on standard benchmarks can draw an artificially rosy picture because they often overlook changing camera perspectives. Under novel views, simple average pooling on image FMs collapses to **8–11 % mean acc**; max pooling adds **+3–5 % mean acc** over average pooling under view shift; and DINOv2 plus SelfAtt(ALL) reaches **21 % mean acc** on IKEA, compared with **11 %** for X-CLIP. On DriveAct, X-CLIP leads with **17.6 % mean acc**, but DINOv2 plus MaxPooling nearly matches it at **17.4 %**. The paper’s recommendation is therefore conditional rather than universal: use attention-based temporal aggregation for viewpoint variation, but consider specialized video FMs when rare actions are critical [2407.15605].

For unified video intelligence, UniVBench proposes a different evaluation philosophy. It assesses four core abilities—video understanding, video generation, video editing, and video reconstruction—through six tasks: V2T, T2V, R2V, TV2V, RV2V, and V2V. The dataset comprises **200 videos total: 100 single-shot and 100 multi-shot**, with detailed captions across **21 fine-grained sub-dimensions**, **864 high-fidelity reference images**, and an agentic evaluation engine, UniV-Eval, that standardizes prompting, instruction parsing, and scoring. The reported results show **Gemini 2.5 Pro** at **54.1 % average accuracy** for V2T, **Seedance-1.0-Pro** at **77.9 % overall** on T2V, and **Wan2.1-VACE-14B** leading TV2V, RV2V, and V2V at **65.1 %**, **66.4 %**, and **62.7 %**, respectively. A consistent pattern is that the **“Action” dimension is hardest across all tasks**, while static style attributes are easier; the paper emphasizes that no single model dominates every task [2602.21835].

Cross-domain scientific transfer adds yet another evaluation perspective. SciVid benchmarks five tasks spanning animal behavior, surgical tissue tracking, weather forecasting, and cyclone pressure forecasting, using simple readout modules on six frozen or partially fine-tuned ViFMs. It reports that frozen ViFMs plus simple readouts can match or exceed domain-specific baselines in several applications: **VideoPrism-g + 400 k steps: 92.5** mAP on FlyVsFly, **V-JEPA-H: 92.4** on CalMS21, and **4DS-L (frozen) 5.23** RMSE on Digital Typhoon. However, gaps remain on STIR and WeatherBench2 even after fine-tuning, and the paper concludes that no single backbone dominates all scientific tasks [2507.03578].

## 5. Representative systems and application domains

System-level work often assembles multiple ViFMs rather than relying on a single monolithic model. ChatVideo is explicitly **tracklet-centric**: it decomposes a video into spatiotemporal tracklets, annotates them with a “Foundation Model Pool” containing OmniTracker, OmniVL, Whisper, and Wav2Vec2-based emotion recognition, stores the outputs in a `Tracklets` database, and answers free-form questions by translating them into SQL queries. The result is a multimodal video understanding system oriented toward appearance, motion, and audio-related questions on in-the-wild videos [2304.14407].

In human–robot interaction, ViFMs have been used for intermittent supervision rather than recognition alone. “What did the Robot do in my Absence?” processes **40 minutes at 15 fps** of egocentric robot video and produces summaries as storyboards, short videos, or text. Its user study with **30 participants** finds that query-driven summaries significantly improve retrieval accuracy versus raw video with **$\beta_Q = +0.549$ ($p = 0.005$)**, that query-driven summaries outperform generic summaries with **$\beta = +0.9453$ ($p < 10^{-6}$)**, and that storyboards improve accuracy over video with **$\beta = +0.9054$ ($p = 0.004$)**, while text degrades accuracy with **$\beta = -0.9546$ ($p = 0.005$)**. The reported interpretation is that storyboards are particularly effective for object-related queries, whereas still frames are weaker for ongoing events [2411.10016].

ViFMs have also become semantic teachers for compression. Free-VSC introduces a VFM-shared semantic alignment layer with VFM-specific prompts, allowing several off-the-shelf VFMs—such as VideoMAEv2, DINOv2, and SSL-Swin—to define a shared semantic space for unsupervised video semantic compression. It adds a dynamic trajectory-based inter-frame compression scheme that predicts future semantics in latent space. The paper reports improvements over previous coding methods on action recognition, tracking, and video object segmentation, including **62.24 %** on HMDB51 at **0.02 bpp**, **72.80 % MOTA** and **73.21 % IDF1** on MOT17 at **0.01 bpp**, and **76.85 % $\mathcal{J}+\mathcal{F}$** on DAVIS2017 at **0.01 bpp** [2409.11718].

Unified perception is an especially clear application of the ViFM paradigm. THFM reports **Mean∠ = 11.01°** and **Median = 8.96°** on Hi4D surface normals, human depth RMSE/AbsRel of **0.021/0.011** on Face, **0.028/0.006** on UpperBody, and **0.037/0.007** on FullBody, and 3D keypoint accuracy on RICH of **PA-MPJPE = 34.7 mm**, **MPJPE = 55.0 mm**, and **Accel = 3.1 m/s²**. The model is trained exclusively on **20 000 synthetic human-centric videos** yet is reported to generalize at test time to multiple people, animals, and anthropomorphic characters. The paper attributes these emergent properties to diffusion-based video representations [2603.25892].

World modeling in ViFM feature space extends the same logic to prediction. VFMF compresses **3072-D features to 16-D latents**, reducing conditioning compute by approximately **200×**, uses a **12-layer** transformer forecaster, and samples with about **10 forward passes** of an Euler solver. On Cityscapes with context length $T=1$, the paper reports **mIoU-All 34.05**, **$\delta_1$ 88.1**, and **AbsRel 0.197** for mean-of-32 VFMF samples, compared with **31.7**, **84.8**, and **0.235** for deterministic DINO-RS; on Kubric, best-of-64 sampling reaches **70.55** mIoU-All and **0.079** AbsRel. The reported qualitative effect is sharper and more physically plausible alternative futures than deterministic regression [2512.11225].

## 6. Contested findings, limitations, and likely directions of development

A persistent misconception is that the literature has converged on a single best ViFM family. It has not. The survey literature states that **image-based foundation models consistently outperform video-based models on most video understanding tasks**, and that UFMs leveraging diverse modalities demonstrate superior performance on video tasks [2405.03770]. VideoGLUE, by contrast, reports that **video-native FMs** are generally better than image-native FMs on motion-rich videos, temporal localization, and multi-action video understanding, especially under frozen or lightly adapted regimes [2307.03166]. The cross-view study then shows that an image FM, **DINOv2**, combined with self-attention on all tokens can outperform heavier video FMs in many cross-view settings, while SciVid reports that pixel MAE-based ViFMs such as VideoMAE and 4DS excel in forecasting and tracking whereas DINOv2 lags in temporal understanding [2407.15605][2507.03578]. The safest conclusion is therefore conditional: backbone quality depends strongly on task structure, adaptation budget, and evaluation regime.

Several limitations recur across otherwise different papers. VideoEval reports weak few-shot generalization across diverse tasks [2407.06491]. UniVBench shows that dynamic **Action** understanding remains the hardest dimension across understanding, generation, editing, and reconstruction [2602.21835]. THFM notes mild task conflicts when dense and sparse tasks are jointly trained, and reports that heavy occlusions and very complex backgrounds remain challenging [2603.25892]. SciVid identifies severe domain gaps between scientific videos and web video, with weather forecasts still showing artifacts and over-smoothing [2507.03578]. In robot supervision, text summaries were disliked because they lacked visual evidence, and ViFMs were reported to underperform on spatial localization and counting [2411.10016]. Free-VSC adds an explicit caution that methods lowering the bitcost of semantic signals could be misused in privacy-sensitive surveillance or unwanted behavior monitoring [2409.11718].

The future directions proposed in the literature are correspondingly diverse. Several papers advocate **hybrid pretraining paradigms** that combine self-supervised and multimodal contrastive signals [2407.06491]. The cross-view literature calls for explicit **cross-view and multi-view pretraining** and lightweight attention-based temporal fusion modules [2407.15605]. THFM proposes expansion beyond human-centric perception toward open-domain video understanding, richer language grounding, and additional 4D tasks such as camera pose estimation, scene flow, and material or lighting inference [2603.25892]. UniVBench points toward audio, dialogues, and interactive editing loops within unified evaluation [2602.21835]. SciVid highlights long-range modeling and physics-informed backbones for scientific forecasting [2507.03578]. Taken together, these directions suggest that the field is moving from “video classification with larger backbones” toward a broader conception of ViFMs as reusable spatiotemporal substrates for perception, generation, retrieval, compression, forecasting, and instruction-following across domains.

Source: https://www.emergentmind.com/topics/video-foundation-models-vifms