Video Foundation Models: Unified Video Intelligence

• Video Foundation Models (ViFMs) are large-scale architectures that learn shared spatiotemporal representations using both generative and discriminative pretraining.
• They encompass image-based, video-based, and universal models that leverage transformer designs for clip-based and instance-based analysis.
• ViFMs demonstrate strong scaling and adaptation capabilities, yet face challenges in benchmark design and cross-view generalization.

Video Foundation Models (ViFMs) are large-scale, general-purpose video models intended to support a broad range of video tasks through shared representations rather than task-specific architectures. In formal terms, a ViFM can be written as a parameterized function $f_\theta : \{V, X\} \to Z$ that maps raw video $V\in\mathbb{R}^{T\times H\times W\times C}$ and, in multimodal settings, auxiliary inputs $X$ such as text or audio to a shared embedding space $Z\in\mathbb{R}^d$; pretraining typically combines generative and discriminative objectives over large, weakly curated corpora (Madan et al., 2024). The term now covers not only video understanding but also generation, editing, instruction following, unified perception, and world modeling, reflecting a shift from isolated task solvers toward shared spatiotemporal backbones that can be adapted, prompted, or decoded for multiple modalities and outputs (Wei et al., 25 Feb 2026, Wang et al., 26 Mar 2026).

1. Conceptual scope and taxonomic organization

A standard taxonomy divides ViFMs into three categories. Image-based ViFMs adapt existing image models for video tasks, typically by adding temporal adapters, temporal prompt tuning, or post-pretraining. Video-based ViFMs use video-specific encoding methods such as encoder-only or encoder-decoder transformers operating on video tubes or spatiotemporal patches. Universal Foundational Models (UFMs) combine multiple modalities—image, video, audio, and text—within a single framework, often through joint-, dual-, or mix-encoder designs (Madan et al., 2024).

A complementary taxonomy distinguishes clip-based ViFMs from instance-based ViFMs. Clip-based systems operate on short video clips or frame sequences to produce sequence-level outputs, as in OmniVL, InternVideo, and VATT. Instance-based systems perform dense per-object inference over time and unify tasks such as single object tracking, multiple object tracking, and video object segmentation, as in OmniTracker, Unicorn, and UNINEXT. In this view, ViFMs function as “off-the-shelf” experts that can be composed to extract appearance cues, temporal dynamics, and audio/video semantics for downstream systems such as video question answering, content moderation, recommendation, and AR/VR (Wang et al., 2023).

The standard pretraining formulation emphasizes the coexistence of reconstruction and alignment objectives. The survey literature writes pretraining as

$$\theta^* = \arg\min_\theta \mathbb{E}_{(V,X)\sim D}\big[\alpha L_{\mathrm{gen}}(f_\theta;V,X)+\beta L_{\mathrm{dis}}(f_\theta;V,X)\big],$$

with masked video modeling as a canonical generative term and video–text contrastive learning as a canonical discriminative term. This formalization is useful because it captures the two dominant intuitions in the field: temporal structure can be learned by reconstruction of masked spatiotemporal content, and semantic structure can be learned by aligning video with auxiliary modalities such as language (Madan et al., 2024).

2. Architectures, objectives, and scaling regimes

The architectural center of gravity in ViFM research is the transformer. Image-based systems usually begin from a pretrained ViT or CLIP-like encoder and add temporal structure after the fact, whereas video-based systems tokenize space-time directly. Canonical pretraining objectives include masked video modeling

$$L_{\mathrm{MVM}}=\|M\odot(V-\mathrm{Dec}(\mathrm{Enc}(M\odot V)))\|_2^2$$

and video–text contrastive loss

$$L_{\mathrm{VTC}}=-\log\frac{\exp(\mathrm{sim}(z_V,z_X)/\tau)}{\sum_{X'\in B}\exp(\mathrm{sim}(z_V,z_{X'})/\tau)}.$$

Universal models add further matching, captioning, or next-token objectives, often with audio and language branches (Madan et al., 2024).

A representative large-scale design is InternVideo2, which uses a progressive training approach that unifies masked video modeling, crossmodal contrastive learning, and next token prediction, while scaling the video encoder to 6B parameters. Its data pipeline prioritizes spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions, and the reported pretraining corpus comprises approximately 412M unique training entries across modalities. The paper reports state-of-the-art results in video recognition, video-text tasks, video-centric dialogue, and long video understanding, with gains from model scaling and from multimodal caption fusion (Wang et al., 2024).

A different line of work argues that video capabilities can be harvested efficiently from image foundation models. “Harvest Video Foundation Models via Efficient Post-Pretraining” starts from CLIP’s pretrained ViT-B/16 and applies two simple modifications during post-pretraining: aggressive video patch dropping and text masking. With a default drop ratio $p=0.90$, the method keeps only 10% of patches, reduces per-GPU memory for batch size 128 from approximately 37 GB to approximately 16 GB, and completes 50k steps on 8×A100 GPUs in about 192 GPU-hours, using only WebVid-10M. The reported result is state-of-the-art performance on a range of zero-shot and fine-tuned video-language tasks at a fraction of the cost of conventional video-language pretraining (Li et al., 2023).

At the systems level, scaling modern generative VFMs requires re-engineering the full pipeline from curation to inference. The NVIDIA NeMo recipe combines NeMo Curator for Internet-scale clipping, annotation, and WebDataset sharding; Megatron Energon for high-throughput mixed-modal loading and sequence packing; Megatron Core and Transformer Engine for 4D-parallel diffusion-transformer training; and a context-parallel inference engine. The paper describes operation at 100 PB+ of raw video, cluster-wide GPU utilization of 90 + % even at 1 K GPUs, MFU of up to 48 % on H100 clusters, and strong scaling efficiency > 95 % when scaling from 8→32 nodes. For inference, Cosmos-1.0-Diffusion-7B-Text2World is reported at 10 fps on a single GPU in BF16 and 310 fps total on 32 GPUs with FP8 plus tensor and context parallelism (Patel et al., 17 Mar 2025).

3. Adaptation mechanisms and unified task interfaces

Because many ViFMs are pretrained once and reused many times, adaptation is a core design variable. VideoGLUE evaluates four standard strategies: full end-to-end finetuning; a frozen backbone with a single-layer cross-attention pooler; a frozen backbone with multi-layer attention pooling (MLAP); and frozen backbones augmented with low-rank adapters of rank 64. The same study argues that both the task and the adaptation method materially affect downstream performance, especially when comparing video-native and image-native backbones (Yuan et al., 2023).

Temporal aggregation is a particularly important adaptation point for image-based backbones. In the cross-view study of fine-grained action understanding, per-frame embeddings $\{z_t\}_{t=1}^T$ are combined through average pooling, max pooling, self-attention over full token sets or CLS tokens, cross-attention with a learnable query, LSTM, or TCN. The paper’s key methodological contribution is not a new backbone but a systematic comparison of temporal fusion schemes under viewpoint shift; it recommends modern image FMs, especially DINOv2, combined with lightweight self-attention on all tokens when deployment involves substantial viewpoint variation (Ponbagavathi et al., 2024).

ViFMs are also being recast as promptable unified predictors rather than feature extractors. THFM is derived from a pretrained text-to-video diffusion model and repurposes it as a single-forward-pass perception model. It jointly addresses dense tasks—depth, normals, segmentation, dense pose—and sparse tasks—2D/3D keypoint estimation—within a single architecture, modulated by text prompts such as “estimate depth” or “output 3D keypoints.” For dense outputs, the model decodes a single denoiser pass into multi-channel RGB video encoding the target modality; for sparse outputs, it appends learnable query tokens with 3D Rotary Position Embedding and predicts keypoints with an MLP head. The entire model is trained exclusively on synthetic human-centric video rendered in Blender (Wang et al., 26 Mar 2026).

A different unification strategy appears in VFMF, which treats a vision foundation model feature space as the state space for world modeling. The method extracts DINOv2-like features, compresses them with a $\beta$-VAE into a compact latent, and then performs autoregressive rectified-flow forecasting in that latent space. The central claim is that stochastic conditional generation in VFM feature space preserves semantics and geometry while avoiding the full cost of pixel-space video generation; the sampled latent can then be decoded into segmentation, depth, normals, and RGB with lightweight heads (Boduljak et al., 12 Dec 2025). This suggests that ViFM representations are increasingly being used not merely as descriptors of observed video, but as manipulable state variables for prediction and control.

4. Benchmarking, evaluation philosophy, and empirical patterns

A recurring theme in the literature is that benchmark design has lagged behind model ambition. VideoGLUE evaluates video understanding through three hallmark tasks—action recognition, temporal action localization, and spatiotemporal action localization—across eight datasets and adaptation-cost measurements. Its headline conclusion is cautionary: task-specialized models still significantly outperform the seven foundation models it studies, in contrast to the situation in NLP and image understanding. At the same time, VideoGLUE reports that video-native FMs are generally better than image-native FMs on motion-rich classification, temporal localization, and videos containing more than one action, especially under light adaptation (Yuan et al., 2023).

VideoEval makes a related but broader point. It introduces VidTAB for few-shot task adaptation and VidEB for frozen-embedding retrieval, benchmarks 20 open-source vision foundation models, and reports that current VFMs exhibit weak generalization across diverse tasks. The paper further states that increasing video data, whether labeled or weakly labeled video-text pairs, does not necessarily improve task performance, that some pretraining paradigms may not be fully validated by conventional benchmarks, and that combining different pretraining paradigms yields +3–8 TA-score improvement in multi-stage settings such as UMT-stage2 and InternVideo2-stage2. It also reports that DINOv2 outperforms most VFMs on VidEB retrieval, suggesting that spatial modeling quality remains a limiting factor for many video backbones (Li et al., 2024).

Viewpoint robustness exposes another fault line. The cross-view study on DriveAct and IKEA-ASM argues that high accuracies on standard benchmarks can draw an artificially rosy picture because they often overlook changing camera perspectives. Under novel views, simple average pooling on image FMs collapses to 8–11 % mean acc; max pooling adds +3–5 % mean acc over average pooling under view shift; and DINOv2 plus SelfAtt(ALL) reaches 21 % mean acc on IKEA, compared with 11 % for X-CLIP. On DriveAct, X-CLIP leads with 17.6 % mean acc, but DINOv2 plus MaxPooling nearly matches it at 17.4 %. The paper’s recommendation is therefore conditional rather than universal: use attention-based temporal aggregation for viewpoint variation, but consider specialized video FMs when rare actions are critical (Ponbagavathi et al., 2024).

For unified video intelligence, UniVBench proposes a different evaluation philosophy. It assesses four core abilities—video understanding, video generation, video editing, and video reconstruction—through six tasks: V2T, T2V, R2V, TV2V, RV2V, and V2V. The dataset comprises 200 videos total: 100 single-shot and 100 multi-shot, with detailed captions across 21 fine-grained sub-dimensions, 864 high-fidelity reference images, and an agentic evaluation engine, UniV-Eval, that standardizes prompting, instruction parsing, and scoring. The reported results show Gemini 2.5 Pro at 54.1 % average accuracy for V2T, Seedance-1.0-Pro at 77.9 % overall on T2V, and Wan2.1-VACE-14B leading TV2V, RV2V, and V2V at 65.1 %, 66.4 %, and 62.7 %, respectively. A consistent pattern is that the “Action” dimension is hardest across all tasks, while static style attributes are easier; the paper emphasizes that no single model dominates every task (Wei et al., 25 Feb 2026).

Cross-domain scientific transfer adds yet another evaluation perspective. SciVid benchmarks five tasks spanning animal behavior, surgical tissue tracking, weather forecasting, and cyclone pressure forecasting, using simple readout modules on six frozen or partially fine-tuned ViFMs. It reports that frozen ViFMs plus simple readouts can match or exceed domain-specific baselines in several applications: VideoPrism-g + 400 k steps: 92.5 mAP on FlyVsFly, V-JEPA-H: 92.4 on CalMS21, and 4DS-L (frozen) 5.23 RMSE on Digital Typhoon. However, gaps remain on STIR and WeatherBench2 even after fine-tuning, and the paper concludes that no single backbone dominates all scientific tasks (Hasson et al., 4 Jul 2025).

5. Representative systems and application domains

System-level work often assembles multiple ViFMs rather than relying on a single monolithic model. ChatVideo is explicitly tracklet-centric: it decomposes a video into spatiotemporal tracklets, annotates them with a “Foundation Model Pool” containing OmniTracker, OmniVL, Whisper, and Wav2Vec2-based emotion recognition, stores the outputs in a Tracklets database, and answers free-form questions by translating them into SQL queries. The result is a multimodal video understanding system oriented toward appearance, motion, and audio-related questions on in-the-wild videos (Wang et al., 2023).

In human–robot interaction, ViFMs have been used for intermittent supervision rather than recognition alone. “What did the Robot do in my Absence?” processes 40 minutes at 15 fps of egocentric robot video and produces summaries as storyboards, short videos, or text. Its user study with 30 participants finds that query-driven summaries significantly improve retrieval accuracy versus raw video with $V\in\mathbb{R}^{T\times H\times W\times C}$0 ($V\in\mathbb{R}^{T\times H\times W\times C}$1), that query-driven summaries outperform generic summaries with $V\in\mathbb{R}^{T\times H\times W\times C}$2 ($V\in\mathbb{R}^{T\times H\times W\times C}$3), and that storyboards improve accuracy over video with $V\in\mathbb{R}^{T\times H\times W\times C}$4 ($V\in\mathbb{R}^{T\times H\times W\times C}$5), while text degrades accuracy with $V\in\mathbb{R}^{T\times H\times W\times C}$6 ($V\in\mathbb{R}^{T\times H\times W\times C}$7). The reported interpretation is that storyboards are particularly effective for object-related queries, whereas still frames are weaker for ongoing events (Katuwandeniya et al., 2024).

ViFMs have also become semantic teachers for compression. Free-VSC introduces a VFM-shared semantic alignment layer with VFM-specific prompts, allowing several off-the-shelf VFMs—such as VideoMAEv2, DINOv2, and SSL-Swin—to define a shared semantic space for unsupervised video semantic compression. It adds a dynamic trajectory-based inter-frame compression scheme that predicts future semantics in latent space. The paper reports improvements over previous coding methods on action recognition, tracking, and video object segmentation, including 62.24 % on HMDB51 at 0.02 bpp, 72.80 % MOTA and 73.21 % IDF1 on MOT17 at 0.01 bpp, and 76.85 % $V\in\mathbb{R}^{T\times H\times W\times C}$8 on DAVIS2017 at 0.01 bpp (Tian et al., 2024).

Unified perception is an especially clear application of the ViFM paradigm. THFM reports Mean∠ = 11.01° and Median = 8.96° on Hi4D surface normals, human depth RMSE/AbsRel of 0.021/0.011 on Face, 0.028/0.006 on UpperBody, and 0.037/0.007 on FullBody, and 3D keypoint accuracy on RICH of PA-MPJPE = 34.7 mm, MPJPE = 55.0 mm, and Accel = 3.1 m/s². The model is trained exclusively on 20 000 synthetic human-centric videos yet is reported to generalize at test time to multiple people, animals, and anthropomorphic characters. The paper attributes these emergent properties to diffusion-based video representations (Wang et al., 26 Mar 2026).

World modeling in ViFM feature space extends the same logic to prediction. VFMF compresses 3072-D features to 16-D latents, reducing conditioning compute by approximately 200×, uses a 12-layer transformer forecaster, and samples with about 10 forward passes of an Euler solver. On Cityscapes with context length $V\in\mathbb{R}^{T\times H\times W\times C}$9, the paper reports mIoU-All 34.05, $X$0 88.1, and AbsRel 0.197 for mean-of-32 VFMF samples, compared with 31.7, 84.8, and 0.235 for deterministic DINO-RS; on Kubric, best-of-64 sampling reaches 70.55 mIoU-All and 0.079 AbsRel. The reported qualitative effect is sharper and more physically plausible alternative futures than deterministic regression (Boduljak et al., 12 Dec 2025).

6. Contested findings, limitations, and likely directions of development

A persistent misconception is that the literature has converged on a single best ViFM family. It has not. The survey literature states that image-based foundation models consistently outperform video-based models on most video understanding tasks, and that UFMs leveraging diverse modalities demonstrate superior performance on video tasks (Madan et al., 2024). VideoGLUE, by contrast, reports that video-native FMs are generally better than image-native FMs on motion-rich videos, temporal localization, and multi-action video understanding, especially under frozen or lightly adapted regimes (Yuan et al., 2023). The cross-view study then shows that an image FM, DINOv2, combined with self-attention on all tokens can outperform heavier video FMs in many cross-view settings, while SciVid reports that pixel MAE-based ViFMs such as VideoMAE and 4DS excel in forecasting and tracking whereas DINOv2 lags in temporal understanding (Ponbagavathi et al., 2024, Hasson et al., 4 Jul 2025). The safest conclusion is therefore conditional: backbone quality depends strongly on task structure, adaptation budget, and evaluation regime.

Several limitations recur across otherwise different papers. VideoEval reports weak few-shot generalization across diverse tasks (Li et al., 2024). UniVBench shows that dynamic Action understanding remains the hardest dimension across understanding, generation, editing, and reconstruction (Wei et al., 25 Feb 2026). THFM notes mild task conflicts when dense and sparse tasks are jointly trained, and reports that heavy occlusions and very complex backgrounds remain challenging (Wang et al., 26 Mar 2026). SciVid identifies severe domain gaps between scientific videos and web video, with weather forecasts still showing artifacts and over-smoothing (Hasson et al., 4 Jul 2025). In robot supervision, text summaries were disliked because they lacked visual evidence, and ViFMs were reported to underperform on spatial localization and counting (Katuwandeniya et al., 2024). Free-VSC adds an explicit caution that methods lowering the bitcost of semantic signals could be misused in privacy-sensitive surveillance or unwanted behavior monitoring (Tian et al., 2024).

The future directions proposed in the literature are correspondingly diverse. Several papers advocate hybrid pretraining paradigms that combine self-supervised and multimodal contrastive signals (Li et al., 2024). The cross-view literature calls for explicit cross-view and multi-view pretraining and lightweight attention-based temporal fusion modules (Ponbagavathi et al., 2024). THFM proposes expansion beyond human-centric perception toward open-domain video understanding, richer language grounding, and additional 4D tasks such as camera pose estimation, scene flow, and material or lighting inference (Wang et al., 26 Mar 2026). UniVBench points toward audio, dialogues, and interactive editing loops within unified evaluation (Wei et al., 25 Feb 2026). SciVid highlights long-range modeling and physics-informed backbones for scientific forecasting (Hasson et al., 4 Jul 2025). Taken together, these directions suggest that the field is moving from “video classification with larger backbones” toward a broader conception of ViFMs as reusable spatiotemporal substrates for perception, generation, retrieval, compression, forecasting, and instruction-following across domains.