V-JEPA2: Self-Supervised Video Model

• V-JEPA2 is a self-supervised video model that uses latent prediction instead of pixel reconstruction to generate unified spatiotemporal embeddings.
• It processes complete video clips with a Vision Transformer backbone, capturing motion trajectories, temporal coherence, and semantic detail.
• Empirical studies show V-JEPA2 delivers stable temporal representations for diverse actions, though it may offer less class separation than frame-wise models.

V-JEPA2 is a self-supervised video model in the Joint Embedding Predictive Architecture family. In recent literature it is described, on the one hand, as a scaled-up latent video prediction system supporting downstream understanding, prediction, and planning, and, on the other, as a video representation model that jointly processes clips and performs masked prediction in representation space rather than pixel reconstruction (Pandya, 11 Jun 2026, Kodathala et al., 25 Sep 2025). Across downstream studies, V-JEPA2 is repeatedly used as a frozen visual backbone whose principal value lies in temporally informative, semantically rich spatiotemporal features, especially for tasks in which motion trajectories, anticipation, temporal coherence, or event timing matter more than static appearance alone (Rowles et al., 24 Oct 2025).

1. JEPA-family position and conceptual role

Within the JEPA lineage, V-JEPA2 is cited alongside I-JEPA and V-JEPA as part of the family initiated by the JEPA principle. In that framing, its defining orientation is latent prediction rather than input-space reconstruction, and its downstream promise is not limited to representation extraction but extends to understanding, prediction, and planning (Pandya, 11 Jun 2026).

The comparative action-analysis study gives the most explicit baseline-level summary of how V-JEPA2 is understood in practice. It describes the pretrained facebook/vjepa2-vitl-fpc64-256 model as a Video Joint Embedding Predictive Architecture with a Vision Transformer Large backbone. Unlike frame-wise image encoders, V-JEPA2 “jointly processes the complete 16-frame sequence,” performs spatiotemporal tokenization, and generates representations that “explicitly model temporal dependencies and cross-frame relationships.” Its self-supervised pretraining is described as using masked prediction objectives in representation space rather than pixel reconstruction. The same study presents V-JEPA2 as the temporally integrated counterpart to DINOv3’s frame-independent spatial processing (Kodathala et al., 25 Sep 2025).

A recurrent theme across the cited works is that V-JEPA2 is not treated as a generic video encoder. In Foley generation, the stated motivation for choosing it is that it is a large self-supervised predictive video model designed to capture “predictable dynamics such as motion trajectories,” with strong results on motion understanding, anticipation, and temporal reasoning benchmarks. This positioning is important because the downstream objective there is not merely semantic recognition but temporal grounding of local events such as contact timing, motion evolution, and likely transient onsets (Rowles et al., 24 Oct 2025).

2. Representational interface and model variants

The available papers describe V-JEPA2 primarily through the interfaces exposed to downstream systems rather than through a full internal architectural specification. Even so, several concrete usage patterns recur. In the UCF Sports comparison, V-JEPA2 receives a full clip of shape $(16, 3, 256, 256)$ and outputs a unified sequence-level embedding

$v_{seq} \in \mathbb{R}^{1024}.$

This is contrasted directly with DINOv3, which produces frame-level $768$-dimensional vectors that must then be pooled over time (Kodathala et al., 25 Sep 2025).

The surgical phase segmentation study evaluates two V-JEPA2 backbones as frozen feature extractors. The specified variants are V-JEPA2 ViT-L, with input resolution $256$, clip length $64$, feature dimension $1024$, and $326.0$M parameters, and V-JEPA2 ViT-g/16, with input resolution $384$, clip length $64$, feature dimension $1408$, and $v_{seq} \in \mathbb{R}^{1024}.$0B parameters. Both are listed as self-supervised video models from Meta, and both are integrated into the same cached-feature pipeline so that representation quality can be isolated from temporal-head design (Spencer et al., 12 Apr 2026).

A different interface appears in video-guided Foley. There, V-JEPA2 is run on $v_{seq} \in \mathbb{R}^{1024}.$1 FPS video streams. For each $v_{seq} \in \mathbb{R}^{1024}.$2-second segment, $v_{seq} \in \mathbb{R}^{1024}.$3 frames are sampled and encoded; because the encoder operates with stride $v_{seq} \in \mathbb{R}^{1024}.$4, one “effective frame” corresponds to two input frames, yielding $v_{seq} \in \mathbb{R}^{1024}.$5 effective temporal units per segment. Although the encoder originally provides patch-level spatiotemporal embeddings, the deployed configuration pools each effective frame to a single token, producing $v_{seq} \in \mathbb{R}^{1024}.$6 tokens per $v_{seq} \in \mathbb{R}^{1024}.$7 seconds and up to $v_{seq} \in \mathbb{R}^{1024}.$8 tokens for a $v_{seq} \in \mathbb{R}^{1024}.$9-second clip. The authors summarize the pooling conceptually as

$768$0

and then pass the pooled sequence through a lightweight two-layer MLP with GELU and residual addition,

$768$1

while keeping V-JEPA2 itself frozen (Rowles et al., 24 Oct 2025).

These interfaces show that V-JEPA2 can support both sequence-level and token-sequence use. A plausible implication is that the model’s downstream utility often depends less on end-to-end retraining than on how its pretrained spatiotemporal latents are exposed to later modules.

3. Temporal representation characteristics

The clearest direct evidence about V-JEPA2’s representational behavior comes from the UCF Sports comparison between DINOv3 and V-JEPA2. In that study, V-JEPA2 processes all $768$2 videos successfully, corresponding to a $768$3 feature-extraction success rate, and produces $768$4-dimensional sequence embeddings. On clustering metrics, it achieves a Silhouette score of $768$5 and a Calinski-Harabasz index of $768$6. On $768$7-nearest-neighbors classification, the reported accuracies are $768$8 for $768$9, $256$0 for $256$1, and $256$2 for $256$3 (Kodathala et al., 25 Sep 2025).

The most informative numbers concern similarity structure. The mean intra-class similarity is reported as $256$4, while the mean inter-class similarity is $256$5. Per-action intra-class values range from $256$6 to $256$7, and inter-class similarities range from $256$8 to $256$9. The action-level variance is

$64$0

substantially below the comparison model’s $64$1. The study interprets these values as evidence that V-JEPA2 yields coherent and stable representations across action categories, including motion-dependent ones, but that its embedding space is less sharply discriminative because different classes remain relatively close together (Kodathala et al., 25 Sep 2025).

This temporal profile is not universally dominant. The same study reports that DINOv3 achieves better clustering separation and slightly better nearest-neighbor accuracy, especially for pose-identifiable actions. The practical trade-off it identifies is therefore not simple superiority, but a distinction between temporal reliability and class-separation sharpness. V-JEPA2 is presented as more dependable across heterogeneous action categories, whereas frame-wise spatial features can be more discriminative when static pose cues dominate (Kodathala et al., 25 Sep 2025).

4. V-JEPA2 in video-guided Foley

The most detailed downstream integration of V-JEPA2 appears in “Foley Control,” which connects V-JEPA2 video embeddings to a frozen Stable Audio Open latent diffusion transformer. The design keeps the pretrained single-modality backbones frozen and learns only a small cross-attention bridge, on the premise that V-JEPA2 already supplies temporally informative, semantically rich video features and Stable Audio Open already supplies a strong audio prior and text-conditioned audio generation marginal. The central claim is that only the audio–video dependency required for synchronization must be learned (Rowles et al., 24 Oct 2025).

The audio side follows the standard latent audio diffusion setup: a waveform $64$2 is encoded by an audio VAE into latent audio tokens

$64$3

while a video segment $64$4 is encoded into video tokens

$64$5

Stable Audio Open is used as a frozen DiT trained with velocity prediction, summarized as predicting

$64$6

at diffusion step $64$7, with the new conditioning entering only through the bridge (Rowles et al., 24 Oct 2025).

Architecturally, Foley Control does not merge text and video into a single multimodal sequence. Each DiT block is modified to insert a video cross-attention sublayer immediately after the existing text cross-attention and before the feed-forward network: $64$8 The rationale is explicit: text prompts set global semantics and overall scene structure first, and video then refines timing and localized dynamics. With audio hidden states $64$9 as queries and adapted V-JEPA2 tokens $1024$0 as keys and values, the added attention is described in standard form,

$1024$1

$1024$2

followed by output projection. Rotary position embeddings are applied independently to audio queries and video keys, with separate position indices and phases for the two modalities; the authors state that this was sufficient to obtain stable temporal alignment in their adapter setting (Rowles et al., 24 Oct 2025).

The empirical ablation most directly about V-JEPA2 compares dense $1024$3 grid tokens against a single pooled embedding per effective frame. Using KL-PANNs on a validation split without text prompts, the reported values are essentially tied: at $1024$4k steps, $1024$5 for Grid8 versus $1024$6 for single pooled; at $1024$7k, $1024$8 versus $1024$9; at $326.0$0k, $326.0$1 versus $326.0$2; at $326.0$3k, $326.0$4 versus $326.0$5; and at $326.0$6k, $326.0$7 versus $326.0$8. The authors conclude that “the mid-level spatial resolution of the single-pooled embeddings captures sufficient motion and context cues for Foley synchronization.” This result is central to their interpretation of V-JEPA2: aggressive compression barely harms performance, which suggests that its representations already encode compact motion semantics useful for audiovisual alignment (Rowles et al., 24 Oct 2025).

5. Controlled downstream evaluations beyond audiovisual generation

In surgical phase segmentation on SICS-155, V-JEPA2 is evaluated as a frozen visual encoder under an otherwise identical MS-TCN++ pipeline. Because V-JEPA2 is clip-based rather than frame-based, the study uses a sliding-window procedure over $326.0$9-frame clips, assigns each clip embedding to its center frame, extracts features every fourth frame, and linearly interpolates back to full annotation rate. Under this setup, V-JEPA2 ViT-L obtains $384$0 accuracy, $384$1 macro-F1, $384$2 edit score, $384$3 PR-AUC, and $384$4 mIoU. V-JEPA2 ViT-g/16 is lower at $384$5 accuracy, $384$6 macro-F1, $384$7 edit score, $384$8 PR-AUC, and $384$9 mIoU. In this controlled regime, V-JEPA2 ViT-L improves over ResNet-50 on all listed metrics, but does not beat I3D overall and is clearly below DINOv3 ViT-L and DINOv3 ViT-7B. The paper’s explanation is that stride-$64$0 extraction plus interpolation may smooth rapid transitions and weaken boundary cues, and that domain shift from generic pretraining video is a second plausible factor. Its CataractFT transfer experiment reports negative transfer for V-JEPA2 ViT-L, with Acc (FT) $64$1, $64$2Acc $64$3, $64$4Macro-F1 $64$5, and $64$6Edit $64$7 (Spencer et al., 12 Apr 2026).

The action-analysis study provides a complementary picture. There V-JEPA2 is favored when the application requires stable behavior across diverse action types, especially motion-dependent categories such as walking and running, where frame-wise processing degrades more strongly. Yet the same evidence shows that V-JEPA2 is not the most discriminative representation in that benchmark. Taken together, these two downstream studies support a narrower characterization: V-JEPA2 is a strong temporal representation source, but its advantage depends on whether the evaluation rewards temporal coherence, robustness, and uniformity more than fine-grained class separation or sharply localized boundary detection (Kodathala et al., 25 Sep 2025, Spencer et al., 12 Apr 2026).

6. Limitations, misconceptions, and extension directions

A common misconception would be to treat V-JEPA2’s video-native design as sufficient for best-in-class downstream results in every setting. The available evidence does not support that view. In UCF Sports, V-JEPA2 is more stable across action categories but less sharply discriminative than DINOv3. In SICS phase segmentation, it is useful but not the strongest frozen representation in the tested pipeline, and the larger ViT-g/16 variant does not improve on ViT-L (Kodathala et al., 25 Sep 2025, Spencer et al., 12 Apr 2026).

Another misconception would be to assume that downstream use of V-JEPA2 requires dense spatial token retention. Foley Control reports the opposite tendency: pooling each effective frame to a single token preserves performance closely enough that the compressed representation becomes the main configuration. The authors nonetheless note a real limitation: pooled V-JEPA2 tokens may miss rare fine-grained spatial cues, and the current setup caps video duration. They also identify open directions including adaptive tokenization, learned pooling, longer-context conditioning, spatial audio, more varied data, robustness to in-the-wild edits and background music, and extension of the same bridge design to speech and dialogue (Rowles et al., 24 Oct 2025).

The cited literature also leaves some questions open about V-JEPA2 itself. The action-analysis paper explicitly notes that it uses V-JEPA2 only as a baseline and does not provide deeper internal architectural specifics. The surgical study does not test end-to-end fine-tuning, denser extraction for V-JEPA2, alternative temporal heads, or label-fraction scaling curves. These limitations matter because they constrain what can be concluded about V-JEPA2 as opposed to the pipelines built around it (Kodathala et al., 25 Sep 2025, Spencer et al., 12 Apr 2026).

A plausible implication of later JEPA-family work is that future V-JEPA2-like systems may be evaluated not only as frozen encoders but also as latent predictors that adapt under distribution shift. EPM-JEPA, which cites V-JEPA2 as prior JEPA-family context, does not modify V-JEPA2 directly, but it argues that operator-side low-rank modulation of a JEPA predictor can be more promising than hidden-state additive injection when test-time dynamics diverge from training. That proposal is explicitly conceptual rather than architectural one-to-one, but it identifies adaptive predictor design, EMA-target drift, and non-equilibrium training dynamics as issues likely to matter for larger latent video prediction systems as well (Pandya, 11 Jun 2026).