Video Joint-Embedding Predictive Architecture

• The paper demonstrates that V-JEPA learns predictive video embeddings by regressing masked tubelet features in latent space, avoiding pixel-level reconstruction.
• Its architecture employs transformer encoders with EMA-based target networks and multi-block contiguous masking to capture robust spatiotemporal dependencies.
• Results show strong performance in action recognition, motion analysis, and effective transfer to modalities like EEG and vision-language tasks.

A Video Joint-Embedding Predictive Architecture (V-JEPA) is a family of self-supervised spatiotemporal representation learning frameworks for video, designed to learn predictive, high-level visual embeddings by regressing masked patch features in latent space, rather than performing pixel-level reconstruction or contrastive discrimination. V-JEPA architectures are scalable, modular, and can be straightforwardly extended to non-video domains (such as multichannel EEG time series) or to multimodal settings (e.g., vision-language). The V-JEPA principle has led to strong empirical results on video understanding, motion analysis, world modeling for control, and interpretable concept discovery, and has given rise to a variety of architectural innovations and theoretical generalizations in recent literature.

1. Core Architectural Components and Workflow

The canonical V-JEPA consists of a transformer-based encoder, an embedding projector, a mask-conditioned predictor, and a target encoder updated via exponential moving average (EMA) (Bardes et al., 2024, Assran et al., 11 Jun 2025, Hojjati et al., 4 Jul 2025, Eing et al., 14 Jan 2026). The input video is first patchified into non-overlapping spatio-temporal "tubelets." A stochastic masking scheme—typically multi-block contiguous masking—removes a substantial fraction (often ~90%) of tubelets, which are replaced by learned mask tokens plus positional encodings. The encoder processes masked inputs into latent representations, and a shallow projector maps the encoder output into a joint embedding space. The predictor, a narrower transformer or MLP, takes visible context tokens (plus mask-location tokens) and outputs predicted embeddings for each masked position. In parallel, a target encoder (maintained as an EMA of the online encoder) processes the full unmasked video to yield regression targets.

The joint-embedding objective regresses the predictor outputs toward the stop-gradiented target encoder representations using a per-patch $\ell_1$ or $\ell_2$ loss over the masked regions. No contrastive negatives, pixel-space reconstruction, or text are required in the core objective. The formal loss for patch index set $M$ is:

$$\mathcal{L}_{\mathrm{JEPA}} = \sum_{i \in M} \bigl\| \hat{p}_i - \mathrm{sg}(p^*_i)\bigr\|_1 + \lambda R(\theta, \psi, \phi),$$

where $\hat{p}_i$ is the prediction for masked position $i$, $p^*_i$ is the target embedding from the EMA encoder, and $R$ denotes optional regularization (Hojjati et al., 4 Jul 2025, Bardes et al., 2024). EMA updates stabilize training and prevent representational collapse.

This data flow and objective focus the learning signal on semantic prediction in latent space, enabling efficient capture of high-level spatiotemporal structure. Cross-attention pooling with a learned CLS token is commonly used for downstream probing under the "frozen encoder" protocol (Bardes et al., 2024, Assran et al., 11 Jun 2025).

2. Masking Strategies and Predictive Design

Masking is central in V-JEPA's self-supervised signal. The default masking strategy is multi-block contiguous masking, which selects a random mix of short-range small blocks and long-range large 3D spatio-temporal blocks, typically removing up to 90% of tubelets per clip (Bardes et al., 2024, Assran et al., 11 Jun 2025, Li et al., 29 Sep 2025). Sampling strategies may alternate between causal and non-causal blocks for more flexible context/target separation.

Key properties:

• Contiguous block masking enforces modeling of both local and nonlocal dependencies. Empirically, arbitrary block masking outperforms random tube masking, especially for temporal understanding.
• Adaptive mask rate schedules, e.g., linear ramping of mask ratio from $\alpha_{\text{start}}$ to $\alpha_{\text{end}}$, can be used for curriculum learning (Hojjati et al., 4 Jul 2025).
• Spatial/temporal constraints in specialized domains ensure, e.g., channel continuity for EEG patches (Hojjati et al., 4 Jul 2025), or avoid completely masking any one channel.

Mask positions are encoded as input to the predictor; learnable mask tokens integrate positional context for each region to be predicted, allowing efficient mask queries and context-conditioned regression (Bardes et al., 2024).

3. Extensions, Generalizations, and Regularization

V-JEPA has been extended along several axes:

• Domain adaptation: The architecture is adapted to non-image sequences (EEG), with input volumes constructed by sliding multi-channel time windows, 3D convolutional patchifying, domain-specific masking, and ViT-based encoders. This approach yields semantically meaningful, physiologically interpretable EEG embeddings for abnormality detection and concept discovery (Hojjati et al., 4 Jul 2025).
• Multitask extensions: MC-JEPA incorporates optical flow prediction and content prediction jointly in a shared encoder using additional heads and training objectives, resulting in features that encode both motion and content information for video segmentation and flow (Bardes et al., 2023).
• Variance–covariance regularization: VJ-VCR applies VICReg-style variance-covariance regularization at each time step to avoid representational collapse, boosting rank and the information content of the learned representations, especially in convolutional (non-transformer) JEPA variants (Drozdov et al., 2024).
• Bayesian and variational extensions: VJEPA generalizes deterministic JEPA to stochastic latent prediction, defining variational objectives over future latent states, unifying representation learning with Predictive State Representations and Bayesian filtering, and supporting robust uncertainty-aware planning and belief propagation (Huang, 20 Jan 2026).

4. Training Schemes, Efficiency, and Scalability

V-JEPA pretraining operates at significant scale, typically over millions of videos sampled from datasets such as Kinetics, HowTo100M, SSv2, and VideoMix, with clip lengths from 16 up to 64 frames at spatial resolution from $224^2$ to $384^2$ (Bardes et al., 2024, Assran et al., 11 Jun 2025). Large transformer backbones (e.g., ViT-L, ViT-H, ViT-g) are trained for tens to hundreds of thousands of steps using large batch sizes (up to 3072), step-based or progressive resolution scheduling, AdamW optimization, heavy masking, and aggressive data augmentation.

Variants in pretraining schedule and architecture have been shown to affect compute efficiency:

• EMA-based vs. frozen teacher: The Static-teacher Asymmetric Latent Training (SALT) method pre-trains a pixel-space MAE teacher, freezes it, and only then trains the student to predict latent targets. This decoupling is more compute-optimal, allows for smaller teachers and larger students, and provides more consistent loss-to-accuracy correlation, strictly dominating standard EMA V-JEPA 2 in accuracy-FLOPs trade-off (Li et al., 29 Sep 2025).
• Attentive pooling significantly outperforms average pooling for frozen probes, with cross-attention heads yielding +17% on K400 in JEPA compared to baseline averaging (Bardes et al., 2024, Eing et al., 14 Jan 2026).
• Multi-task allocation in MC-JEPA shows that balancing flow and content losses improves dense prediction tasks without degrading segmentation or flow accuracy substantially (Bardes et al., 2023).

5. Downstream Performance and Empirical Results

V-JEPA-based models demonstrate strong generalization across a range of video, image, and multimodal benchmarks. Empirical highlights include:

Model	K400 (frozen)	SSv2 (frozen)	ImageNet1K	State-of-the-art Tasks
V-JEPA H/16_384 (Bardes et al., 2024)	81.9	72.2	77.9	Temporal reasoning (SSv2), label efficiency
V-JEPA 2 g_384 (Assran et al., 11 Jun 2025)	87.3	77.3	85.1	SOTA motion & action anticipation
VJ-VCR (Drozdov et al., 2024)	-	-	-	Physics action probing, collapse avoidance
MC-JEPA (Bardes et al., 2023)	-	-	-	Optical flow + segmentation (67.1 mIoU)
EEG-VJEPA (Hojjati et al., 4 Jul 2025)	-	-	-	Abnormal EEG detection, visualizable concepts

V-JEPA models trained with frozen probes transfer effectively for action recognition, motion understanding, and video question answering (after multimodal alignment), outperforming or matching pixel-reconstruction MAEs, contrastive models, and generative baselines, often with higher label efficiency and superior representation content. In EEG-VJEPA, the extracted embeddings not only improve abnormality classification but also enable downstream visual concept analysis, revealing age, gender, and abnormality clusters, and yielding interpretable spatiotemporal saliency (Hojjati et al., 4 Jul 2025).

6. Downstream Applications and Interpretability

V-JEPA embeddings have been fruitfully applied to:

• Frozen linear probing for action recognition: Cross-attention "attentive probes" over per-patch embeddings yield strong top-1 accuracy on Kinetics-400, Something-Something-v2, and Epic-Kitchens (Bardes et al., 2024, Assran et al., 11 Jun 2025).
• Facial expression recognition: Pretrained V-JEPA video encoders combined with shallow attentive classifiers outperform pixel-level MAEs and show strong cross-dataset generalization without encoder fine-tuning (Eing et al., 14 Jan 2026).
• EEG event detection: Video-adapted V-JEPA encoders reveal physiologically aligned concept embeddings, with attention rollout methods highlighting discriminative signal bands and spatial regions, and reflecting clinical EEG expert knowledge (Hojjati et al., 4 Jul 2025).
• Physics and world modeling: Variational extensions (VJEPA, BJEPA) yield robust, uncertainty-aware latent dynamics for planning, filtering, and zero-shot constraint satisfaction via product-of-experts posterior fusion (Huang, 20 Jan 2026).
• Vision-language and multimodal models: VL-JEPA uses V-JEPA video encoders and joint embedding targets to efficiently bridge video and natural language, supporting open-vocabulary classification, retrieval, VQA, and selective text decoding (Chen et al., 11 Dec 2025).

Qualitative analyses confirm that V-JEPA learns abstract, high-level, and semantically meaningful features as visualized by conditional diffusion decoders and embedding UMAP projections. These features exhibit spatial and temporal consistency, model uncertainty, and preserve rich structure necessary for downstream interpretability (Bardes et al., 2024, Drozdov et al., 2024).

7. Theoretical Foundations and Prospective Directions

The theoretical underpinnings of V-JEPA have been formalized via connections to predictive information, PSRs, and Bayesian filtering. Variational JEPA establishes that predictive latent states (obtained without pixel reconstruction) are sufficient for downstream control and planning, provided the state captures the mutual information between context and future (Huang, 20 Jan 2026). Product-of-experts extensions further allow modular incorporation of prior knowledge or external constraints.

Recent developments include:

• Uncertainty-aware, robust world modeling by extending JEPA to stochastic, variational objectives (Huang, 20 Jan 2026).
• Architectural decoupling and teacher–student scaling for compute-optimal training schedules (Li et al., 29 Sep 2025).
• Variance-covariance and energy-based regularization to ensure non-collapse in more classical CNN or energy-based formulations (Drozdov et al., 2024).

Open research directions involve extending V-JEPA architectures to longer contexts, interactive control, hierarchical planning, scaling to billions of parameters and multi-modality, and integrating with emerging language-vision frameworks (Assran et al., 11 Jun 2025, Chen et al., 11 Dec 2025).

---

References

• (Bardes et al., 2024) Revisiting Feature Prediction for Learning Visual Representations from Video
• (Assran et al., 11 Jun 2025) V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• (Hojjati et al., 4 Jul 2025) From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Visual Concepts in Brain Signal Analysis
• (Eing et al., 14 Jan 2026) Video Joint-Embedding Predictive Architectures for Facial Expression Recognition
• (Drozdov et al., 2024) Video Representation Learning with Joint-Embedding Predictive Architectures
• (Bardes et al., 2023) MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
• (Chen et al., 11 Dec 2025) VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
• (Li et al., 29 Sep 2025) Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
• (Huang, 20 Jan 2026) VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Models