V-JEPA 2 Video Classifier

• V-JEPA 2 video classifier is a neural system using masked latent prediction and a frozen ViT backbone to yield robust spatio-temporal representations.
• It employs a lightweight attentive probe with shallow Transformer layers to aggregate features for effective downstream classification.
• The approach achieves state-of-the-art performance on benchmarks like Something-Something v2 and Kinetics-400, demonstrating strong robustness against input corruptions and distribution shifts.

A V-JEPA 2 video classifier is a neural system that leverages the masked latent prediction paradigm—first instantiated in the Video Joint-Embedding Predictive Architectures (V-JEPA) framework—and its large-scale, self-supervised ViT backbone to yield robust, generalizable video classification with minimal task-specific supervision. In contrast to pixel-reconstruction paradigms, V-JEPA 2 learns to predict the latent representations of masked spatio-temporal patches given unmasked context, with training performed at web scale and downstream probing via lightweight attentive classifiers. This approach has achieved state-of-the-art performance on motion-centric video understanding tasks, robust generalization under distribution shift, and practical deployment across domains including facial expression recognition, action anticipation, and ethological video modeling (Eing et al., 14 Jan 2026, Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Kodathala et al., 25 Sep 2025, Li et al., 29 Sep 2025, Mueller et al., 12 Nov 2025, Alrasheed et al., 15 May 2026).

1. Architectural Foundations: Backbone and Probe

The V-JEPA 2 classifier architecture comprises two main modules: (i) a frozen backbone—usually a large-scale Vision Transformer (ViT) pretrained in a masked latent prediction regime; (ii) a lightweight probing head, most commonly an attentive probe composed of shallow Transformer layers and a learnable query or class token for reading out global sequence representations (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Eing et al., 14 Jan 2026, Mueller et al., 12 Nov 2025, Alrasheed et al., 15 May 2026).

Key elements of the backbone:

• Video tokenization: Inputs are clipped to blocks of 16 frames (commonly at 4–5 fps), each frame split into non-overlapping 16×16 spatial patches, grouped temporally as tubelets of 2 frames, resulting in a space-time grid (e.g., for 256×256 input: 16×16 spatial patches, 8 temporal tubelets, totaling 8×16×16=2048 tokens per clip) (Assran et al., 11 Jun 2025, Eing et al., 14 Jan 2026, Kodathala et al., 25 Sep 2025, Mueller et al., 12 Nov 2025).
• Patch embedding: Each patch is linearly projected into a high-dimensional embedding (D=1024–2048 depending on backbone scale).
• 3D positional encoding: Positional signals (either sinusoidal or rotary embeddings) are added to all tokens across time, height, and width (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026).
• Transformer core: Standard ViT includes 24–48 Transformer layers, typically with 16–32 heads and hidden dimensions matched to D (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Kodathala et al., 25 Sep 2025).
• Frozen encoder: For downstream classification, all encoder weights are fixed, ensuring the learned video representations are not corrupted or overfit by limited supervised data (Assran et al., 11 Jun 2025, Eing et al., 14 Jan 2026, Alrasheed et al., 15 May 2026, Mueller et al., 12 Nov 2025).

The “attentive probe” head operates as follows:

• Shallow Transformer stack: 2–4 self-attention blocks, each with dimension D′=D or a downprojected width (e.g., D'=64 for data-limited regimes) (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Eing et al., 14 Jan 2026, Mueller et al., 12 Nov 2025).
• Query or class token: A learned query, or per-class token(s), performs cross-attention to aggregate all spatio-temporal tokens into a pooled representation.
• Classification layer: A linear head maps the representation to class logits, followed by softmax or sigmoid activation (Mur-Labadia et al., 15 Mar 2026, Mueller et al., 12 Nov 2025, Eing et al., 14 Jan 2026).
• Training regime: Only the probe is trained, typically with AdamW, cross-entropy loss, and moderate learning rates (1e-4–5e-4) (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Eing et al., 14 Jan 2026, Mueller et al., 12 Nov 2025, Alrasheed et al., 15 May 2026).

2. Self-Supervised Pretraining: Masked Latent Prediction

V-JEPA 2's training is centered on latent space prediction, not pixel-level reconstruction. The core objective is to predict the representations of masked tubelets (patches) from unmasked context, using two parallel but weight-decoupled encoders:

• Context encoder $E_\theta$: Processes masked input, attending only to visible tubelets (Assran et al., 11 Jun 2025, Eing et al., 14 Jan 2026).
• Target encoder $\hat{E}_\theta$: Exponential moving average of $E_\theta$; processes the full, unmasked sequence and generates latent targets.
• Predictor $P_\phi$: Receives context features and masking pattern, outputs predicted embeddings for masked patches (Assran et al., 11 Jun 2025, Eing et al., 14 Jan 2026, Mueller et al., 12 Nov 2025).

The principal loss (for V-JEPA 2, L1 norm) is:

$$\mathcal{L}_{\rm JEPA} = \sum_{i\in \text{masked}} \| s_y^{(i)} - \hat{s}_y^{(i)} \|_1$$

where $s_y^{(i)}$ are target encoder embeddings and $\hat{s}_y^{(i)}$ are predictor outputs for the $i$-th masked tubelet (Eing et al., 14 Jan 2026, Assran et al., 11 Jun 2025, Mueller et al., 12 Nov 2025).

In V-JEPA 2.1, the loss is extended to "dense predictive loss" by adding a context consistency term for visible tokens, and is applied hierarchically at multiple encoder depths (Mur-Labadia et al., 15 Mar 2026). This approach strongly encourages spatially and temporally grounded representations.

The pretraining masking regime is aggressive: typically, 70–90% of tubelets are masked, using both blockwise spatial masks and full-temporal tubelets. Optimizer is AdamW with large batch sizes (up to 3072 clips), and multi-stage learning rate schedules include initial warmup and learning rate annealing coupled with increased input resolution and longer clip duration (“cooldown”) (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026).

3. Downstream Classification Protocols

The standard V-JEPA 2 classifier workflow for video categorization consists of:

1. Backbone freezing: The pretrained ViT-based encoder is frozen and used as a feature extractor (Assran et al., 11 Jun 2025, Alrasheed et al., 15 May 2026).
2. Attentive probe head: A new classifier, typically 2–4 layers of self-attention with a query- or class-token, is attached for label prediction (Alrasheed et al., 15 May 2026, Mur-Labadia et al., 15 Mar 2026, Eing et al., 14 Jan 2026, Mueller et al., 12 Nov 2025).
3. Input processing: Videos are parsed into overlapping 16-frame clips, spatially cropped to 224–256^2, with color/flip/temporal jitter augmentations during probe training (Eing et al., 14 Jan 2026, Assran et al., 11 Jun 2025, Alrasheed et al., 15 May 2026, Mueller et al., 12 Nov 2025).
4. Probe training: The classifier is trained via cross-entropy on only the labeled probe weights, using moderate learning rates and regularization. Batch sizes range from 32 to 256 depending on GPU scale (Assran et al., 11 Jun 2025, Eing et al., 14 Jan 2026, Mueller et al., 12 Nov 2025, Alrasheed et al., 15 May 2026).
5. Evaluation and aggregation: At inference, predictions are made for each overlapping video clip and combined (e.g., via posteriors-based voting or mean logits) to yield final video-level predictions (Eing et al., 14 Jan 2026, Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Alrasheed et al., 15 May 2026).
6. Alternative probe regimes: For small or non-parametric deployments, k-NN on latent features (as from [CLS] or global tokens) is also used (Kodathala et al., 25 Sep 2025).

4. Empirical Results and Robustness

V-JEPA 2 video classifiers are state-of-the-art on a variety of benchmarks, with particular strength on motion-centric and robustness axes:

• Something-Something v2: V-JEPA 2 ViT-G@384 achieves 77.7% top-1 (SOTA, state-of-the-art) (Mur-Labadia et al., 15 Mar 2026). V-JEPA 2 ViT-g@384 achieves 77.3% (Assran et al., 11 Jun 2025).
• Kinetics-400: V-JEPA 2.1 ViT-G: 87.7%, V-JEPA 2 ViT-g: 87.3%, close to best supervised InternVideo2 systems (Mur-Labadia et al., 15 Mar 2026).
• Facial expression recognition: On CREMA-D, Frozen V-JEPA 2 + shallow probe achieves 78.86% WAR, outperforming MAE-DFER and all other vision-based methods. RAVDESS test: 76.40% UAR, 72.93% WAR (Eing et al., 14 Jan 2026).
• UCF Sports k-NN (appearance-motion balance): V-JEPA 2 achieves 87.9% accuracy (k=1), with consistent intra-class similarity for both pose- and motion-centric actions (Kodathala et al., 25 Sep 2025).
• Corruption and occlusion robustness: V-JEPA 2 + frozen probe retains higher accuracy under patch dropout and a range of input corruptions compared to VideoMAE and TimeSformer (e.g., on SSv2 at patch severity 5: 16.8% vs. 10.2% for VideoMAE), and maintains temporal direction sensitivity (DSCS: 0.31 vs. 0.08 for VideoMAE) (Alrasheed et al., 15 May 2026).
• Generalization: Cross-dataset expression recognition shows robust transfer: training on CREMA-D, tested on RAVDESS yields 75.59% WAR; converse direction achieves 59.82% WAR (Eing et al., 14 Jan 2026).
• Comparison to image-only SSL: V-JEPA 2 outperforms frozen DINOv2/3 and InternVideo2 (which combine image and text pretraining) on motion-centric tasks (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Alrasheed et al., 15 May 2026).

These findings establish that the joint latent prediction approach yields features that are not only highly discriminative for classification but are also resistant to superficial perturbations and occlusions, and encode temporal structure beyond static appearance.

5. Practical Implementation and Domain Adaptation

Best practices for deploying a V-JEPA 2 classifier include:

• Data pipeline: Overlapping multi-view crops per video during probe training; temporal and spatial augmentations mitigate overfitting (Eing et al., 14 Jan 2026, Alrasheed et al., 15 May 2026, Mueller et al., 12 Nov 2025).
• Probe dimension and regularization: The probe width may be reduced for label-scarce settings (e.g., D'=64 with as few as ~0.2M probe head parameters) (Mueller et al., 12 Nov 2025). Shallow, deep-narrow probes avoid overfitting on small datasets.
• Object-centric input: Cropping to bounding boxes of key objects (e.g. faces, animals) as pretraining data boosts subject-relevant feature learning and accuracy by 5–6% (Mueller et al., 12 Nov 2025).
• Frozen backbone regime: Prevents catastrophic forgetting and leverages web-scale self-supervision (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026, Alrasheed et al., 15 May 2026, Mueller et al., 12 Nov 2025).
• Two-stage or continual pretraining: Additional unsupervised pretraining on domain-specific footage can yield a 2–8% boost on mAP/B-Acc in low-shot domains (Mueller et al., 12 Nov 2025).
• Clipping and inference scalability: Uniform 16-frame sampling and small heads enable real-time inference on a single GPU; curation and normalization are critical for data efficiency (Mueller et al., 12 Nov 2025, Alrasheed et al., 15 May 2026).
• Architecture adaptation: Hybrid pooling (temporal convolution or LSTM atop attentive probe), factorized spatial-temporal attention, and multi-scale patching can further extend utility for general video tasks (Eing et al., 14 Jan 2026).

6. Trade-offs, Alternatives, and Comparative Studies

A significant line of research has contrasted V-JEPA 2 with both prior SSL and alternative masked-prediction approaches:

• SALT (Static-teacher Asymmetric Latent Training): A compute-efficient variant decoupling teacher-student from EMA, where a frozen pixel-reconstruction-trained teacher yields highly competitive or superior probing accuracy to V-JEPA 2 under matched compute (SALT ViT-L: 74.9% top-1 SSv2 vs. V-JEPA 2: 73.7%) (Li et al., 29 Sep 2025).
• Tradeoff with fine-tuning: While fully fine-tuned VideoMAE/TimeSformer yield higher clean accuracy, V-JEPA 2’s frozen-probe regime offers larger gains under adversarial corruption and occlusion, and at a fraction of the compute cost (probe-only training ~400 GPU-hours vs. thousands for full fine-tune) (Alrasheed et al., 15 May 2026).
• Architecture reliability: Compared to DINOv3 (pure spatial frame models), V-JEPA 2 delivers consistent performance across dynamic and static actions (variance σ=0.094), at some expense of clustering sharpness (Silhouette 0.206 vs. 0.310), but avoids major drop-offs on motion-intensive categories (Kodathala et al., 25 Sep 2025).

7. Extensions and Applications Beyond Human Video

V-JEPA 2 and its classifier instantiations have been applied to domains beyond standard human video benchmarks:

• Ethological video modeling: Pretrained and probed V-JEPA 2 encoders on the PriVi primate dataset establish SOTA results for chimpanzee, baboon, and lemur behavior classification, with best practices involving domain-centric data curation and two-stage unlabeled learning (Mueller et al., 12 Nov 2025).
• Robotic planning: Latent-prediction pretraining (V-JEPA 2, V-JEPA 2-AC) enables robotic manipulation and navigation with minimal label supervision, using attentive probe-based classifiers or planning heads (Assran et al., 11 Jun 2025, Mur-Labadia et al., 15 Mar 2026).
• Dense, spatially structured prediction: V-JEPA 2.1 enables dense prediction tasks (e.g. depth, object anticipation) by leveraging its dense predictive loss and deep supervision (Mur-Labadia et al., 15 Mar 2026).
• Alternate loss regimes: Dense prediction losses and hierarchical supervision refine spatial grounding and global coherence, demonstrated in V-JEPA 2.1 (Mur-Labadia et al., 15 Mar 2026).

• "Video Joint-Embedding Predictive Architectures for Facial Expression Recognition" (Eing et al., 14 Jan 2026)
• "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning" (Assran et al., 11 Jun 2025)
• "V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning" (Mur-Labadia et al., 15 Mar 2026)
• "Temporal vs. Spatial: Comparing DINOv3 and V-JEPA2 Feature Representations for Video Action Analysis" (Kodathala et al., 25 Sep 2025)
• "Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers" (Li et al., 29 Sep 2025)
• "PriVi: Towards A General-Purpose Video Model For Primate Behavior In The Wild" (Mueller et al., 12 Nov 2025)
• "Latent Video Prediction Learns Better World Models" (Alrasheed et al., 15 May 2026)