V-JEPA 2.1: Advancing Visual Self-Supervision

• The paper introduces V-JEPA 2.1, which integrates dense visual representation learning with a quasi-metric goal-conditioned planning approach for superior performance.
• It employs techniques like masked-prediction, multi-modal tokenization, and deep self-supervision to extract high-quality spatial features and robust global scene understanding.
• Empirical evaluations demonstrate state-of-the-art improvements in benchmarks such as action recognition, depth estimation, and robotic control over previous JEPA versions.

V-JEPA 2.1 is a family of models built on Joint-Embedding Predictive Architectures (JEPA) that advance visual self-supervised learning across both dense and goal-conditioned planning tasks. In the visual domain, V-JEPA 2.1 defines a paradigm for extracting high-quality, spatially structured visual representations from images and videos while preserving robust global scene understanding. In reinforcement learning and planning, its value-guided variant introduces a quasi-metric goal-conditioned value function into JEPA world models, facilitating effective model-based control. The architecture systematically integrates dense predictive losses, deep self-supervision, multi-modal tokenization, and effective scaling strategies. Across both domains, V-JEPA 2.1 establishes state-of-the-art performance on benchmarks spanning visual semantics, depth, action recognition, anticipation, and robotic control (Destrade et al., 28 Dec 2025, Mur-Labadia et al., 15 Mar 2026).

1. Model Architecture and Principles

V-JEPA 2.1 architectures are defined in two principal contexts:

• Dense visual representation learning utilizes Vision Transformers (ViT) with masked-prediction jointly embedding images and videos.
• Goal-conditioned planning utilizes a convolutional or residual encoder for low-dimensional states or images, a latent dynamical predictor, and a value function shaped by a goal-conditioned quasi-metric.

Dense Visual Models

• Tokenization: Separate 2D and 3D convolutional patch embeddings produce tokens for images and videos, respectively. Learned modality embeddings are prepended, and rotary positional encodings (RoPE) are applied accordingly.
• Masking: Spatio-temporal masking creates a corrupted (“student”) view $x$ and a clean “teacher” view $y$.
• Encoding/Prediction: The student encoder $E_\theta$ yields per-patch tokens; a predictor $P_\phi$ fuses visible (context) and mask tokens to predict teacher targets.
• Multi-level Outputs: Extracted from various intermediate transformer blocks and concatenated for downstream prediction, enabling deep self-supervision.

Goal-Conditioned World Models

• State encoder $E_\theta$: Maps $s\in S_0$ to embedding $e_\theta(s)\in\mathbb{R}^d$.
• Action encoder $A$: Small MLP or identity function mapping $a_t$ to latent $\bar{a}_t$.
• Predictor $P_\phi$: Predicts future embedding $\hat{z}_{t+1|t}=P_\phi(e_\theta(s_t),\bar{a}_t)$.
• Goal-conditioned value: $V_\psi(s,g)\approx-d_\psi(e_\theta(s),e_\theta(g))$ with $d_\psi$ defined via either Euclidean or learnable quasi-metric form.

2. Loss Functions and Training Methodologies

Dense Predictive Loss

The dense loss encourages learning of both local (per-patch) and global features. For visual models, the loss at layer $\ell$ is:

$$L^\ell_{\text{dense}} = \frac{1}{|M|}\sum_{(i,t)\in M}\|\hat{z}^\ell_{i,t} - z^\ell_{i,t}\|_1 + \frac{1}{|C|}\sum_{(j,s)\in C} \lambda_{j,s}\|\hat{z}^\ell_{j,s} - z^\ell_{j,s}\|_1$$

where $\lambda_{j,s}$ is a weighting dependent on proximity to masked regions, with the final recipe:

$\lambda_{j,s} = \lambda_0 / \sqrt{d_\min((j,s),M)}$

Deep self-supervision sums such losses over multiple encoder depths $\ell$.

Value-Shaping and Predictive Objectives

The goal-conditioned planning variant incorporates:

• JEPA prediction loss:

$$L_{\text{pred}}(\theta, \phi) = \mathbb{E}_t \|P_\phi(e_\theta(s_t),\bar{a}_t) - e_\theta(s_{t+1})\|_2^2 + \lambda_{\text{VC}} L_{\text{VCReg}}(\{e_\theta(s)\})$$

where $L_{\text{VCReg}}$ (e.g., VICReg) regularizes the embedding to prevent collapse.

• Goal-conditioned value loss (IQL-style):

$$L_{\text{VF}}(\theta, \psi) = \mathbb{E}_{(s_t,s_{t+1},g)} L_\tau^2(r_t(s_t,g) + \gamma \bar{V}_\psi(s_{t+1},g) - V_\psi(s_t,g))$$

with expectile regression, where $r_t(s_t,g) = -\mathbf{1}_{s_t\neq g}$, $\gamma$ is a discount factor, and $\tau$ is the expectile level.

Training schedules include both joint minimization and a “separate” regime where the encoder is value-shaped, then frozen.

3. Value-Guided Goal Conditioned Planning (Quasi-Distance)

The hallmark feature of V-JEPA 2.1 in world modeling is the use of a learnable quasi-metric to approximate the negative value function for goal reaching. The quasi-metric network $Q_\psi:\mathbb{R}^d\rightarrow\mathbb{R}^k$ yields:

$d_\psi(u, v) = \|Q_\psi(u) - Q_\psi(v)\|_2$

The approach enables encoding of non-symmetric and subspace-constrained value relations. Empirically, this adjustment substantially improves planning success rates over both pure prediction-based and Euclidean value shaped models.

4. Key Empirical Results and Ablations

V-JEPA 2.1 outperforms prior models across dense and global visual tasks, control, and planning:

Task	Metric	V-JEPA 2.0	V-JEPA 2.1 ViT-g	V-JEPA 2.1 ViT-G
Ego4D STA	mAP_All (↑)	6.02	6.75	7.71
EK100 Anticip.	Recall@5 (↑)	39.7	38.4	40.8
SSv2 Action Rec.	Top-1 (↑)	77.3	−	77.7
NYU Depth	RMSE (↓)	0.642	0.350	0.307
Wall (WS; VF_q)	Success Rate (↑)	0.55	−	0.71
Maze (VF_q)	Success Rate (↑)	0.54	−	0.63

Dense representation ablations reveal:

• Weighted context loss and deep self-supervision are essential for jointly optimizing local and global visual tasks.
• Multi-modal tokenization and dataset scaling further improve quantitative performance.
• For planning, a quasi-metric value function (VF_quasi) delivers the highest success rates, especially notable in maze and wall navigation settings (Destrade et al., 28 Dec 2025, Mur-Labadia et al., 15 Mar 2026).

5. Planning and Control via Latent Model Predictive Control (MPC)

V-JEPA 2.1 deploys latent MPC using Model Predictive Path Integral (MPPI) control. The algorithm samples trajectories by rolling out action sequences in latent space, scores them using the learned embedding-space quasi-distance to the goal, and updates the action sequence via importance weighting:

1. Initialize mean action sequence $\mu$.
2. Sample $M$ candidate sequences via $\epsilon \sim \mathcal{N}(0, \Sigma)$.
3. Predict latent rollouts using $P_\phi$.
4. Compute cost as $d_\psi(z, z_{\text{goal}})$ at each rollout.
5. Update $\mu$ by weighted averaging.
6. Execute the first action, shift sequence, and repeat.

Standard hyperparameters include $M=2000$, $\Sigma=12I$, $\lambda_{\text{temp}}=0.005$, and horizon $H=96$ (varies by environment).

6. Scaling Strategies and Multi-Modal Learning

• Model: ViT-L (300M), ViT-g (1B), ViT-G (2B) parameter scales.
• Data: VisionMix163M (142M images, 19M video clips with rebalanced sampling).
• Training: Image and video batches are processed jointly using the multi-modal tokenizer; mixed resolution and extended training phases improve performance on depth and semantics.
• Tokenization: Efficient modality-specific convolutions with unified positional encoding allow seamless transfer and training across images and videos.

Empirical effects of scaling indicate monotonic improvements in benchmarks such as semantic segmentation (ADE20K mIoU +25 points) and depth estimation (NYUv2 RMSE halved), with global classification preserved.

7. Significance, Limitations, and Future Directions

V-JEPA 2.1 establishes a unified, scalable framework for dense visual representation learning and value-informed model-based planning. Key empirical findings include:

• Quasi-metric value shaping and deep self-supervision are essential for simultaneously achieving global scene understanding and robust per-patch semantics.
• Embedding distance-matching, particularly with learnable metrics, directly translates to improved planning and control in high-dimensional spaces.
• Jointly optimizing prediction and value objectives may degrade value geometry; separate training regimes are typically superior for planning-centric applications.
• Scaling data, model size, and resolution yields consistent, incremental benefits across modalities and tasks.

Remaining limitations include possible trade-offs between dense token learning and global representation, and the interaction of loss-weighting schedules with convergence and stability.

These methods provide plug-in extensions for JEPA-style models that can be adopted in both vision and reinforcement learning domains, supporting reproduction and extension in new domains with minimal modification (Destrade et al., 28 Dec 2025, Mur-Labadia et al., 15 Mar 2026).