---
title: 'V-JEPA 2.1: Advancing Visual Self-Supervision'
url: https://www.emergentmind.com/topics/v-jepa-2-1
type: topic
---

# V-JEPA 2.1: Advancing Visual Self-Supervision

V-JEPA 2.1 is a family of models built on Joint-Embedding Predictive Architectures (JEPA) that advance visual self-supervised learning across both dense and goal-conditioned planning tasks. In the visual domain, V-JEPA 2.1 defines a paradigm for extracting high-quality, spatially structured visual representations from images and videos while preserving robust global scene understanding. In reinforcement learning and planning, its value-guided variant introduces a quasi-metric goal-conditioned value function into JEPA world models, facilitating effective model-based control. The architecture systematically integrates dense predictive losses, deep self-supervision, multi-modal tokenization, and effective scaling strategies. Across both domains, V-JEPA 2.1 establishes state-of-the-art performance on benchmarks spanning visual semantics, depth, action recognition, anticipation, and robotic control [2601.00844, 2603.14482].

## 1. Model Architecture and Principles

V-JEPA 2.1 architectures are defined in two principal contexts:
- **Dense visual representation learning** utilizes Vision Transformers (ViT) with masked-prediction jointly embedding images and videos.
- **Goal-conditioned planning** utilizes a convolutional or residual encoder for low-dimensional states or images, a latent dynamical predictor, and a value function shaped by a goal-conditioned quasi-metric.

### Dense Visual Models

- **Tokenization**: Separate 2D and 3D convolutional patch embeddings produce tokens for images and videos, respectively. Learned modality embeddings are prepended, and rotary positional encodings (RoPE) are applied accordingly.
- **Masking**: Spatio-temporal masking creates a corrupted (“student”) view $x$ and a clean “teacher” view $y$.
- **Encoding/Prediction**: The student encoder $E_\theta$ yields per-patch tokens; a predictor $P_\phi$ fuses visible (context) and mask tokens to predict teacher targets.
- **Multi-level Outputs**: Extracted from various intermediate transformer blocks and concatenated for downstream prediction, enabling deep self-supervision.

### Goal-Conditioned World Models

- **State encoder $E_\theta$**: Maps $s\in S_0$ to embedding $e_\theta(s)\in\mathbb{R}^d$.
- **Action encoder $A$**: Small MLP or identity function mapping $a_t$ to latent $\bar{a}_t$.
- **Predictor $P_\phi$**: Predicts future embedding $\hat{z}_{t+1|t}=P_\phi(e_\theta(s_t),\bar{a}_t)$.
- **Goal-conditioned value**: $V_\psi(s,g)\approx-d_\psi(e_\theta(s),e_\theta(g))$ with $d_\psi$ defined via either Euclidean or learnable quasi-metric form.

## 2. Loss Functions and Training Methodologies

### Dense Predictive Loss

The dense loss encourages learning of both local (per-patch) and global features. For visual models, the loss at layer $\ell$ is:

$$
L^\ell_{\text{dense}} = \frac{1}{|M|}\sum_{(i,t)\in M}\|\hat{z}^\ell_{i,t} - z^\ell_{i,t}\|_1 + \frac{1}{|C|}\sum_{(j,s)\in C} \lambda_{j,s}\|\hat{z}^\ell_{j,s} - z^\ell_{j,s}\|_1
$$

where $\lambda_{j,s}$ is a weighting dependent on proximity to masked regions, with the final recipe:

$$
\lambda_{j,s} = \lambda_0 / \sqrt{d_\min((j,s),M)}
$$

Deep self-supervision sums such losses over multiple encoder depths $\ell$.

### Value-Shaping and Predictive Objectives

The goal-conditioned planning variant incorporates:

- **JEPA prediction loss**:
  $$
  L_{\text{pred}}(\theta, \phi) = \mathbb{E}_t \|P_\phi(e_\theta(s_t),\bar{a}_t) - e_\theta(s_{t+1})\|_2^2 + \lambda_{\text{VC}} L_{\text{VCReg}}(\{e_\theta(s)\})
  $$
  where $L_{\text{VCReg}}$ (e.g., VICReg) regularizes the embedding to prevent collapse.

- **Goal-conditioned value loss (IQL-style)**:
  $$
  L_{\text{VF}}(\theta, \psi) = \mathbb{E}_{(s_t,s_{t+1},g)} L_\tau^2(r_t(s_t,g) + \gamma \bar{V}_\psi(s_{t+1},g) - V_\psi(s_t,g))
  $$
  with expectile regression, where $r_t(s_t,g) = -\mathbf{1}_{s_t\neq g}$, $\gamma$ is a discount factor, and $\tau$ is the expectile level.

**Training schedules** include both joint minimization and a “separate” regime where the encoder is value-shaped, then frozen.

## 3. Value-Guided Goal Conditioned Planning (Quasi-Distance)

The hallmark feature of V-JEPA 2.1 in world modeling is the use of a learnable quasi-metric to approximate the negative value function for goal reaching. The quasi-metric network $Q_\psi:\mathbb{R}^d\rightarrow\mathbb{R}^k$ yields:

$$
d_\psi(u, v) = \|Q_\psi(u) - Q_\psi(v)\|_2
$$

The approach enables encoding of non-symmetric and subspace-constrained value relations. Empirically, this adjustment substantially improves planning success rates over both pure prediction-based and Euclidean value shaped models.

## 4. Key Empirical Results and Ablations

V-JEPA 2.1 outperforms prior models across dense and global visual tasks, control, and planning:

| Task            | Metric                | V-JEPA 2.0 | V-JEPA 2.1 ViT-g | V-JEPA 2.1 ViT-G |
|-----------------|----------------------|------------|------------------|------------------|
| Ego4D STA       | mAP\_All (↑)         | 6.02       | 6.75             | 7.71             |
| EK100 Anticip.  | Recall@5 (↑)         | 39.7       | 38.4             | 40.8             |
| SSv2 Action Rec.| Top-1 (↑)            | 77.3       | −                | 77.7             |
| NYU Depth       | RMSE (↓)             | 0.642      | 0.350            | 0.307            |
| Wall (WS; VF_q) | Success Rate (↑)     | 0.55       | −                | 0.71             |
| Maze (VF_q)     | Success Rate (↑)     | 0.54       | −                | 0.63             |

Dense representation ablations reveal:
- Weighted context loss and deep self-supervision are essential for jointly optimizing local and global visual tasks.
- Multi-modal tokenization and dataset scaling further improve quantitative performance.
- For planning, a quasi-metric value function (VF_quasi) delivers the highest success rates, especially notable in maze and wall navigation settings [2601.00844, 2603.14482].

## 5. Planning and Control via Latent Model Predictive Control (MPC)

V-JEPA 2.1 deploys latent MPC using Model Predictive Path Integral (MPPI) control. The algorithm samples trajectories by rolling out action sequences in latent space, scores them using the learned embedding-space quasi-distance to the goal, and updates the action sequence via importance weighting:

1. Initialize mean action sequence $\mu$.
2. Sample $M$ candidate sequences via $\epsilon \sim \mathcal{N}(0, \Sigma)$.
3. Predict latent rollouts using $P_\phi$.
4. Compute cost as $d_\psi(z, z_{\text{goal}})$ at each rollout.
5. Update $\mu$ by weighted averaging.
6. Execute the first action, shift sequence, and repeat.

Standard hyperparameters include $M=2000$, $\Sigma=12I$, $\lambda_{\text{temp}}=0.005$, and horizon $H=96$ (varies by environment).

## 6. Scaling Strategies and Multi-Modal Learning

- **Model**: ViT-L (300M), ViT-g (1B), ViT-G (2B) parameter scales.
- **Data**: VisionMix163M (142M images, 19M video clips with rebalanced sampling).
- **Training**: Image and video batches are processed jointly using the multi-modal tokenizer; mixed resolution and extended training phases improve performance on depth and semantics.
- **Tokenization**: Efficient modality-specific convolutions with unified positional encoding allow seamless transfer and training across images and videos.

Empirical effects of scaling indicate monotonic improvements in benchmarks such as semantic segmentation (ADE20K mIoU +25 points) and depth estimation (NYUv2 RMSE halved), with global classification preserved.

## 7. Significance, Limitations, and Future Directions

V-JEPA 2.1 establishes a unified, scalable framework for dense visual representation learning and value-informed model-based planning. Key empirical findings include:

- Quasi-metric value shaping and deep self-supervision are essential for simultaneously achieving global scene understanding and robust per-patch semantics.
- Embedding distance-matching, particularly with learnable metrics, directly translates to improved planning and control in high-dimensional spaces.
- Jointly optimizing prediction and value objectives may degrade value geometry; separate training regimes are typically superior for planning-centric applications.
- Scaling data, model size, and resolution yields consistent, incremental benefits across modalities and tasks.

Remaining limitations include possible trade-offs between dense token learning and global representation, and the interaction of loss-weighting schedules with convergence and stability.

These methods provide plug-in extensions for JEPA-style models that can be adopted in both vision and reinforcement learning domains, supporting reproduction and extension in new domains with minimal modification [2601.00844, 2603.14482].

Source: https://www.emergentmind.com/topics/v-jepa-2-1