Joint-Embedding Predictive Architectures

Updated 12 November 2025

JEPAs are self-supervised frameworks that learn predictive representations by regressing latent embeddings from correlated input views, bypassing traditional reconstruction.
They use lightweight predictors and EMA-stabilized encoders to prevent collapse, achieving competitive performance across vision, audio, language, and graph tasks.
Empirical studies and theoretical insights show that JEPAs prioritize high-influence features, laying the groundwork for advances in generative modeling and multimodal learning.

Joint-Embedding Predictive Architectures (JEPAs) are a class of self-supervised learning frameworks in which the goal is to learn representations by predicting latent embeddings of one “view” or portion of an input signal from another, typically related, view. As a family, JEPAs are characterized by their reliance on embedding-space prediction—contrasting with input-space reconstruction (autoencoders, masked modeling) or contrastive SSL—and their avoidance of explicit negative pairs. This paradigm has influenced design across vision, audio, language, graph, and multimodal representation learning, yielding a unifying framework with distinct inductive biases, failure modes, and theoretical underpinnings.

1. Formal Definition and Theoretical Foundations

Let $x$ denote an input sample, and let $c, t$ denote two correlated “views” (subsets, crops, or modality slices) sampled from $x$ . JEPAs employ a parameterized encoder $f_\theta$ to embed these views as $z_c = f_\theta(c)$ and $z_t = f_\theta(t)$ . A lightweight predictor $g_\phi$ (often an MLP or shallow transformer) transforms $z_c$ into a prediction $\hat z_t = g_\phi(z_c)$ . The central objective is a regression loss in latent space, typically

$\mathcal{L}_{\mathrm{JEPA}} = \mathbb{E}_{x}\bigl[ \ell(g_\phi(z_c),\ \mathrm{stopgrad}(z_t)) \bigr],$

where $\ell(\cdot, \cdot)$ is a distance function (e.g., $\ell_2$ or Huber). To stabilize training and avoid collapsed (trivial, constant) solutions, it is common to use an exponential moving average (EMA) copy of $f_\theta$ for $z_t$ as in BYOL and I-JEPA, or incorporate additional regularization (e.g., variance/covariance constraints).

A key insight is that, unlike reconstruction objectives, the JEPA loss inherently biases the model toward features that are predictive across $c$ and $t$ —abstract, high-influence attributes—while disregarding low-level noise or non-predictive details (Littwin et al., 3 Jul 2024). Theoretical results show that, in deep linear networks, JEPAs converge to representations dominated by directions with high regression coefficients, i.e., those most useful for target prediction, and thus suppress spurious or noisy features.

2. Architectural Variants and Modality-Specific Instantiations

JEPAs have been instantiated in a diverse range of domains:

Vision: The canonical I-JEPA adopts a dual-branch Vision Transformer (ViT) encoder architecture with separate context and target branches (the latter maintained via EMA), using mask-based sampling to define context and target windows in image patches (Littwin et al., 14 Oct 2024). The predictor is typically a shallow transformer or MLP.
Audio: Audio-JEPA translates the I-JEPA architecture to mel-spectrogram patches, demonstrating that random unstructured patch masking (as opposed to block-masking in vision) provides superior generalization, reflecting differences in the structure of audio compared to images (Tuncay et al., 25 Jun 2025, Riou et al., 14 May 2024).
Time series: TS-JEPA divides time series into patch tokens, uses high masking ratios (>70%), and employs lightweight transformers as encoder/predictor, outperforming masked input-reconstruction and contrastive baselines in both classification and long-term forecasting tasks (Ennadir et al., 29 Sep 2025).
Graphs: Graph-JEPA partitions input graphs into node clusters (patches), uses GNN-based subgraph encoders, and targets either Euclidean or hyperbolic projections of target patches. Variants predict either the latent subgraph code directly or its mapped coordinates on a hyperbola to capture hierarchies (Skenderi et al., 2023).
Trajectories: T-JEPA enriches trajectory point embeddings with adjacency information, then samples/predicts masked portions of the path embedding via transformers, outperforming contrastive learning on similarity and ranking benchmarks with improved robustness (Li et al., 13 Jun 2024).
Language (LLM-JEPA): Applied to paired textual views (e.g., NL/code), JEPAs use tied-weight transformer encoders, an embedding-space predictor, and an additional loss term atop the standard generative objective, yielding improved generalization across diverse benchmarks (Huang et al., 11 Sep 2025).
Multimodal (TI-JEPA, Mask-JEPA): TI-JEPA combines frozen ViT and text transformers with cross-attention bridges and a joint-predictive ViT for masked text/image fusion, aligning modalities in a shared energy-based latent space (Vo et al., 9 Mar 2025). Mask-JEPA leverages a transformer predictor atop pixel-level encoders to pretrain mask-classification segmentation models (Kim et al., 15 Jul 2024).

3. Loss Functions, Regularization, and Collapse Prevention

Collapse—degenerate solutions with constant embeddings—is endemic in non-contrastive JEPAs. Architectural remedies include:

Use of a momentum/EMA target encoder, ensuring one branch remains a moving average of the other and gradients are stopped;
Lightweight, asymmetric predictors, preventing identity mappings;
Explicit regularization: variance (per-dimension standard deviation above a threshold), covariance (decorrelation), and invariance (alignment across augmentations) constraints as in VICReg and C-JEPA (Mo et al., 25 Oct 2024);
Sketched Isotropic Gaussian Regularization (SIGReg): matches the full embedding distribution to an isotropic Gaussian by enforcing univariate moments across random 1D projections (Balestriero et al., 11 Nov 2025). This regularization minimizes risk for both linear and k-NN probes, is scalable, and eliminates heuristic loss schedules.

In some variants, additional auxiliary tasks anchor representations to application-relevant semantics (rewards, Q-values, etc.), provably preventing "unhealthy representation collapse" by ensuring only task-irrelevant distinctions are collapsed in the latent space (Yu et al., 12 Sep 2025).

4. Masking, Prediction, and Sampling Strategies

The choice of context/target splits and the definition of masking domains are nontrivial and highly modality-dependent:

Vision: Structured block masking (as in I-JEPA) is optimal for vision, as blocks correspond to contiguous, semantically meaningful image regions.
Audio/Time-Series: Unstructured random masking preserves spectro-temporal diversity, crucial for audio tasks; block masking degrades performance by destroying spectral/temporal coherence (Tuncay et al., 25 Jun 2025, Riou et al., 14 May 2024, Ennadir et al., 29 Sep 2025).
Graphs/Trajectories: Partitioning into node or point clusters via graph partitioning or random walks, followed by masking at the cluster level, ensures the model reconstructs high-level, non-local graph or path features.
Multimodal: Masking may be performed either in each modality or jointly (e.g., masking image regions conditioned on textual queries), enabling cross-modal alignment in latent space.

Prediction is always formulated as regression in latent space, with loss functions such as mean squared error, Huber, or smooth L1 applied patch-wise, token-wise, or to cluster/graph-level summaries. Additional heads or regularizers may implement auxiliary prediction losses or domain constraints.

5. Empirical Results and Practical Guidelines

JEPAs achieve state-of-the-art or highly competitive results across domains, underlining their data and computational efficiency:

Audio: Audio-JEPA reaches or surpasses wav2vec 2.0 and data2vec results using only ~1/5 of the data, notably on music and environmental sound benchmarks, while remaining competitive (though trailing) on fine-grained speech tasks (Tuncay et al., 25 Jun 2025).
Vision: EC-IJEPA improves on I-JEPA via spatial conditioning, raising ImageNet-1k linear-probe accuracy from 74.8% to 76.7% (ViT-L/16); C-JEPA stabilizes learned embeddings and accelerates convergence (Littwin et al., 14 Oct 2024, Mo et al., 25 Oct 2024).
Graphs: Graph-JEPA achieves SOTA on five of seven standard classification tasks, is highly data-efficient, and nearly perfectly classifies non-isomorphic graphs (Skenderi et al., 2023).
Reinforcement Learning: JEPA encoders for RL, when combined with collapse-prevention and action conditioning, support rapid learning in classical environments, outperforming uninformed encoder baselines (Kenneweg et al., 23 Apr 2025).
Time-Series: TS-JEPA is robust to confounding and outperforms input-space and contrastive baselines, achieving SOTA or near-SOTA on most UCR-style classification and forecasting (Ennadir et al., 29 Sep 2025).
Interpretability and Transfer: Adding group-sparse penalties (SparseJEPA) or interpretability constraints improves transfer accuracy by 3–5 points and enables more efficient and semantically meaningful feature discovery (Hartman et al., 22 Apr 2025).

Empirically optimal JEPA design choices include: unstructured audio-domain masking for audio; segment lengths matched to the dominant time-scale of target tasks; stronger (wider/deeper) context encoders with lightweight predictors; and explicit regularization via embedding distribution matching (Riou et al., 14 May 2024, Balestriero et al., 11 Nov 2025). Over-reliance on image-domain masking heuristics or masking in latent space can be detrimental, especially for sequential or spectral modalities.

6. Theoretical Insights, Biases, and Limitations

Analyses using deep linear models reveal an implicit bias of JEPAs toward learning high-influence directions—those with large regression coefficients for predicting $y$ from $x$ —and away from high-variance but non-predictive (noisy) directions. The effect becomes more pronounced with network depth, yielding low-rank, semantically robust embeddings (Littwin et al., 3 Jul 2024). However, JEPAs may fail in the presence of fixed slow features (e.g., static distractors), which may dominate the latent space if they are more predictable than relevant signals (Sobal et al., 2022).

Auxiliary tasks can be strategically introduced to anchor the learned equivalence relations: the auxiliary function $p$ should encode precisely the distinctions to be preserved; too weak of an auxiliary induces coarse clumping; too rich or random an auxiliary enforces over-granular, uncompressed embeddings (Yu et al., 12 Sep 2025).

7. Extensions, Generalization, and Future Directions

JEPAs are being continually generalized and extended:

Generative Modeling: D-JEPA integrates latent prediction with diffusion and flow-matching losses for continuous-data generation across images, video, and audio, outperforming prior generative models at scale (Chen et al., 2 Oct 2024).
Distribution Matching: LeJEPA introduces SIGReg, demonstrating that constraining embeddings to an isotropic Gaussian (via linear-cost, random-projection–based regularization) yields optimal downstream performance for both linear and kernel probes, and offers a practical, scalable, heuristics-free training recipe (Balestriero et al., 11 Nov 2025).
Multimodal Learning: Flexible cross-modal architectures (e.g., TI-JEPA, LLM-JEPA) fuse text, image, and even trajectory information, matching or surpassing specialized multimodal and language-modeling objectives (Vo et al., 9 Mar 2025, Huang et al., 11 Sep 2025).
Dense Prediction: Mask-JEPA extends representation learning to segmentation by coupling a shared pixel decoder and transformer-based predictor, raising mask-classification performance in both low- and full-data regimes (Kim et al., 15 Jul 2024).
Time-Series Foundations: Scaling TS-JEPA to multi-modal, multivariate data (sensor, weather, etc.) and combining JEPA with contrastive losses represents an ongoing direction (Ennadir et al., 29 Sep 2025).

Future work will focus on refining masking strategies (adaptive, semantically informed), integrating stronger invariance constraints, achieving better bias–variance trade-offs in embedding distributions, and extending JEPA pretraining to arbitrarily large, heterogeneous, or multi-modal data collections. Empirical and theoretical questions remain regarding the optimal balancing of predictive, contrastive, and statistical regularization in diverse domains.