--- title: 'JEPA: Joint-Embedded Predictive Architectures' url: https://www.emergentmind.com/topics/joint-embedded-predictive-architectures-jepa type: topic --- # JEPA: Joint-Embedded Predictive Architectures Joint-Embedding Predictive Architectures (JEPAs) are a family of representation-learning models in which a context view and a target view are mapped into a common latent space, a predictor estimates the target embedding from the context embedding, and training minimizes discrepancy in that latent space rather than reconstructing raw pixels, waveforms, coordinates, or tokens. Across recent work, the same template has been interpreted as a self-supervised predictor, an energy-based model, a latent world model, a quasimetric value function, and, in probabilistic extensions, a predictive information state for control; it has also been instantiated in images, video, time series, audio, trajectories, molecular graphs, ultrasound, text-to-image generation, vision-language modeling, and masked diffusion language models [2602.12245, 2602.03604, 2509.25449, 2507.02915, 2406.12913, 2512.10942, 2606.00091]. ## 1. Formal structure and architectural template In the standard formulation, a JEPA contains a context encoder, a target encoder, a predictor, and a comparator in latent space. A common notation is \[ z_x = f_o(x), \qquad z_y = f_a(y), \qquad \hat z_y = p_\epsilon(z_x; c), \] where \(x\) is a context input, \(y\) is a target input, \(c\) is optional conditioning, and \(D(\cdot,\cdot)\) measures discrepancy between the predicted and target embeddings. This induces a scalar compatibility or energy \[ E(x,y) := D\big(p_\epsilon(f_o(x)); f_a(y)\big), \] so that observed context–target pairs are assigned low energy and incompatible pairs higher energy [2602.12245]. This template is deliberately broader than masked-image prediction. In image JEPA variants, the context and target are different spatial regions or masked blocks. In video and time-indexed settings, the context is a past or partially observed segment and the target is a future or masked segment. In planning-oriented variants, the same idea becomes latent dynamics: \[ z_t = \mathcal{E}_\theta(s_t), \qquad \hat z_{t+1} = f_\psi(z_t, a_t), \] or, with an explicit action encoder, \(u = q_\omega(a)\) and a predictor \(g_\phi(z,u)\). In all cases, prediction is performed in representation space rather than observation space [2601.00844, 2602.03604]. A central distinction from pixel-level or autoregressive prediction is that JEPA does not require the model to regenerate the full raw output. The target of learning is a latent representation that can discard nuisance factors and preserve predictive or task-relevant structure. This is the recurring rationale across image, video, time-series, audio, and world-model instantiations [2509.25449, 2507.02915]. ## 2. Objectives, regularization, and collapse avoidance The simplest JEPA loss is a latent regression objective, \[ \mathcal{L}_{\text{pred}}=\mathbb{E}\big[\ell(g_\phi(f_\theta(\text{context})), f_{\bar\theta}(\text{target}))\big], \] often implemented with an EMA target encoder, stop-gradient on the target branch, and either squared error, \(L_1\), cosine, or Smooth \(L_1\). However, predictive consistency alone admits trivial collapse. In action-conditioned settings, for example, a constant encoder and constant latent transition can minimize the latent dynamics loss exactly, which makes collapse an explicit design problem rather than a pathology confined to a few implementations [2509.12249]. Several stabilization strategies have therefore appeared. One line uses variance–covariance regularization, as in VCReg or VICReg, either directly on embeddings or on a projector space. Another uses SIGReg, which regularizes projected embeddings toward an isotropic Gaussian. EB-JEPA packages these mechanisms in a modular energy-based formulation and reports that ablations on Two Rooms planning drop from \(97 \pm 2\%\) success for the full model to \(47\%\) without the variance term, \(46\%\) without the covariance term, \(61\%\) without temporal similarity, and \(1\%\) without the inverse-dynamics loss, illustrating that non-collapse and control utility depend strongly on the regularizer class [2602.03604]. A second line critiques the sufficiency of EMA asymmetry by itself. C-JEPA argues that EMA in I-JEPA is ineffective in preventing entire collapse and that I-JEPA prediction is inadequate for learning the mean of patch representations; it therefore augments JEPA with VICReg-style invariance, variance, and covariance terms [2410.19560]. A third line replaces the online teacher entirely: US-JEPA adopts Static-teacher Asymmetric Latent Training, using a frozen domain-specific ultrasound encoder to provide stable latent targets, thereby decoupling student–teacher optimization and avoiding the hyperparameter-brittle EMA loop in noisy medical imaging [2602.19322]. Recent theory makes the role of auxiliary tasks explicit. For a deterministic MDP, a JEPA encoder \(E_\phi\), latent transition \(T_\psi\), and auxiliary head \(P_\theta\) trained jointly on \[ \mathcal{L}_{\mathrm{dyn}} + c_p \mathcal{L}_p \] satisfy a No Unhealthy Representation Collapse theorem: if both latent-transition consistency and auxiliary regression are driven to zero, then any pair of non-equivalent observations—those that do not share the same transition dynamics or auxiliary label—must map to distinct latent states [2509.12249]. This establishes that auxiliary tasks do not merely improve optimization; they define which distinctions the representation is forced to preserve. Rectified LpJEPA generalizes Gaussian regularization by replacing dense isotropic targets with a sparse, non-negative Rectified Generalized Gaussian prior and matching projected feature distributions to that prior via Rectified Distribution Matching Regularization. In that formulation, the expected \(\ell_0\) norm is analytically controllable, and sparsity becomes an explicit property of the target latent law rather than an indirect consequence of post hoc thresholding [2602.01456]. ## 3. Energy, geometry, and probabilistic reformulations One influential reinterpretation treats JEPA as an energy-based model over state pairs. In that view, the JEPA-induced discrepancy is not merely a training loss but a scalar compatibility energy \(E(x,y)\). A particularly structured class is the intrinsic or least-action energy \[ E(x,y)= \begin{cases} \inf_{\gamma \in \mathcal{T}(x \to y)} \int_0^T L(\gamma(t), \dot\gamma(t))\,dt, & \mathcal{T}(x\to y)\neq \varnothing \\ +\infty, & \mathcal{T}(x\to y)=\varnothing , \end{cases} \] where \(\mathcal{T}(x\to y)\) is a set of admissible trajectories and \(L\) is a non-negative local effort density. Under closure, additivity, and mild regularity assumptions, this intrinsic energy is a quasimetric: it satisfies reflexivity, non-negativity, identity of indiscernibles, and the triangle inequality, but not necessarily symmetry [2602.12245]. The asymmetry is not incidental. If admissibility is directed or local effort is anisotropic, then generally \(E(x,y)\neq E(y,x)\). The same paper proves that a symmetric finite energy cannot encode genuinely one-way reachability: if finite energy is equivalent to reachability and the energy is symmetric, then reachability itself must be symmetric. This makes symmetric latent distances structurally mismatched to irreversible dynamics, directional graphs, or goal-reaching control under asymmetric constraints [2602.12245]. A parallel reformulation makes JEPA explicitly probabilistic. VJEPA replaces deterministic latent regression with a predictive distribution \(p_\phi(Z_T\mid Z_C,\xi_T)\) and an inference model \(q_{\theta'}(Z_T\mid x_T)\), trained with a variational latent objective, \[ \mathcal{L}_{\text{VJEPA}} = \mathbb{E}_{Z_T \sim q_{\theta'}(\cdot \mid x_T)} \big[-\log p_\phi(Z_T \mid Z_C,\xi_T)\big] + \beta\,\mathrm{KL}\big(q_{\theta'}(Z_T \mid x_T)\,\|\,p(Z_T)\big). \] In this formulation, deterministic JEPA appears as the fixed-variance Gaussian special case of latent maximum likelihood. The paper further connects VJEPA to predictive state representations and Bayesian filtering, argues that sequential modeling does not require autoregressive observation likelihoods, and proves that under stated realizability conditions the learned latent can act as a sufficient information state for optimal control without pixel reconstruction [2601.14354]. These geometric and probabilistic views are complementary rather than exclusive. The energy-based perspective emphasizes directed compatibility and compositionality; the variational perspective emphasizes uncertainty, belief propagation, and latent predictive sufficiency. Together they broaden JEPA from a training recipe into a representation-theoretic framework. ## 4. JEPA as a world model and planning substrate In model-based control, JEPA is typically instantiated as an action-conditioned latent dynamics model. The canonical setup uses a state encoder \(\mathcal{E}_\theta\), an action encoder or direct action concatenation, and a predictor rolled out recursively over a planning horizon. A standard planning cost in earlier JEPA world models is the latent distance between predicted future states and a goal embedding, \[ J(\mathbf{a})=\sum_{k=1}^{H}\|\hat z_{t+k}(\mathbf{a})-z_g\|_2^2, \] but purely predictive JEPA representations are often not aligned with control difficulty or reachability [2601.00844]. Value-guided JEPA addresses this by shaping the latent geometry so that a distance or quasi-distance between embeddings approximates the negative goal-conditioned value for a reaching cost: \[ V_\theta(s,g) = -d_\theta(s,g). \] The encoder is trained with an IQL-style Bellman consistency loss, and planning then combines the learned JEPA dynamics with MPC and MPPI. On Wall and Maze benchmarks, the best-performing variant is the separate-training quasidistance model VF\_quasi, with success rates \(0.71\) on WS, \(0.96\) on WB, and \(0.63\) on Maze; joint training with prediction or VCReg generally degrades planning performance relative to this value-shaped geometry [2601.00844]. The control-theoretic connection becomes sharper under the intrinsic-energy interpretation. Optimal goal-reaching cost-to-go functions have exactly the least-action form above, so any JEPA trained to approximate such intrinsic energies lies in the same quasimetric value-function class targeted by quasimetric reinforcement learning. This is the basis of the IE-JEPA notion: a JEPA whose induced energy is intrinsic is, by construction, a quasimetric cost-to-go representation [2602.12245]. Practical world-model implementations follow the same pattern. EB-JEPA includes an action-conditioned Video-JEPA with multistep rollout losses, temporal similarity, and inverse-dynamics regularization, and reports a \(97\%\) planning success rate on the Two Rooms navigation task. VJEPA and BJEPA go further by factorizing predictive belief into a learned dynamics expert and a modular prior expert combined as a Product of Experts, thereby enabling zero-shot insertion of goals or constraints into latent planning [2602.03604, 2601.14354]. The practical implication is that JEPA becomes most effective for control when its latent geometry is not merely predictive but also value-aware, asymmetric where necessary, and explicitly regularized against collapse. ## 5. Modality-specific instantiations Recent work has turned JEPA from a vision-centric template into a broadly reusable design pattern. | Domain | Representative variant | Distinctive JEPA mechanism | |---|---|---| | Time series | TS-JEPA [2509.25449] | Temporal patch masking with Transformer encoder, predictor, and EMA target encoder | | Motion and content | MC-JEPA [2307.12698] | Shared encoder for optical flow prediction and VICReg-style content learning | | Trajectories | T-JEPA [2406.12913] | Representation-space target resampling from GPS or check-in trajectories | | Audio | Audio-JEPA [2507.02915] | Masked spectrogram-patch prediction with ViT context and EMA target encoders | | Polymer graphs | Graph JEPA [2506.18194] | Context and target subgraph embeddings predicted with positional conditioning | | Ultrasound | US-JEPA [2602.19322] | SALT objective with frozen ultrasound teacher and region-conditioned masking | TS-JEPA adapts JEPA to univariate time series by dividing sequences into non-overlapping temporal patches, masking \(70\%\) of them during pretraining, encoding visible patches with a Transformer, predicting masked latent codes, and using an EMA target encoder with \(m=0.998\). It is reported to match or surpass current state-of-the-art baselines on standard datasets while maintaining a strong balance across classification and forecasting, with better long-term rollout behavior than autoregressive baselines on two of three forecasting datasets [2509.25449]. MC-JEPA unifies motion and content learning inside a shared ConvNeXt-T encoder by combining a motion-prediction branch for optical flow with a VICReg content branch. The model achieves performance on-par with existing unsupervised optical flow benchmarks and with common self-supervised approaches on downstream semantic segmentation, while showing that motion and content objectives can improve one another within a single JEPA-style encoder [2307.12698]. T-JEPA applies the paradigm to GPS and check-in trajectories by predicting latent subsets of a trajectory from partially observed context trajectories, using AdjFuse for local spatial aggregation and Transformer encoders on both target and context branches. The method is reported to outperform or match strong contrastive baselines on three urban trajectory datasets and two Foursquare datasets, particularly under distortion and irregular sampling [2406.12913]. Audio-JEPA is a direct translation of I-JEPA to mel-spectrograms. It uses a ViT-Base backbone, random masking of \(40\%\) to \(60\%\) of patches, an EMA target encoder, and an \(L_2\) latent prediction objective on masked spectrogram regions. On X-ARES, it shows comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data, with especially strong kNN results on ESC-50, FMA small, and GTZAN [2507.02915]. Graph JEPA for polymers defines context and target subgraphs, encodes them with weighted D-MPNN backbones, and predicts target subgraph embeddings from a context embedding plus random-walk structural encodings. The paper reports that JEPA-based self-supervised pretraining improves downstream performance across all tested polymer datasets when labels are scarce, with particularly strong gains in the smallest labeled-data regimes [2506.18194]. US-JEPA shows how JEPA can be specialized for a harsh imaging modality. By pairing a randomly initialized ViT-B/16 student with a frozen URFM teacher under SALT and restricting masks to valid anatomical regions via Ultrasound Region Conditioning, it achieves performance competitive with or superior to domain-specific and universal vision foundation model baselines on UltraBench, while exhibiting stronger robustness to blur and speckle than reconstruction-style ultrasound models [2602.19322]. ## 6. Multimodal, generative, and language-model extensions JEPA has recently moved beyond pure representation learning into multimodal prediction and conditional generation. JEPA-T is a text-to-image generator in which images are tokenized by a VAE, text is encoded by a frozen CLIP ViT-B/16 text encoder, and a joint-embedding predictive Transformer predicts masked or noisy visual token embeddings. The model combines a JEPA masked-prediction loss with conditional flow matching, \[ \mathcal{L}_{\text{total}}=\mathcal{L}_{\text{FM}}+\lambda_{\text{jepa}}\mathcal{L}_{\text{JEPA}}, \] and uses late cross-attention after the predictor plus objective-level text injection. On ImageNet-1K at \(256\times256\), the reported full model reaches FID \(1.42\), IS \(298.3\), Precision \(0.79\), and Recall \(0.63\), outperforming non-fusion and late-fusion ablations [2510.00974]. VL-JEPA replaces token autoregression with prediction of continuous target-text embeddings. With a V-JEPA 2 vision encoder, a predictor initialized from the last 8 layers of Llama-3.2-1B, a Y-Encoder initialized from EmbeddingGemma-300M, and an optional lightweight text decoder used only when needed, the model trains in embedding space via InfoNCE rather than token cross-entropy. In a controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having \(50\%\) fewer trainable parameters. It also supports selective decoding, reducing decoding operations by \(2.85\times\) at similar performance, and its embedding space directly supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without architecture modification [2512.10942]. DLLM-JEPA adapts the idea to masked diffusion language models. Two semantically distinct views of the same sequence are produced by different masking rates \(t_L