JEPA-Style Latent Dynamics Model

• JEPA-style latent dynamics models jointly optimize context and target encoders with a predictor to forecast latent embeddings without explicit reconstruction.
• The approach uses saliency-guided region ranking and sequential autoregressive predictions, establishing a curriculum that mirrors human attention.
• Empirical results demonstrate improved performance in linear probing, fine-grained benchmarks, and detection tasks by structuring latent dynamics.

A Joint-Embedding Predictive Architecture (JEPA) style latent dynamics model is an unsupervised visual or multi-modal representation learning paradigm in which an encoder and a predictor are jointly optimized to forecast the latent embeddings of held-out or masked content, conditioned on context, in an entirely latent space. Unlike reconstruction-based approaches, JEPA-style models never explicitly reconstruct observations; instead, all predictive supervision and training occur by comparing predicted and target latent embeddings, often accompanied by regularization to prevent collapse. The JEPA framework has undergone significant diversification, with DSeq-JEPA exemplifying an advanced variant that integrates saliency-driven region selection and sequential autoregressive predictions into the latent structure, introducing true latent dynamics and a curriculum-like, discriminative forecasting pipeline (He et al., 21 Nov 2025).

1. Architectural Foundations of JEPA-Style Latent Dynamics

A JEPA model fundamentally comprises three neural network modules: a context encoder, a target encoder, and a predictor. In the canonical image-based instantiation (I-JEPA), both encoders are transformer-based (e.g., ViT), producing patchwise latent embeddings and a [CLS] token. The context encoder serves as the "student", the target encoder (whose weights are a momentum-updated EMA of the student) as the "teacher". The predictor, typically a lightweight MLP or transformer, operates solely in latent space.

DSeq-JEPA extends this by injecting a saliency-aware controller, responsible for:

• Computing a similarity-based saliency map from the class token and patch tokens (cosine similarity).
• Extracting and ranking a set of non-overlapping regions $\{R_1, \dots, R_N\}$ by averaged normalized saliency score, producing an explicit ordering from most to least discriminative.
• Masking and sequentially feeding these ordered regions to the predictor, enabling causal conditioning ($$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$).
• Aligning the prediction to the target encoder's output embedding for $R_{k+1}$, yielding a truly sequential, autoregressive latent dynamics chain.

This pipeline departs from previous, permutation-invariant JEPA-style methods by imposing an explicit and structured causal order on latent predictions, reflecting both the spatial organization and semantic progression of regions.

2. Sequential Latent Prediction Objective

Classical I-JEPA minimizes a flat sum of embedding prediction errors for randomly sampled, independently masked regions:

$$L_{\rm IJEPA} = \sum_{i=1}^M \ell\bigl(g_\theta(p_i, s_{\rm context}), s^{\rm target}_i\bigr),$$

with $\ell$ typically a Huber or squared error.

DSeq-JEPA replaces this independence with an autoregressive sequence over discriminative regions. For $k=1, \ldots, N-1$:

$$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1},\dots,s_{R_k}),$$

and the overall loss:

$$L_{\rm DSeq} = \frac{1}{N-1} \sum_{k=1}^{N-1} \ell( \hat s_{R_{k+1}}, s_{R_{k+1}} ),$$

with $\ell(u, v)$ a Huber loss ($\psi$ function, $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$0), introducing causal dependence and allowing each prediction to condition on all prior region embeddings. This latent Markov chain injects non-trivial dynamics into the representation and abolishes the permutation symmetry inherent in I-JEPA (He et al., 21 Nov 2025).

3. Saliency-Guided Curriculum and Easy-to-Hard Sequencing

Discriminative region ranking in DSeq-JEPA utilizes the self-attention connectivity between the class token and patch tokens at a particular transformer layer. Similarity maps $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$1, are normalized and thresholded (Otsu's method) to produce binary masks. Connected-component labeling extracts regions $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$2, with each assigned its mean normalized saliency $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$3 and sorted.

This ranking is central: prediction proceeds from $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$4 (primary, highly informative regions—object parts, semantic anchors) to $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$5 (background). Empirically, stepwise prediction loss increases with $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$6, confirming that the model faces an easy-to-hard progression as it traverses the saliency-ordered curriculum. Training further employs a probabilistic curriculum: early epochs interpolate between uniform I-JEPA region sampling and discriminative selection, annealing to full curriculum use as raw saliency stabilizes.

4. Empirical Performance and Ablation Insights

DSeq-JEPA achieves consistent improvement over both baseline I-JEPA and weakly sequential ablations:

• Linear probing (ImageNet, ViT-B/16): 72.4% (I-JEPA) → 73.5% (DSeq, +1.1%)
• Fine-grained benchmarks (iNat21/CUB/Cars): e.g., 65.3% (I-JEPA, CUB) → 66.2% (DSeq); 65.9% (Cars) → 67.3%
• Detection/segmentation (MS-COCO, ADE20K): AP$$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$7 +0.6, AP$$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$8 +0.5, mIoU +0.5
• Low-level reasoning (Clevr/Count, Dist): +0.8 / +0.3 absolute improvement

Notably, neither discriminative selection nor sequential prediction alone suffices; random order sequencing degrades performance (71.7% vs 72.0%), and spatial scan order yields only partial improvements (72.7%). It is the synergy of saliency-driven order and sequential prediction that delivers the largest gains (73.5%), establishing the importance of explicitly structured latent dynamics in JEPA-style models (He et al., 21 Nov 2025).

5. Methodological Extensions and Open Research Directions

DSeq-JEPA, as a framework for JEPA-style latent dynamics, supports multiple axes of methodological extension:

• Learnable saliency mechanisms: Replacing the heuristic (CLS-token similarity + Otsu) with an end-to-end optimized region proposal network for dynamic and adaptive region selectivity.
• Variable region set size: Allowing $$\hat s_{R_{k+1}} = g_\theta(p_{R_{k+1}}, s_{R_1}, \ldots, s_{R_k})$$9 to adapt per image, or equipping the controller with a learned stopping rule to modulate sequence length based on scene complexity.
• Cross-modal sequential dynamics: Adapting the approach to predict multi-modal latent structures (e.g., caption tokens or speech features) ordered by saliency and semantic prominence.
• Longer-horizon planning: Interpreting the sequential predictor as a mini-latent RNN capable of multi-step rollout for visual planning or video forecasting, tightly coupling perceptual saliency with latent trajectory generation.
• Reinforcement-style selection: Framing region ordering as a learned policy whose reward is tied to downstream discriminability or task performance.

These directions suggest a general blueprint in which structured causal order, curriculum learning, and task-relevant saliency coalesce to furnish JEPA models with both robust generalization and interpretable latent dynamics.

6. Theoretical and Practical Implications

The principal contribution of DSeq-JEPA is to demonstrate, both theoretically and empirically, that breaking the flat, order-agnostic symmetry of classic JEPAs injects non-trivial temporal and structural inductive bias into the latent space. The latent dynamics induced by sequential, saliency-driven prediction promote the emergence of representations that are (1) discriminative (focusing on information-rich cues and generalizable cues), and (2) generalizable (outperforming strong I-JEPA variants on both coarse and fine-grained recognition, and on spatial reasoning tasks).

By mirroring human attentional deployment and semantic progression within the model’s prediction pathway, DSeq-JEPA provides a foundation for further research into world models where latent space evolves according to curriculum-aligned, task-driven causal structures, enabling both improved sample efficiency and downstream task transfer (He et al., 21 Nov 2025).