SiamJEPA: Siamese JEPA for Vision Learning
- SiamJEPA is a JEPA variant that introduces masked Siamese student encoders and an EMA teacher to predict latent embeddings.
- The model employs disjoint masking and a KL divergence loss to enforce consistent global representations and prevent collapse.
- Empirical results show improved early-stage learning efficiency and enhanced representation separability on ImageNet benchmarks.
Searching arXiv for SiamJEPA and closely related JEPA papers to ground the article in current literature. arXiv search: SiamJEPA, JEPA variants, and related works. SiamJEPA is a Joint Embedding Predictive Architecture (JEPA) variant that introduces masked Siamese student encoders together with an exponential moving average (EMA) teacher network for self-supervised representation learning in vision. In contrast to existing JEPA-based methods such as I-JEPA and V-JEPA, which typically employ a single encoder in the student network, SiamJEPA investigates whether Siamese student encoders function as a useful inductive bias for latent prediction. The formulation is presented as being more naturally aligned with brain-inspired representation learning frameworks, and it can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. The reported findings show that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training (Yamada, 4 Jul 2026).
1. Conceptual position within the JEPA family
JEPA models learn representations by predicting latent embeddings of masked regions rather than reconstructing pixels. This latent-space predictive paradigm distinguishes JEPA from Masked Autoencoders (MAE), which reconstruct pixels, and from contrastive methods that rely on positive and negative pair construction. Within this family, SiamJEPA addresses a specific architectural question: whether the student side of JEPA should remain single-branch, as in existing JEPA-based methods, or whether a Siamese design offers a better inductive bias for predictive representation learning (Yamada, 4 Jul 2026).
The motivation is framed against two backgrounds. First, standard JEPA practice had generally adopted a single student encoder. Second, brain-inspired approaches such as PhiNet use Siamese student encoders, motivated by hippocampus-neocortex interplay and temporal prediction in the brain. On this basis, SiamJEPA studies Siamese student encoders not as a superficial architectural variation, but as a structural bias that may alter the geometry and optimization of JEPA training (Yamada, 4 Jul 2026).
A useful contextual comparison comes from work connecting JEPA to contrastive self-supervised learning. C-JEPA identifies two limitations in I-JEPA: the inefficacy of EMA in preventing entire collapse and the inadequacy of I-JEPA prediction in accurately learning the mean of patch representations. It augments JEPA with Variance-Invariance-Covariance Regularization (VICReg) to improve stability and representation quality (Mo et al., 2024). This context suggests that SiamJEPA belongs to a broader line of research investigating how additional architectural or objective-level structure can regularize JEPA training.
2. Architecture and masking design
SiamJEPA proposes masked Siamese student encoders equipped with an EMA teacher network. The two student branches process independently masked views of the same input image, and the branches share weights while receiving different masked inputs. The teacher follows the student-teacher paradigm already common in self-supervised learning, with teacher parameters updated by EMA rather than direct gradient descent (Yamada, 4 Jul 2026).
A central implementation detail is disjoint masking. Each Siamese branch receives a different disjoint mask, and this is described as crucial for preventing shortcut learning and ensuring that the network cannot trivially copy over visible features. The architecture uses ViT-Base encoders, training is performed on ImageNet-1K, and the model adopts standard self-supervised training setups. A projector head with batch norm is used, and a shallow transformer predictor of 1–2 layers is reported to suffice (Yamada, 4 Jul 2026).
The EMA schedule moves from 0.99 in early training to 0.9999 late in training. Evaluation is conducted through linear probing with a linear classifier on mean-pooled features. The paper also reports mask ratios of 0.7, 0.75, and 0.8 for each branch, with the effective mask ratio lower due to symmetry (Yamada, 4 Jul 2026).
This design differs from several adjacent JEPA variants. TS-JEPA, for example, adapts JEPA to time series with a transformer encoder operating on non-masked patches, a transformer predictor, and an EMA encoder target, but it remains organized around latent prediction of masked regions rather than Siamese student branches (Ennadir et al., 29 Sep 2025). BiJEPA, by contrast, emphasizes bi-directional prediction and cycle-consistent predictability; it is symmetric in predictive direction rather than specifically Siamese in the student architecture (Huang, 10 Feb 2026).
3. Objective function and regularization mechanism
SiamJEPA combines two objectives. The first, termed Sim-1, aligns global representations from the two Siamese students via a Kullback–Leibler (KL) divergence loss between the posterior and prior of their global embeddings. The second, termed Sim-2, performs masked latent prediction by predicting the masked patch tokens of one view from the other using normalized mean squared error (NMSE) (Yamada, 4 Jul 2026).
The final loss is given as
Here, denotes NMSE between true and predicted patch features on unobserved masked patches for each view, and denotes the KL divergence with stop-gradient enforcing distributional alignment between the two student encoder outputs. The hyperparameter scales the Siamese consistency term (Yamada, 4 Jul 2026).
A notable property of the formulation is that setting regresses to standard JEPA with a single encoder or branch, while increasing enforces stronger coupling between the Siamese branches. This creates a unified formulation interpolating between a standard JEPA regime and the full Siamese variant (Yamada, 4 Jul 2026).
The KL term is described as the principal regularizer responsible for faster early-stage learning and improved separability. This role parallels, but is not identical to, the role of variance/covariance regularization in C-JEPA. C-JEPA uses variance and covariance terms to prevent entire collapse and an invariance term to align means of augmented views, yielding a combined loss that supplements JEPA prediction with VICReg (Mo et al., 2024). A plausible implication is that SiamJEPA and C-JEPA represent two different regularization strategies for JEPA: one architectural and distributional through Siamese student alignment, the other explicitly moment-based through VICReg.
4. Empirical results and ablations
SiamJEPA is evaluated on the ImageNet-1K linear probing benchmark. The paper reports the following results for top-1 linear probe accuracy:
| Method | Epochs | ImageNet Top-1 Linear Probe Accuracy (%) |
|---|---|---|
| MAE (official, 400e) | 400 | 61.9 |
| MAE (He et al., 1600e) | 1600 | 68.0 |
| CAE (1600e) | 1600 | 70.4 |
| I-JEPA (600e) | 600 | 72.9 |
| SiamJEPA (400e) | 400 | 70.2 |
These numbers are used to support two claims. First, SiamJEPA at 400 epochs matches or exceeds the performance of MAE trained for 1600 epochs. Second, SiamJEPA achieves higher linear probing accuracy than MAE while requiring longer training for MAE to reach similar performance. The paper characterizes this as evidence of improved learning efficiency under limited training budgets (Yamada, 4 Jul 2026).
The ablation studies emphasize the role of KL regularization. Increasing improves linear probe performance at all training stages; an explicit example given is that at 100 epochs accuracy is 64% versus 58% for strong versus weak regularization. At the same time, very high KL values may slow convergence, so the empirical conclusion is that balance is required (Yamada, 4 Jul 2026).
Further ablations report that block masking outperforms random masking, though both remain viable; slightly higher weight decay can yield gains with longer training; mean pooling from an intermediate layer outperforms the CLS token especially for fewer epochs; and shallow predictors of 1–2 layers suffice for strong performance in SiamJEPA, unlike deeper predictors in standard JEPA (Yamada, 4 Jul 2026). The paper also states that the KL regularization substantially boosts early-stage learning, with large performance gains at fewer epochs.
These results are narrower in scope than some later JEPA derivatives. For instance, MJEPA reports audio-visual gains across AudioSet-20K, ESC-50, FSD50K, and video benchmarks, with the central conclusion that cross-modal prediction is critical in a shared-encoder JEPA setting (Teotia et al., 23 Jun 2026). SiamJEPA, by contrast, is specifically a study of student-network structure in vision JEPA.
5. Interpretation as inductive bias
The principal interpretive claim of SiamJEPA is that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. The mechanism advanced for this claim is explicit global-level coupling between differently masked views through the KL term. This regularization constrains the two masked representations to be similar at the global level, thereby promoting invariance and robustness to masking and context while reducing the chance of collapse and smoothing optimization (Yamada, 4 Jul 2026).
The reported benefits are threefold. First, the Siamese objective yields enhanced regularization by constraining the representation space. Second, it yields faster convergence, particularly in early training. Third, it yields improved representational quality, described through better transferability and separability under linear probing (Yamada, 4 Jul 2026).
The paper also provides a biological framing. The structure is described as reflecting mechanisms in the brain and facilitating the modeling of both fast and slow learning and more robust memorization and generalization, with reference to Complementary Learning Systems theory. This is presented as part of the rationale for why Siamese student encoders may be more naturally aligned with brain-inspired representation learning frameworks (Yamada, 4 Jul 2026).
Related papers underscore that the JEPA literature increasingly treats structural bias as a central design variable. BiJEPA argues that bi-directional, cycle-consistent prediction can regularize semantic structure, but requires explicit norm control because symmetric prediction induces representation explosion (Huang, 10 Feb 2026). Sub-JEPA argues that latent regularization should match intrinsic geometry and applies Gaussian constraints in random subspaces to better balance bias and variance in JEPA-based world models (Zhao et al., 10 May 2026). In that landscape, SiamJEPA specifically isolates the role of Siamese student encoders as the load-bearing structural modification.
6. Relations to adjacent JEPA variants and possible extensions
SiamJEPA is best understood as one member of a rapidly expanding JEPA family in which different works alter masking geometry, encoder asymmetry, regularization, subspace structure, or modality coupling. TS-JEPA adapts JEPA to time series by reconstructing masked portions of a time series in latent space using non-masked context, with an EMA target encoder and an latent prediction loss (Ennadir et al., 29 Sep 2025). CF-JEPA removes masking in time series entirely, replacing it with crop-based multi-horizon forward prediction and exploiting an asymmetry between online and EMA encoders for downstream routing: classification is assigned to the online encoder, while forecasting and anomaly detection are assigned to the EMA target encoder, with a reported 27% reduction in multivariate forecasting mean squared error at no additional training cost (Lee et al., 5 Jun 2026).
Other variants modify the predictive structure itself. BiJEPA adds bi-directional prediction and cycle-consistent predictability between views, stabilized by norm regularization, and reports gains on synthetic periodic signals, Lorenz attractors, and MNIST (Huang, 10 Feb 2026). SD-JEPA explicitly decomposes the latent into orthogonal progression and content subspaces, applying cosine-margin triplet loss and SIGReg on disjoint coordinates, and reports that the 8-dimensional progression subspace explains 72–95% of task-progress variance across environments (Thil et al., 29 May 2026). ScaleAware-JEPA ties masking geometry to diffusion scale in multiscale physical fields rather than arbitrary patch coordinates, using Constrained Diffusion Decomposition to construct dense structural atlases without labels (Li, 29 Jun 2026). MJEPA extends JEPA to audio-visual learning with a single unified encoder and both intra-modal and cross-modal prediction, reporting that without cross-modal prediction a shared encoder degrades below unimodal baselines, whereas with it each modality benefits from the other (Teotia et al., 23 Jun 2026).
Within this broader taxonomy, SiamJEPA’s distinctive contribution is narrow but precise: it studies whether duplicated student encoders with shared parameters and disjoint masked views, together with KL-based alignment of global embeddings, improve JEPA learning. The evidence reported supports the view that they do, especially under limited training budgets and in the early stages of optimization (Yamada, 4 Jul 2026).
A recurring misconception is that SiamJEPA is simply JEPA with two copies of the same encoder. The formulation given in the paper argues otherwise. The decisive ingredients are the disjoint masking strategy, the KL-based coupling between posterior and prior of global embeddings, and the interpolation controlled by , which turns the Siamese structure into an explicit regularizer rather than a redundant duplication of parameters (Yamada, 4 Jul 2026).
7. Significance and open directions
SiamJEPA contributes to a broader re-evaluation of what stabilizes and improves JEPA training. One line of work emphasizes explicit anti-collapse penalties, as in C-JEPA’s variance/covariance regularization (Mo et al., 2024). Another emphasizes geometry-aware priors, as in Sub-JEPA’s Subspace Gaussian Regularization (Zhao et al., 10 May 2026). SiamJEPA instead argues that a carefully constructed Siamese student architecture itself can regularize the JEPA objective and accelerate the acquisition of useful invariances (Yamada, 4 Jul 2026).
The paper states that the design is not limited to images and that SiamJEPA’s principles port directly to JEPA-based architectures in video and multi-modal domains. This suggests a broader architectural hypothesis rather than a conclusion already established by direct experiments in those domains. A plausible implication is that future work may combine Siamese students with other JEPA innovations, such as VICReg-style anti-collapse terms, bi-directional prediction, subspace constraints, or cross-modal prediction.
In the current JEPA literature, SiamJEPA therefore occupies a specific and technically meaningful place. It provides a systematic study of Siamese student encoders in JEPA, formulates a loss that couples latent masked prediction with global KL alignment, demonstrates improved early-stage learning and competitive linear-probe performance on ImageNet-1K under constrained training budgets, and argues that student-network symmetry should be treated as a substantive inductive bias in predictive self-supervised representation learning (Yamada, 4 Jul 2026).