SiamJEPA: On the Role of Siamese Student Encoders in JEPA

Abstract: Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.

• The paper demonstrates that Siamese student encoders significantly enhance latent representation quality in JEPA by enforcing invariance through dual masking and KL divergence regularization.
• It employs an EMA teacher and dual predictors, achieving 70.2% ImageNet linear probing accuracy in just 400 epochs, thereby reducing training time compared to single-encoder methods.
• The findings suggest that the architectural innovations lower compute requirements and provide scalable insights for advancing self-supervised visual learning and related video applications.

SiamJEPA: The Role and Efficacy of Siamese Student Encoders in Joint Embedding Predictive Architectures

Introduction and Context

Self-supervised learning (SSL) in vision has progressively transitioned from contrastive learning approaches to reconstructive paradigms and, more recently, to predictive latent space modeling. Joint Embedding Predictive Architectures (JEPA) exemplify this direction by forgoing pixel-level reconstruction for more semantically driven latent prediction objectives. While prior JEPA implementations primarily deployed a single student encoder, the adoption and systematic investigation of Siamese student encoders—motivated by neurobiological hypotheses and recent brain-inspired representation learning frameworks—have been limited. This paper introduces SiamJEPA, a JEPA extension where masked Siamese student encoders, regularized by an exponential moving average (EMA) teacher, form the basis for both empirical and mechanistic analyses.

Architectural Innovations in SiamJEPA

The fundamental distinction in SiamJEPA lies in the introduction of two parallel student encoders, each ingesting independently masked versions of an image, thus establishing a strong inductive bias for invariance. The architecture comprises:

• Siamese Student Encoders: Both branches receive independently masked views, enforcing invariance and mitigating shortcut solutions.
• EMA Teacher Network: Parameters are updated in an EMA fashion, ensuring stable targets and preventing representational collapse.
• Dual Predictors: A linear predictor aligns global (CLS) representations between the branches, and a transformer-based predictor reconstructs masked latent tokens.

The inclusion of a regularization term on the KL divergence between posterior and prior latent distributions aligns network outputs, acting as both a consistency constraint and a mechanism for uncertainty modeling, reminiscent of PhiNet.

Figure 1: A JEPA architecture illustrating both single- and Siamese-encoder variants, with dashed lines denoting stop-gradient operations and disjoint masking.

Learning Objectives and Optimization

SiamJEPA optimizes two key losses:

1. Sim-1 (KL Alignment): KL divergence between posterior (dual-branch) and prior (single-branch) latent distributions, modulated by a regularization coefficient $\lambda_{\text{KL}}$. This forces consistent latent representations and mitigates view-specific noise.
2. Sim-2 (Masked Latent Prediction): Normalized mean squared error between predicted and ground-truth masked token embeddings, driving the core predictive objective of JEPA.

The masking strategy explicitly enforces disjoint support between the views to avoid trivial latent prediction and to maximize signal diversity.

Empirical Validation and Results

Comprehensive ablations validate SiamJEPA’s architectural hypotheses across multiple axes:

• Training Efficiency: SiamJEPA achieves competitive ImageNet linear probing accuracy (70.2% at 400 epochs) with fourfold fewer epochs relative to MAE (68.0% at 1600 epochs) and provides faster convergence than single-encoder JEPA under matched settings.
• Effect of KL Regularization: Increasing $\lambda_{\text{KL}}$ strengthens alignment, accelerates convergence, and boosts final accuracy, confirming the regularizer’s role as an effective inductive bias during early training.

Figure 2: The effect of varying $\lambda_{\mathrm{KL}}$ on KL divergence and total loss throughout training, illustrating regularization-dependent convergence rates and stability.

• Masking Schemes: Block masking outperforms random masking, but the method is robust to the precise scheme, suggesting architectural invariants generalize well.
• Representation Pooling: Mean pooling of intermediate ViT layers outperforms CLS token extraction, indicating information relevant to transfer resides in distributed token representations rather than highly compressed global tokens.
• Regularization Hyperparameters: Larger weight decay promotes continued improvement in longer training regimes, and optimal free-bit thresholds in KL regularization (e.g., 0.05) further tune performance.

Theoretical and Practical Implications

The results establish Siamese student encoders not as a mere architectural variant but as a critical inductive bias in predictive latent representation learning. Key theoretical insights include:

• Improved Latent Separability: The KL regularizer enforces that context representations from different masked views, sharing latent semantic content, coalesce. This yields embeddings more suitable for classification and transfer, potentially enhancing class boundary formations in high-dimensional representation spaces.
• Accelerated Learning Dynamics: Regularization by dual-view alignment yields more rapid ascent in downstream accuracy curves. This effect is most pronounced in initial epochs, likely reflecting earlier exploitation of mutual context information and stronger suppression of trivial solutions and collapse.
• Parameter Efficiency: Empirical findings suggest strong performance is achievable with shallow predictors and smaller decoder heads, implying architectural simplifications in JEPA models without loss of transfer capacity.

Practically, these findings indicate that the deployment of Siamese-encoded JEPA models can yield highly competitive visual representations at significantly reduced training budgets, lowering compute barriers while maintaining state-of-the-art performance.

Future Directions

Several directions emerge:

• Scaling Laws: Scaling SiamJEPA to larger ViT backbones (ViT-Large, ViT-Huge) and broader datasets (e.g., ImageNet-22K, JFT) could test the observed regularization advantages under Data/Compute scaling.
• Extension to Video and World Models: Given the superior performance of Siamese encoders in video JEPA variants and world models, deeper analyses of temporal predictive regularization and invariance in spatiotemporal domains are warranted.
• Neuro-inspired Mechanisms: Origins in the Complementary Learning Systems (CLS) theory and PhiNet architecture suggest cross-pollination with further neuroscientific hypotheses, potentially informing continual learning, rapid adaptation, and hierarchical predictive architectures.

Conclusion

SiamJEPA conclusively demonstrates that Siamese student encoders are a principled, empirically validated regularizer in the JEPA paradigm. They consistently improve latent representation quality, accelerate training, and reduce required computation. Importantly, these architectural insights refine the design space of self-supervised predictive frameworks for both image and video domains, and offer a robust, scalable foundation for future advances in biologically-motivated, self-supervised visual learning.

Reference: "SiamJEPA: On the Role of Siamese Student Encoders in JEPA" (2607.04044)