CNN-JEPA: CNN Self-Supervision via JEPA

Updated 14 July 2026

The paper demonstrates that applying the JEPA framework to CNNs with a tailored masking strategy yields competitive ImageNet accuracy using a ResNet-50 backbone.
The method integrates a sparse CNN encoder with depthwise separable convolutions in a student-teacher setup, aligning masked latent predictions with teacher features.
CNN-JEPA emphasizes network-aware, multi-block masking to efficiently exploit convolutional inductive biases while reducing training time compared to other self-supervised methods.

Searching arXiv for CNN-JEPA and closely related JEPA papers. CNN-JEPA is a self-supervised learning method that adapts the Joint Embedding Predictive Architecture (JEPA) paradigm to convolutional neural networks, with the explicit aim of bringing masked latent prediction to CNN backbones rather than Vision Transformers. It was introduced as a response to the observation that recent JEPA advances, especially I-JEPA, had shown promising results for Vision Transformers, while adapting such methods to Convolutional Neural Networks presents unique challenges. The method combines a sparse CNN encoder for masked inputs, a fully convolutional predictor using depthwise separable convolutions, and an improved masking strategy, and was reported to achieve 73.3% linear top-1 accuracy on ImageNet-100 with a standard ResNet-50 encoder (Kalapos et al., 2024).

1. Conceptual position within JEPA-based self-supervision

JEPA learns representations by predicting target embeddings from context embeddings in latent space rather than reconstructing raw inputs. In the formulation discussed around CNN-JEPA, the predictive task is to infer the representations of masked image regions from visible regions, thereby occupying a position between instance-discriminative and reconstruction-based self-supervision. The motivation for a CNN-specific variant is explicit: despite the rise of Vision Transformers, Convolutional Neural Networks retain strong inductive biases such as spatial locality and translational invariance, and existing masked image modeling approaches were mainly tailored to ViTs (Kalapos et al., 2024).

The central claim of CNN-JEPA is therefore not merely that a JEPA objective can be placed on top of a convolutional backbone, but that CNNs require a distinct treatment of masking, feature prediction, and masked convolution. The paper identifies three practical obstacles: masking strategy, feature prediction, and masked convolution. This directly counters a common simplification that CNN-JEPA is just I-JEPA with a ResNet backbone. In the CNN setting, hierarchical downsampling and dense receptive fields make patch masking, predictor design, and information isolation materially different problems than in token-based transformer architectures (Kalapos et al., 2024).

2. Architecture and computational design

CNN-JEPA uses a student-teacher predictive framework built around a standard CNN backbone such as ResNet-50. The student encoder is a sparse CNN encoder: after each convolutional layer, outputs corresponding to masked input regions are zeroed out, propagating the mask throughout the network. This is intended to ensure that masked-out regions do not leak information through convolutional receptive fields. The teacher encoder is architecturally identical, but operates on unmasked images with standard convolutions; its parameters are updated through an exponential moving average of the student encoder (Kalapos et al., 2024).

The predictor is fully convolutional and uses two or three layers of depthwise separable convolutions, each consisting of depthwise $3 \times 3$ , pointwise $1 \times 1$ , BatchNorm, and ReLU. This design addresses a specific CNN bottleneck: deep but spatially small feature maps make standard convolutional predictors expensive because parameter count grows quadratically with depth, whereas with depthwise separable convolutions it grows linearly with depth. In the reported ablations, this predictor design was described as crucial both for good accuracy and for reduced parameter and compute cost (Kalapos et al., 2024).

A learnable mask token is inserted at masked positions in the student feature map before prediction. The mask token is shared across all batches and locations. This preserves a fully convolutional predictor interface while marking missing regions explicitly in latent space. CNN-JEPA further omits a separate projector network, which distinguishes it from SimCLR, BYOL, and VICReg-style CNN self-supervision. The omission is presented as part of the method’s simplification: the predictive structure itself is intended to encourage generalizable features without a dedicated projection head (Kalapos et al., 2024).

3. Masking strategy and latent prediction objective

The masking strategy is specifically adapted to CNN downsampling geometry. CNN-JEPA uses multi-block masking, in which the mask consists of several contiguous rectangular regions rather than randomly scattered patches. It also introduces network-aware masking: the mask patch size is chosen to match the CNN downsampling factor, for example $32 \times 32$ pixels for a ResNet-50 with $32\times$ downsampling, so that entire regions in the final low-resolution feature map are masked. In addition, a mixed masking strategy randomly alternates between multi-block and random masking per minibatch, and this mixed strategy is reported to yield the best results in the ablations (Kalapos et al., 2024).

The predictive target is the teacher feature map at masked spatial locations. Let $\hat{\mathbf{z}}_i$ denote the predictor output at masked position $i$ , and let $\mathbf{z}_{\text{teacher},i}$ denote the teacher feature at the same location. The loss is an $\ell_2$ regression applied only on masked positions:

$\mathcal{L} = \sum_{i \in \text{masked positions}} \left\|\hat{\mathbf{z}}_i - \mathbf{z}_{\text{teacher},i}\right\|_2^2.$

Only the student encoder and predictor are updated by gradient descent; the teacher is updated by EMA (Kalapos et al., 2024).

A notable feature of the method is its low dependence on data augmentation. The reported training recipe uses only basic augmentations—random resized crop and masking—and explicitly avoids the stronger color and geometric pipelines common in contrastive and self-distillation SSL. This suggests a different inductive bias: instead of relying on hand-crafted augmentation invariances, CNN-JEPA relies on masked latent prediction and teacher-student consistency to shape the representation space (Kalapos et al., 2024).

4. Empirical results and ablation structure

The empirical evaluation in the original study covered ImageNet-100 and ImageNet-1k with $224 \times 224$ images, using a mask patch size of $1 \times 1$ 0. On ImageNet-100, CNN-JEPA with ResNet-50 was reported to achieve 73.3% linear top-1 accuracy and 59.2% k-NN top-1 accuracy. BYOL, SimCLR, and VICReg remained stronger in absolute accuracy, but CNN-JEPA outperformed I-JEPA with ViT-Small and ViT-Base on the same dataset, where the reported linear top-1 range was 42.3–46.4% and k-NN top-1 was 31–35% (Kalapos et al., 2024).

On ImageNet-1k, CNN-JEPA with ResNet-50 reached 54.2% linear top-1 accuracy. The paper’s summary is therefore deliberately qualified: CNN-JEPA does not quite match the very best CNN-based SSL methods on accuracy, but it approaches the linear and k-NN top-1 accuracies of BYOL, SimCLR, and VICReg while requiring 17–35% less training time for the same number of epochs. The reported schedules were 13 hours for 200 epochs on ImageNet-100 and 70 hours for 100 epochs on ImageNet-1k, both on 4×A100 GPUs (Kalapos et al., 2024).

Method	Linear Top-1	k-NN Top-1
CNN-JEPA (ResNet-50, ImageNet-100)	73.3%	59.2%
BYOL (ResNet-50, ImageNet-100)	79.0%	68.2%
SimCLR (ResNet-50, ImageNet-100)	76.2%	67.3%
VICReg (ResNet-50, ImageNet-100)	74.2%	64.3%
I-JEPA (ViT-S/ViT-B, ImageNet-100)	42.3–46.4%	31–35%

The ablations isolate three design choices as particularly consequential. First, multi-block masking and mixed masking perform much better than random masking alone; one reported comparison on ImageNet-100 is 73.3% versus 57.6% linear top-1. Second, depthwise separable convolutions in the predictor improve both efficiency and accuracy, reducing parameter count and compute by over 90% compared to standard convolutions in the reported analysis. Third, predictor depth of two to three layers and a $1 \times 1$ 1 kernel were identified as the best-performing settings (Kalapos et al., 2024).

5. Relation to subsequent JEPA developments

Later JEPA work places CNN-JEPA within a broader family of predictive architectures rather than treating it as an isolated vision variant. "Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning" argues that I-JEPA’s EMA mechanism is insufficient to prevent collapse and that mean prediction is inaccurate, then introduces C-JEPA by integrating VICReg’s variance, invariance, and covariance regularization into JEPA. The paper states that these principles are model-agnostic and directly applicable to convolutional JEPAs, suggesting a concrete route for strengthening collapse prevention and statistical alignment in CNN-JEPA-like systems (Mo et al., 2024).

The most explicit downstream adoption appears in "WirelessJEPA: A Multi-Antenna Foundation Model using Spatio-temporal Wireless Latent Predictions", which states that it adopts CNN-JEPA as the underlying convolutional JEPA design. WirelessJEPA reshapes multi-antenna IQ streams into a 2D antenna-time representation, applies convolutional processing with block masking and efficient sparse computation over unmasked patches, and introduces antenna masking, time masking, and multi-block masking as structured spatio-temporal mask geometries. This is a direct demonstration that the CNN-JEPA recipe can be transferred beyond natural images to structured signal domains (Chu et al., 28 Jan 2026).

A broader theoretical extension is supplied by "Var-JEPA: A Variational Formulation of the Joint-Embedding Predictive Architecture -- Bridging Predictive and Generative Self-Supervised Learning", which argues that the variational framework is agnostic to encoder and predictor architecture and can therefore use CNNs for images and videos. That work reframes standard JEPA as a deterministic specialization and replaces heuristic anti-collapse mechanisms with an ELBO-based latent-variable formulation. This suggests a plausible path for future CNN-JEPA variants in which uncertainty quantification and anti-collapse behavior are handled probabilistically rather than through EMA and architectural heuristics alone (Gögl et al., 20 Mar 2026).

6. Significance, misconceptions, and open directions

CNN-JEPA’s significance lies less in setting the top benchmark among CNN SSL methods than in showing that JEPA is not intrinsically transformer-bound. The paper’s own framing is measured: the method offers a simpler, more efficient alternative to existing SSL methods for CNNs, requiring minimal augmentations and no separate projector network, while remaining competitive with established CNN baselines and clearly stronger than the ViT I-JEPA configurations tested on ImageNet-100 (Kalapos et al., 2024).

One common misconception is to equate JEPA-style efficiency with complete immunity to collapse or optimization pathologies. Later JEPA work makes clear that EMA alone may be insufficient to prevent entire collapse in I-JEPA, and proposes explicit variance/covariance/invariance regularization as a remedy (Mo et al., 2024). Applied back to CNN-JEPA, this does not invalidate the original method, but it does imply that CNN-specific JEPA research remains open on questions of stability, regularization, and mean prediction.

A second misconception is that CNN-JEPA is relevant only where CNNs remain the dominant backbone. Subsequent domain-specific work indicates a broader pattern: convolutional JEPA designs can be useful wherever the data possess strong local geometry and structured masking priors, as in multi-antenna wireless signals (Chu et al., 28 Jan 2026). This suggests that CNN-JEPA is best understood not simply as a ResNet pretraining recipe, but as the canonical JEPA instantiation for grid-structured or quasi-grid-structured domains where convolutional inductive biases remain advantageous.

In that sense, CNN-JEPA occupies a foundational position inside the JEPA lineage. It established a practical convolutional implementation with sparse masked encoding, latent regression on masked regions, depthwise separable fully convolutional prediction, and network-aware masking. Later work on regularization, probabilistic reformulation, and domain adaptation indicates that these ingredients are extensible rather than final, and that the method’s lasting importance may be as a design template for CNN-based predictive representation learning rather than only as a single ImageNet result (Kalapos et al., 2024).