Scalable Patch-Level Self-Supervised Learning

Abstract: Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption, stipulating that task-relevant content is captured by the information common to different views, we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student-teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm, an influential foundation for today's strongest visual SSL methods. Notably, at 7B parameters, it exceeds the performance of DINOv3 on panoptic segmentation, despite being trained on $12\times$ less data without refinement stages. These results demonstrate that we can indeed design an SSL algorithm that learns strong representations, is principled and stable.

• The paper introduces JEM, a joint-embedding self-supervised learning method that achieves robust dense visual representations without combining multiple independent losses by aligning student and teacher croppings and using mutual information regularizers and multilayer supervision.
• JEM's robust dense representation stemming from mutual information maximization boosts performance in semantic, instance, and panoptic segmentation tasks structures across different encoders: ViT-L (303M) demonstrates a 48.1 mIoU on ADE20k; ViT-g (1.1B) reaches 51.2 mIoU; and ViT-7B gets the top 27.3 PQ on COCONut dataset and 23.9 ADE20k.
• Ablation studies show that local crops and masking are crucial for maintaining performance in semantic segmentation tasks whereas patch alignment has a substantial impact on representational outcomes.

The paper introduces a patch-level self-supervised learning method, referred to in the experiments as JEM, whose objective is derived from a multi-view InfoMax formulation rather than assembled from independently motivated losses. Its central claim is that strong dense visual representations can be obtained with a single patch-level joint-embedding objective, provided that cross-view alignment is accompanied by explicit controls for both statistical and structural collapse. The method is evaluated with ViT-L, ViT-g, and ViT-7B encoders and is reported to scale stably from 300 million to 7 billion parameters (2610.10013).

Problem setting and motivation

Contemporary visual SSL systems frequently combine global self-distillation, masked patch prediction, entropy or redundancy regularization, and additional stabilization mechanisms. DINOv2, for example, combines image-level DINO loss, patch-level iBOT prediction, and KoLeo regularization; DINOv3 adds Gram anchoring and high-resolution refinement to preserve spatial structure during extended large-scale training. The paper questions whether this accumulation of objectives is necessary, or whether the same functional requirements can be expressed through a more coherent information-theoretic construction.

The proposed starting point is the multi-view assumption: task-relevant information is assumed to be present in the information shared by different views of the same image. The views are generated through cropping, masking, and photometric transformations. Under this assumption, the representation should retain information about a teacher target that is predictable from the student view. The paper emphasizes that the assumption is conditional on the view construction and downstream task; it is therefore used as an idealized design principle rather than as an unconditional statement about visual information (2610.10013).

The resulting method differs from image-level self-distillation in two ways. First, the learning signal is applied to spatially corresponding patch representations rather than to a class token. Second, the anti-collapse mechanism is formulated explicitly in terms of per-element mutual information and cross-element redundancy. This formulation is intended to address failure modes that are specific to dense representations and are not controlled by image-level entropy balancing alone.

Information-theoretic derivation

Let $X_S$ and $X_T$ denote paired student and teacher views, let $R=f(X_S)$ be the student representation, and let $T$ be a stochastic teacher target generated from $X_T$. The paper seeks to maximize $I(R;T)$, the mutual information between the student representation and the teacher target. By the data-processing inequality,

$I(X_S;X_T) \geq I(X_S;T) \geq I(R;T).$

A variational lower bound yields

$$I(R;T) \geq I(X_T;T) - \mathbb{E}_{X_S,X_T} D_{\mathrm{KL}}\!\left( p(T\mid X_T)\,\|\,q(T\mid X_S) \right).$$

This decomposition has a direct algorithmic interpretation. The KL term is a cross-view alignment objective, while $I(X_T;T)$ is an anti-collapse term requiring the teacher targets to remain informative about the input. The teacher is implemented as an EMA of the student, so the anti-collapse quantity is not optimized directly through teacher parameters. Instead, the method applies corresponding regularization to student predictions and relies on the EMA dynamics to transmit the constraint to the teacher.

The target $T$ is factorized into elements indexed by patch position and feature layer. Under the factorization assumptions used in the derivation, the target mutual information decomposes into a sum of per-element information terms minus a total-correlation term:

$X_T$0

This decomposition identifies two distinct collapse modes. Per-element information can vanish, producing global collapse in which categorical predictions become uniform or constant. Alternatively, each element can retain information while different patches encode largely redundant information, producing structural collapse in which spatial features become indistinguishable within an image. The method assigns one regularizer to each mode.

The derivation is conceptually useful, but it does not make the practical objective assumption-free. In particular, the factorization of the conditional target distribution and the use of a structural proxy instead of direct total-correlation estimation are modeling choices. The information-theoretic lower bound motivates the losses, but does not establish that optimizing the implemented proxy is equivalent to maximizing the original mutual information.

JEM objective and architecture

JEM uses three components:

1. Patch-wise alignment: Student and teacher views are placed on a common patch lattice. For overlapping patches, the student categorical distribution is matched to the corresponding teacher distribution using a KL divergence. The objective is applied to patch-layer pairs, allowing intermediate teacher layers and final-layer representations to contribute to training.
2. Global collapse regularization: For every patch position, the method estimates mutual information between the input batch and the student categorical predictions. This estimate is the difference between the entropy of the batch-averaged categorical distribution and the average conditional entropy of individual predictions. Maximizing it encourages broad marginal usage of the $X_T$1 categories while keeping predictions input-dependent and relatively sharp.
3. Structural regularization: The method does not estimate $X_T$2 directly. Instead, the final student representation is projected through learned nonlinear maps and trained to reproduce pairwise cosine similarities from early and intermediate teacher layers. The structural loss therefore preserves information about the teacher’s spatial dependency structure without forcing the final representation to equal an intermediate-layer embedding.

This design distinguishes structural regularization from a simple diversity penalty. The objective permits semantically related patches to remain similar, but discourages a degenerate representation in which all patches encode the same image-level signal. In the authors’ interpretation, intermediate-layer geometry provides a proxy for local and spatial information that must remain recoverable in the final embedding.

The view construction combines mechanisms that are often treated separately. Each sample produces one teacher view, one global student view, and four local student views. Global and teacher crops are $X_T$3 pixels, while local crops are $X_T$4 pixels. Crops are shifted on a patch-aligned grid so that overlapping patches have exact one-to-one correspondences. Eighty percent of global and local student views are masked, with mask ratios of 65% and 50%, respectively. Student and teacher views receive independent photometric transformations. The method can therefore express ordinary masked prediction as a special case, but does not require fully overlapping crops or masking in principle.

The encoders are ViTs without class tokens and with four register tokens. All models use $X_T$5 patches. The principal scaling experiments use ViT-L with approximately 303 million parameters, ViT-g with approximately 1.13 billion, and ViT-7B with approximately 6.7 billion parameters.

Ablation evidence

The ablations clarify which parts of the method are essential and which primarily improve robustness or scaling.

The full ViT-L model trained on ImageNet-22k reaches 82.8% ImageNet-1k attention-probe accuracy and 48.1 mIoU on ADE20k. Removing local crops reduces these scores to 81.2% and 46.2 mIoU. Removing masking has a smaller effect on classification, 82.3%, but reduces ADE20k performance to 45.3 mIoU. Removing the student-teacher crop shift leaves classification unchanged at 82.8% but lowers ADE20k mIoU to 46.6. By contrast, removing independent photometric augmentation is highly damaging: performance falls to 74.8% on ImageNet-1k and 31.4 mIoU on ADE20k. This result implies that the method’s dense representations depend strongly on invariance to low-level appearance changes, even though the geometric view transformations are not individually indispensable.

The composition of the alignment loss produces a similarly uneven pattern. Removing multi-layer supervision changes ImageNet accuracy from 82.8% to 81.7% and slightly improves ADE20k from 48.1 to 48.3 mIoU. Removing masked-patch alignment produces 81.3% and 46.0 mIoU. However, removing visible-patch alignment causes a severe degradation to 70.8% ImageNet accuracy and 28.1 mIoU. The authors attribute this to the role of visible-patch alignment in preserving spatial correspondence. Thus, the result contradicts the common assumption that masked patches are the defining or indispensable component of patch-level SSL: in this setup, visible-patch supervision is substantially more important.

The objective ablation isolates the collapse mechanisms. Removing alignment reduces performance to 63.1% on ImageNet-1k and 22.4 mIoU on ADE20k. Removing global collapse regularization is catastrophic, producing 10.3% classification accuracy and 0.5 mIoU. This confirms that patch-wise alignment alone admits trivial solutions. The coefficient of the global regularizer is relatively insensitive once it is nonzero across a broad range, but setting it to zero produces degeneration.

Structural regularization has little impact at ViT-L scale: removing it changes ImageNet accuracy from 82.8% to 82.3% and ADE20k mIoU from 48.1 to 48.0. Its role becomes more consequential for larger models. At ViT-g scale, disabling the term causes patch representations to become progressively more spatially uniform after approximately 250,000 iterations and reduces ADE20k performance by 2.7 mIoU. The implication is that structural collapse is a scale-dependent failure mode: it may be nearly invisible in small-model ablations while becoming important during prolonged training of larger encoders.

Benchmark performance

The evaluation uses frozen backbones and lightweight probes for image classification, semantic segmentation, instance segmentation, panoptic segmentation, and monocular depth. Comparisons are made against published SSL methods and against retrained DINOv2 baselines using a closely matched implementation and training setup.

On ImageNet-1k-trained ViT-L, JEM obtains 82.8% classification accuracy, compared with 83.8% for the controlled DINOv2 baseline. The small classification deficit is accompanied by consistent dense-task gains:

Model	ImageNet-1k	ADE20k mIoU	VOC mIoU	Cityscapes mIoU	COCO mask AP	ADE20k PQ	COCONut PQ
DINOv2* ViT-L	83.8	45.7	83.3	66.5	12.1	18.0	22.1
JEM ViT-L	82.8	46.4	84.9	67.4	13.5	20.1	24.3

At ViT-g scale trained on ImageNet-22k, JEM again trails DINOv2 slightly on classification, 85.5% versus 86.5%, but improves substantially on dense prediction. It reaches 51.2 versus 50.0 ADE20k mIoU, 85.9 versus 83.6 VOC mIoU, 69.6 versus 68.5 Cityscapes mIoU, 14.7 versus 13.0 COCO mask AP, 22.4 versus 18.1 ADE20k PQ, and 25.7 versus 21.1 COCONut PQ. The four-point panoptic improvements are particularly important because they indicate that the method improves representations requiring both semantic recognition and instance-level spatial organization.

The results support a specific trade-off rather than uniform dominance. JEM is stronger on dense tasks while DINOv2 remains stronger on global classification in the controlled comparisons. This pattern is consistent with the method’s exclusive emphasis on patch representations and its omission of a class-token objective.

Scaling to 7 billion parameters

The main scalability claim is that JEM trains stably at 7 billion parameters without adding a separate refinement stage. ViT-L, ViT-g, and ViT-7B models are trained for 500,000 iterations on the A140M dataset, a curated 140-million-image corpus constructed for comparability with prior work. JEM improves task-averaged performance as model size increases and outperforms the matched DINOv2 baseline at all three scales.

At 7B, JEM uses approximately 1.85 trillion student tokens, compared with approximately 4.06 trillion for DINOv3. It also uses the A140M dataset rather than DINOv3’s LVD-1689M dataset and does not use DINOv3’s Gram-anchoring or high-resolution refinement stages. Nevertheless, the reported dense-task results are strong:

Model	Parameters	Training tokens	ImageNet-1k	ADE20k mIoU	VOC mIoU	Cityscapes mIoU	COCO mask AP	ADE20k PQ	COCONut PQ
DINOv3	7B	4.06T	89.0	55.9	86.7	75.2	16.0	21.6	27.0
JEM	7B	1.85T	86.9	54.4	87.1	73.6	15.6	23.9	27.3

The most consequential comparisons are panoptic segmentation: JEM reaches 23.9 PQ on ADE20k, compared with 21.6 for DINOv3, and 27.3 PQ on COCONut, compared with 27.0. It also exceeds DINOv3 on VOC semantic segmentation, 87.1 versus 86.7, and KITTI depth estimation, RMSE 2.125 versus 2.242. DINOv3 remains clearly superior on ImageNet classification, ADE20k semantic segmentation, COCO instance segmentation, and NYUv2 depth. Accordingly, the paper’s claim is not that JEM dominates every benchmark, but that a simpler patch-only objective can achieve competitive or superior dense representations under substantially less data and compute.

The comparison with DINOv3 should nevertheless be interpreted carefully. Although the training data and refinement stages favor DINOv3 as a more comprehensive foundation-model recipe, the models are not identical in architecture, data composition, or optimization history. The results establish strong empirical competitiveness under the reported protocols, but they do not isolate the contribution of the objective from every difference in data and training configuration.

Limitations and open questions

The multi-view assumption is the principal conceptual limitation. The method assumes that each view retains sufficient task-relevant information and that the intersection of view information is the appropriate target for representation learning. Strong cropping, masking, or photometric transformations can violate this condition for tasks requiring view-specific details, and the paper does not provide a formal characterization of when the assumption is valid.

The structural regularizer is also a proxy rather than a direct optimization of the derived objective. Total correlation over a large collection of discrete patch variables is intractable, so JEM matches pairwise similarity structures from selected teacher layers. This depends on the assumption that early and intermediate layers encode useful spatial dependencies and that those dependencies should be preserved in the final representation. The experiments support this proxy empirically, especially at large scale, but do not establish that it controls total correlation in a quantitatively identifiable way.

The reported evaluation is based on frozen-backbone probes, and several probes include hyperparameter searches and task-specific constructions. These protocols are appropriate for measuring representation quality, but they do not determine how the features perform after end-to-end fine-tuning or in settings outside the evaluated image datasets. The paper also leaves open whether the structural regularizer remains necessary under other architectures, patch sizes, resolutions, or training corpora.

Finally, the method retains substantial engineering choices despite its principled derivation: view overlap, patch-grid alignment, independent augmentations, categorical target dimensionality, regularizer schedules, EMA momentum, and layer-selection policies all matter. The claim that multi-objective SSL is unnecessary should therefore be understood narrowly. JEM replaces several semantically distinct losses with one derived objective plus two explicitly motivated regularizers; it does not eliminate the need for careful data augmentation, optimization schedules, and architectural inductive biases.

Conclusion

“Scalable Patch-Level Self-Supervised Learning” (2610.10013) presents JEM as a patch-only SSL framework derived from a variational multi-view InfoMax objective. Its three-part design—cross-view patch alignment, per-patch mutual-information regularization, and structural similarity preservation—targets the principal collapse modes of dense joint-embedding learning. The empirical results show a consistent advantage over controlled DINOv2 baselines on semantic, instance, and panoptic segmentation, while maintaining competitive classification and depth performance. The most notable result is stable training at 7 billion parameters with 12 times less data than DINOv3 and higher reported performance on several dense benchmarks. The remaining technical question is how broadly the proposed information-theoretic decomposition and structural proxy transfer across view distributions, architectures, and downstream adaptation regimes.