Scalable Patch-Level Self-Supervised Learning
This presentation introduces JEM, a principled self-supervised learning method that learns visual representations from unlabeled images by maximizing information shared across different views. Unlike complex multi-objective approaches, JEM uses a unified patch-level framework derived from information theory, consisting of three components: cross-view alignment, global collapse prevention, and structural regularization. The method scales stably to 7 billion parameters and achieves state-of-the-art results on dense prediction tasks like semantic segmentation and depth estimation, demonstrating that simpler, theoretically grounded objectives can outperform more complex SSL designs.Script
Self-supervised learning typically combines multiple complex objectives to learn visual representations, but what if a single principled approach could scale to 7 billion parameters while delivering stronger results on dense prediction tasks?
The researchers introduce JEM, a joint-embedding method built on a deceptively simple idea: different views of the same image share task-relevant information, and a good representation should preserve exactly that common information across views.
JEM implements this through three loss components derived from information theory. Patch-wise alignment pulls corresponding patches together. Global regularization prevents all patches from collapsing to identical representations by balancing marginal entropy and prediction confidence. Structural regularization preserves spatial relationships by matching pairwise similarities between student and teacher features.
When trained on ImageNet and evaluated on dense tasks, JEM outperforms the controlled DINOv2 baseline across all scales, with advantages growing as models reach 7 billion parameters. On panoptic segmentation, JEM gains approximately 4 PQ points over DINOv2 at large scale.
Perhaps most striking: JEM trained with 12 times less data and no refinement stages surpasses DINOv3 on panoptic segmentation and KITTI depth, suggesting that a theoretically grounded objective can be more data-efficient than complex multi-stage pipelines.
JEM demonstrates that self-supervised learning at billion-parameter scale need not rely on stacking multiple competing objectives. To dive deeper into this work and create your own research video summaries, visit EmergentMind.com.
