Masked Siamese Networks for Label-Efficient Learning (2204.07141v1)

Abstract: We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available.

• The paper introduces a self-supervised framework that uses mask-denoising in a Siamese architecture to learn robust image representations with minimal labels.
• The methodology employs random patch masking and target sharpening to prevent representation collapse and enhance generalization.
• The approach achieves high accuracy on benchmarks like ImageNet-1K while reducing label dependency and computational resources.

Masked Siamese Networks for Label-Efficient Learning

This paper introduces Masked Siamese Networks (MSN), a self-supervised framework designed to enhance image representation learning by leveraging masked input observations, specifically aiming for efficiency in label-scarce scenarios. The use of MSNs pivots on generating robust, semantically-rich representations while integrating scalability with Vision Transformers (ViT).

The MSN architecture capitalizes on two primary innovations: random patch masking coupled with the classic Siamese network formulation, which has historically been employed to generate similar embeddings for augmented views of the same image. Here, MSN exercises an intriguing variant by employing a masking technique, selectively processing unmasked components to simplify the computation and enhance scalability.

Methodology

MSNs extend joint-embedding architectures by integrating a mask-denoising paradigm without reconstructing the omitted details, aligning with conventional representations like masked auto-encoders. In the MSN framework, multiple views are generated per image: an anchor view undergoes arbitrary masking, dropping a set fraction of image patches, whereas the target view remains intact, providing the ground truth representation. The task involves aligning the representation of the masked anchor with the unaltered target.

To avoid representation collapse—an inherent challenge with unsupervised representations—MSN employs entropy maximization coupled with target sharpening during the training process. Using multiple masked views encourages the model to become invariant towards missing information, an approach consistent with maximizing generalization under limited supervision.

Experimental Results

Empirically, MSN demonstrates substantial promise in low-shot learning scenarios, notably for tasks with a significant reduction in labeled data. On benchmarks such as ImageNet-1K under low-shot settings (1-5 labeled images per class), MSN achieves impressive accuracy improvements over prior methods that either employ contrastive loss functions or mask-based auto-encoders like MAE.

For instance, with only 5,000 labels, MSN realizes a 72.4% top-1 accuracy on the ImageNet-1K benchmark. Under even stricter label constraints (1% of available labels), MSN achieves 75.7% top-1 accuracy, surpassing previous state-of-the-art results with significantly reduced computational footprint.

The results highlight MSN's proficiency in efficiently deriving semantic representations, which notably surpass models that do not incorporate masking. This is further corroborated by qualitative analyses confirming enhanced invariance to missing patches—the robustness reflected in generative samples conditioned on MSN representations.

Computational Implications and Transfer Learning

Additionally, MSNs exhibit advantageous scaling behavior, especially linked to the reduced computational needs due to the selective patch processing strategy. Masking 70% of patches translates into substantial reductions in memory usage and computational time, yielding practical benefits when deploying large ViT models.

In transfer learning applications, MSN consistently demonstrates competitive performance on various datasets, whether through linear evaluation or fine-tuning methodologies. The results suggest robust generalizability across tasks, reinforcing the value of their derived image representations.

Conclusion and Future Directions

The paper posits Masked Siamese Networks as a potent tool for self-supervised learning in imaging tasks, promising both conceptual and empirical advancements. From a theoretical perspective, the framework achieves non-trivial representation learning by enforcing semantic consistency at a global level, eschewing pixel-level reconstruction.

Future research may explore more dynamic adaptive mechanisms for data transformation and inherent model invariances, tailoring them to task-specific needs across a wider variety of datasets. The foundational approach proposed by MSNs has significant versatility, paving the way for further developments in efficient, scalable representation learning across machine learning domains, particularly in scenarios of data scarcity.