SEED: Self-supervised Distillation For Visual Representation (2101.04731v2)

Abstract: This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.

• The paper identifies performance gaps in traditional self-supervised methods and proposes SEED to enhance small model training.
• It introduces a Teacher-Student framework where the Student mimics similarity score distributions from a pre-trained Teacher using unsupervised techniques.
• Empirical results demonstrate significant accuracy improvements, with MobileNet-V3 achieving 68.2% top-1 accuracy on ImageNet-1k.

An In-depth Analysis of Self-Supervised Distillation for Small Model Representation Learning

The paper presented focuses on a nuanced approach to self-supervised learning (SSL) for small models, confronting the limitations posed by existing self-supervised methodologies like contrastive learning when applied to models with a reduced number of parameters. The paper introduces the concept of SElf-SupErvised Distillation (SEED), where knowledge transfer from a larger, pre-trained network (the Teacher) to a smaller network (the Student) is achieved through self-supervised techniques. This innovative paradigm aims to leverage the representational power accrued by large networks to enhance the efficacy of small model training in SSL.

Key Contributions and Methodology

The paper makes several pivotal contributions:

1. Problem Identification: It underscores the performance gap when self-supervised techniques are implemented on smaller models compared to larger architectures. This is evident from empirical results on models like EfficientNet-B0 and MobileNet-V3-Large.
2. Proposed Framework - SEED: The main contribution is the SEED paradigm, which uses a Teacher-Student model structure to transfer knowledge in SSL without labeled data. The Student network is trained to mimic the similarity score distributions of instances as inferred by the Teacher, instead of conventional instance-level discrimination.
3. Decoupling SSL and KD: Unlike existing forms of knowledge distillation (KD) that are predominantly supervised, SEED operates in an unsupervised regime. It adapts principles from contrastive learning and knowledge distillation to self-supervised settings, allowing the Student to learn efficiently even with minimal computational resources.
4. Empirical Validation: Comprehensive experiments illustrate that SEED significantly improves the performance of small models across a range of downstream tasks. For instance, SEED enhances the top-1 accuracy on MobileNet-V3 from 36.3% to 68.2% on the ImageNet-1k dataset compared to self-supervised baselines.
5. Comparative Analysis: The paper involves an exhaustive evaluation across different distillation strategies, hyperparameter settings, and variations in teacher models, thereby establishing SEED's robustness and flexibility.

Numerical Results and Implications

The paper reports compelling numerical evidence to affirm the advantages of SEED:

• Improved top-1 accuracy results (67.6% on EfficientNet-B0 and 68.2% on MobileNet-V3-Large) elucidate SEED's efficacy over traditional SSL methods.
• The Teacher model's depth and training duration correlate positively with the Student's performance, proposing strategic considerations for model architecture selection and SSL training duration in future research.
• Moreover, the paper extends SEED's applicability across other datasets like CIFAR and SUN-397, demonstrating the broader transferability of learned representations.

Theoretical and Practical Implications

Theoretically, SEED challenges the notion that SSL is inherently suited for large models, proposing distillation-based knowledge transfer as a viable pathway for small models to achieve competitive advantage. Practically, the SEED framework can be instrumental for real-world applications requiring efficient computational footprints, such as on-device learning and scenarios demanding rapid inference times.

Future Directions

Looking forward, research may focus on optimizing SEED further, including exploring alternate distillation techniques, refining hyperparameter settings, and applying SEED across more diverse datasets and model architectures. Another promising direction could involve hybrid approaches that integrate SEED with supervised learning post-distillation, thereby potentially amplifying the efficacy in task-specific applications.

In summary, the SEED paradigm offers a substantive advancement in the field of small model learning under self-supervised constraints, marking a notable step in broadening the applicability of SSL methods across different computational environments.