Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning (2101.08732v3)

Abstract: We propose self-adaptive training -- a unified training algorithm that dynamically calibrates and enhances training processes by model predictions without incurring an extra computational cost -- to advance both supervised and self-supervised learning of deep neural networks. We analyze the training dynamics of deep networks on training data that are corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of any label information, highlighting that model predictions could substantially benefit the training processes: self-adaptive training improves the generalization of deep networks under noise and enhances the self-supervised representation learning. The analysis also sheds light on understanding deep learning, e.g., a potential explanation of the recently-discovered double-descent phenomenon in empirical risk minimization and the collapsing issue of the state-of-the-art self-supervised learning algorithms. Experiments on the CIFAR, STL, and ImageNet datasets verify the effectiveness of our approach in three applications: classification with label noise, selective classification, and linear evaluation. To facilitate future research, the code has been made publicly available at https://github.com/LayneH/self-adaptive-training.

• The paper presents self-adaptive training that dynamically updates targets using model predictions to enhance learning with noisy and adversarial data.
• It employs an exponential moving average of predictions to stabilize adaptive targets and mitigate issues like overfitting and double-descent.
• Experiments on CIFAR, STL, and ImageNet confirm improved generalization and robustness against label noise and adversarial attacks.

Overview of Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning

The paper introduces self-adaptive training, an approach aimed at unifying and enhancing both supervised and self-supervised learning by leveraging model predictions to dynamically adjust training processes without additional computational overhead. By analyzing deep learning models' behavior on training data corrupted by noise or structured adversarial examples, the authors reveal that model predictions can amplify the underlying useful information in the data. This phenomenon is observed even in the absence of label information, implying that such predictions can substantially improve training efficiency. This approach proposes insights into deep learning challenges, such as the double-descent phenomenon in empirical risk minimization and challenges in self-supervised learning like representation collapse. Experimental validation on popular datasets such as CIFAR, STL, and ImageNet substantiates the efficacy of the proposed method across various scenarios, including label noise, selective classification, and linear evaluation in constrained resources.

The implications of this research are significant for AI's development, particularly in scenarios where labeled data is scarce or noisy. By facilitating improved performance in such conditions without requiring modifications to existing network architectures or significant computational cost, self-adaptive training represents a practical advancement in both maintaining and enhancing the generalization capabilities of deep neural networks.

Key Contributions

1. Understanding Empirical Risk Minimization Dynamics:
   • The authors provide an in-depth analysis of the empirical risk minimization (ERM) training processes in deep models affected by different types of data corruption, such as randomized labels, Gaussian noise, shuffled pixels, and adversarial manipulations. They identify failure scenarios due to traditional ERM's propensity to overfit noise, supported by empirical evidence of deep models leveraging prediction mechanisms to distill useful data information.
2. Self-Adaptive Training Algorithm:
   • A unified algorithm that bridges supervised and self-supervised learning by dynamically incorporating predictions from the model as adaptive training targets. This approach employs techniques such as exponential moving average of model predictions, which present a refined and stable target updating mechanism that does not require architecture alterations or additional computational costs.
3. Generalization Improvements:
   • Demonstrations of superior generalization under various noise conditions. Self-adaptive training alleviates the double-descent phenomenon, evidenced by experiments where deep networks demonstrate improved error-capacity curves in comparison to ERM when subject to noisy datasets.
4. Robustness Against Adversarial Attacks:
   • By introducing modifications to adversarial training mechanisms like TRADES, self-adaptive training improves the robustness of model predictions against strategic adversarial attacks, showing considerable improvements in robust accuracy compared to the baseline.
5. Self-Supervised Learning without Multi-View Dependency:
   • Unlike prevailing self-supervised methods that require multiple views or augmentations of inputs, self-adaptive training obtains competitive performance with single-view training. This finding questions the necessity of the computationally expensive multi-view setup, advocating for efficiency without sacrificing learning quality.
6. Applications in Noisy Label Learning and Selective Classification:
   • Self-adaptive training achieves state-of-the-art results in learning from datasets with significant label noise and empowers classifiers to perform selective classification effectively by leveraging dynamically adjusted confidence signals.

Implications and Future Directions

The introduction of self-adaptive training presents several implications in the field of AI research and application:

• Practicality in Data-Scarce Environments: By reducing dependency on labels and accommodating noisy data, this methodology can significantly reduce the cost and effort associated with high-quality data acquisition.
• Advancements in Robust AI: Through enhanced resistance to adversarial noise and improved model generalization, self-adaptive training contributes to the journey towards robust and reliable AI systems.
• Enhanced Training Efficiency: The approach foregoes the need for extensive computational resources without compromising performance, suggesting the potential for deployment in environments with limited computational power.

Future work might focus on extending self-adaptive training to broader, more diverse datasets and exploring its integration with emerging model architectures. The exploration of the algorithm's theoretical underpinnings might illuminate canonical principles that guide hyperparameter selections and customizations for specific tasks. Furthermore, the algorithm could potentially be adapted and enhanced for environments where adaptivity and robustness play a critical role, such as in autonomous systems and adaptive user interfaces.