Adversarial Self-Supervised Contrastive Learning (2006.07589v2)

Abstract: Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions, which are then used to augment the training of the model for improved robustness. While some recent works propose semi-supervised adversarial learning methods that utilize unlabeled data, they still require class labels. However, do we really need class labels at all, for adversarially robust training of deep neural networks? In this paper, we propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. Further, we present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data, which aims to maximize the similarity between a random augmentation of a data sample and its instance-wise adversarial perturbation. We validate our method, Robust Contrastive Learning (RoCL), on multiple benchmark datasets, on which it obtains comparable robust accuracy over state-of-the-art supervised adversarial learning methods, and significantly improved robustness against the black box and unseen types of attacks. Moreover, with further joint fine-tuning with supervised adversarial loss, RoCL obtains even higher robust accuracy over using self-supervised learning alone. Notably, RoCL also demonstrate impressive results in robust transfer learning.

Overview of "Adversarial Self-Supervised Contrastive Learning"

This paper presents a novel approach to enhancing adversarial robustness in deep neural networks (DNNs) without the need for labeled data, introducing a self-supervised method called Robust Contrastive Learning (RoCL). The authors address the ongoing challenge posed by adversarial attacks, which exploit perturbations to fool DNNs into making incorrect predictions. RoCL leverages the principles of self-supervised learning, specifically contrastive learning, to generate and utilize instance-wise adversarial attacks that confuse the model at the instance identity level, thus obviating the need for class labels traditionally required in adversarial training.

Key Contributions

1. Instance-wise Adversarial Attacks: The authors propose a novel adversarial attack strategy that operates on the instance level rather than relying on class labels. By maximizing the contrastive loss for instance discrimination, these instance-wise attacks perturb transformed samples, causing the model to misidentify them, thereby providing a path to adversarial training without labeled data.
2. Contrastive Learning Framework: RoCL enhances the self-supervised contrastive learning framework by incorporating adversarial perturbations. The novel objective seeks to maximize similarity between clean samples and adversarially perturbed augmentations of the same instance, thereby reducing the model's sensitivity to such perturbations in the latent representation space.
3. Empirical Validation: RoCL is empirically validated on benchmarks like CIFAR-10 and CIFAR-100, showing comparable performance to state-of-the-art supervised adversarial methods under white-box and black-box attack conditions. The findings suggest that RoCL improves clean accuracy and robustness to unseen adversarial attacks more significantly compared to conventional supervised adversarial training methods.

Implications and Future Directions

Practical Implications:

The proposed method offers significant benefits for situations where labeled data is scarce or unavailable, making adversarial robustness accessible in unsupervised settings. Its reliance on self-supervision aligns well with growing datasets where manual labeling is impractical or expensive.

Theoretical Implications:

This work contributes to the broader discourse on adversarial robustness by challenging the necessity of class labels and suggesting alternatives rooted in unsupervised learning paradigms. The concept of maintaining instance-level identity under transformation and noise could inspire further theoretical advances in understanding robust model representations.

Future Developments:

Future research could explore scaling RoCL to larger and more complex datasets, such as ImageNet, while examining the transferability of learned robust representations to other tasks. Additionally, integrating RoCL with other self-supervised learning tasks or hybrid models combining self-supervised and semi-supervised methods represents a promising avenue for enhancing both robustness and accuracy.

In summary, this paper introduces a groundbreaking approach to adversarial robustness leveraging the strengths of contrastive self-supervised learning, setting the stage for further innovations in deploying DNNs securely in real-world applications.