Automatic Shortcut Removal for Self-Supervised Representation Learning (2002.08822v3)

Abstract: In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for mitigating the effect shortcut features. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.

• The paper presents an adversarial lens network that automatically identifies and mitigates trivial shortcut features in SSL tasks.
• It demonstrates improved representation quality and generalization across tasks and datasets, including ImageNet and Places205.
• The approach enhances model interpretability by shifting focus from texture bias to semantically meaningful, shape-based features.

Automatic Shortcut Removal for Self-Supervised Representation Learning

The paper "Automatic Shortcut Removal for Self-Supervised Representation Learning" addresses a critical challenge in self-supervised learning (SSL) for visual representation: the exploitation of trivial, low-level visual "shortcut" features by neural networks to solve pretext tasks. These shortcuts hinder the learning of semantically meaningful representations that are beneficial for transfer learning. The authors propose a general framework that automatically identifies and mitigates these shortcuts, enhancing the robustness and utility of the learned representations.

Problem and Methodology

In SSL, a neural network is typically pre-trained on a pretext task with automatically generated labels, circumventing the need for manually annotated data. However, networks often find and exploit simple features—such as color aberrations or watermarks—that allow them to solve pretext tasks without developing deeper semantic understanding. Traditional approaches to counteract this involve manually identifying these shortcuts and designing specific augmentation strategies, which is limiting and non-generalizable.

The paper introduces an innovative approach by training an auxiliary "lens" network adversarially. The lens is designed to subtly alter input images, making the pretext task more challenging and thereby steering the main feature extractor network away from relying on easily learnable shortcut features. By removing shortcuts, the network is compelled to learn more comprehensive features. The method is tested on various pretext tasks and datasets, demonstrating consistent improvements in representation quality.

Results and Evaluation

The experiments encompass four common self-supervised tasks: Rotation, Exemplar, Relative Patch Location, and Jigsaw, evaluated across datasets such as ImageNet and Places205. The proposed approach consistently improved representation quality across all tasks and datasets, surpassing alternatives like the Fast Gradient Sign Method (FGSM) for adversarial training. Notably, the method not only enhanced performance on the primary dataset but also improved generalization to unseen datasets, indicating that it fosters learning of more transferable features.

Moreover, the lens method provides a tool for visualizing and interpreting the types of features neural networks focus on for different tasks. This interpretability extends our understanding of task-specific biases and feature importance, offering novel insights for SSL strategies. The method's ability to increase the shape-based decision proportion in networks, reducing the dominant texture bias typical in CNNs, suggests a tangible shift towards more semantic features.

Implications and Future Directions

The proposed framework has significant implications for both the theoretical understanding and practical applications of SSL. It automates the identification and mitigation of shortcut features, a process traditionally reliant on empirical insights and manual intervention, thus reducing human biases and errors. This advancement could accelerate the development of more robust and generalizable SSL models, enhancing their applicability in various domains where labeled data is scarce or unavailable.

Future research could delve into optimizing the balance between retaining potentially useful features and removing detrimental shortcuts using more sophisticated reconstruction losses or diverse lens architectures. Additionally, applying this methodology to supervised learning setups could reveal novel strategies for enhancing resilience against adversarial attacks and improving general feature learning.

Overall, the paper presents a compelling advance in self-supervised visual representation learning, providing a scalable method for enhancing model robustness and interpretability by addressing the longstanding challenge of shortcut exploitation.