Do Adversarially Robust ImageNet Models Transfer Better? (2007.08489v2)

Abstract: Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Our code and models are available at https://github.com/Microsoft/robust-models-transfer .

• The paper shows that adversarially robust models, despite lower source accuracy, extract superior features for transfer learning.
• It demonstrates consistent performance gains in both fixed-feature and full-network transfer settings over 12 classification tasks.
• The study highlights the critical role of hyperparameter tuning, especially the robustness level, in optimizing transfer outcomes.

An Overview of Adversarial Robustness in Transfer Learning

The paper "Do Adversarially Robust ImageNet Models Transfer Better?" explores the intersection of adversarial robustness and transfer learning in the context of computer vision models. The researchers examine whether adversarially robust networks, despite often demonstrating lower accuracy on the source dataset, can improve performance when applied to downstream tasks. This paper is a significant contribution to understanding how model robustness impacts feature extraction in transfer learning scenarios.

Key Contributions

The authors begin by addressing the fundamental question of whether adversarially robust models offer a superior basis for transfer learning compared to their non-robust counterparts. They argue that while traditionally, higher accuracy on datasets like ImageNet correlates with better transfer performance, another dimension—robustness—merits consideration. The paper asserts that adversarially robust models encapsulate improved feature representations, which might lead to better performance on a suite of classification tasks.

Methodology and Findings

The research employs a series of experiments using pre-trained ImageNet classifiers. The researchers evaluate both "fixed-feature" and "full-network" transfer settings, identifying consistent performance improvements in robust models across 12 classification tasks. The "fixed-feature" approach involves training a linear classifier on top of features derived from a pre-trained network, while "full-network" transfer fine-tunes all model parameters for the target task.

The results indicate that:

• In both transfer paradigms, robust models consistently match or outperform standard models, especially in fixed-feature settings.
• ImageNet accuracy alone is insufficient to predict transfer performance; robustness plays an independent role.
• Hyperparameter tuning, particularly the robustness level $\varepsilon$, is critical for optimal transfer performance, with dataset granularity influencing the choice of $\varepsilon$.

Implications

The findings suggest that adversarial robustness should be considered a crucial factor when selecting models for transfer learning. The distinct advantages of using robust models imply that their feature representations align more closely with human perception. The robustness-induced invariance might enable better generalization across diverse datasets, an essential trait for practical applications where labeled data is limited.

Future Directions

The paper opens several avenues for future research:

• Further exploration into the nature of robust feature representations and their intrinsic properties that facilitate transfer learning.
• Investigations into how model architecture—specifically width—affects the relationship between robustness and transfer performance.
• Development of new training paradigms that optimize both accuracy and robustness, balancing the trade-offs identified between source dataset performance and downstream task efficacy.

Conclusion

This research offers pivotal insights into the nuanced role of adversarial robustness in transfer learning. By demonstrating that robust models can provide a more reliable basis for transfer learning, the authors challenge traditional views focused solely on accuracy. This paper underscores the complexity of feature representation and opens discussions on refining model training strategies to harness robustness for enhanced performance in real-world applications.