Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness (2408.05446v1)

Abstract: Adversarial examples pose a significant challenge to the robustness, reliability and alignment of deep neural networks. We propose a novel, easy-to-use approach to achieving high-quality representations that lead to adversarial robustness through the use of multi-resolution input representations and dynamic self-ensembling of intermediate layer predictions. We demonstrate that intermediate layer predictions exhibit inherent robustness to adversarial attacks crafted to fool the full classifier, and propose a robust aggregation mechanism based on Vickrey auction that we call \textit{CrossMax} to dynamically ensemble them. By combining multi-resolution inputs and robust ensembling, we achieve significant adversarial robustness on CIFAR-10 and CIFAR-100 datasets without any adversarial training or extra data, reaching an adversarial accuracy of $\approx$72% (CIFAR-10) and $\approx$48% (CIFAR-100) on the RobustBench AutoAttack suite ($L_\infty=8/255)$ with a finetuned ImageNet-pretrained ResNet152. This represents a result comparable with the top three models on CIFAR-10 and a +5 % gain compared to the best current dedicated approach on CIFAR-100. Adding simple adversarial training on top, we get $\approx$78% on CIFAR-10 and $\approx$51% on CIFAR-100, improving SOTA by 5 % and 9 % respectively and seeing greater gains on the harder dataset. We validate our approach through extensive experiments and provide insights into the interplay between adversarial robustness, and the hierarchical nature of deep representations. We show that simple gradient-based attacks against our model lead to human-interpretable images of the target classes as well as interpretable image changes. As a byproduct, using our multi-resolution prior, we turn pre-trained classifiers and CLIP models into controllable image generators and develop successful transferable attacks on large vision LLMs.

• The paper introduces multi-resolution input representations and dynamic self-ensembling via the CrossMax aggregation mechanism to improve robustness against adversarial attacks.
• It demonstrates significant performance gains on CIFAR-10 and CIFAR-100, achieving up to 78% adversarial accuracy with adversarial training.
• The study leverages the decorrelation of intermediate layer predictions to offer a robust defense that enhances interpretability without heavy reliance on traditional adversarial training.

Essay on "Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness"

Authors: Stanislav Fort, Balaji Lakshminarayanan

Affiliation: Google DeepMind

Adversarial robustness remains a critical challenge for the deployment of deep neural networks (DNNs) in real-world applications. The paper "Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness" by Stanislav Fort and Balaji Lakshminarayanan from Google DeepMind proposes innovative approaches to enhance the adversarial robustness of neural classifiers. This essay provides an expert summary and analysis of the methodologies, results, and implications presented in the paper.

Key Contributions

The authors propose two key techniques for improving adversarial robustness: multi-resolution input representations and dynamic self-ensembling through intermediate layer predictions, combined using a novel Vickrey auction-based aggregation mechanism termed CrossMax.

Methodologies

1. Multi-resolution Input Representations:
   • Drawing inspiration from biological vision systems, the authors introduce an active defense mechanism where images are viewed at multiple resolutions simultaneously. This mimics the effect of microsaccades in human vision, helping to refresh the visual input and enhance robustness.
   • The input image is transformed into a channel-wise stack of downsampled versions, providing the model with multiple perspectives of the same image. This multi-channel input enhances the network's ability to identify adversarial perturbations as it integrates information across resolutions.
   • This approach alone was found to significantly boost adversarial robustness, as evidenced by the results on CIFAR-10 and CIFAR-100 datasets.
2. Dynamic Self-Ensembling via CrossMax Aggregation:
   • Intermediate layer activations exhibit partial robustness; an adversarial example crafted to fool the final layer of a classifier may not equally confuse the intermediate layers.
   • CrossMax, inspired by Vickrey auctions, dynamically ensembles the predictions from intermediate layers, capitalizing on their de-correlation under adversarial attacks. This prevents the adversary from easily manipulating the classifier by exploiting the collective strengths of intermediate layers.
   • This dynamic aggregation is found to further enhance robustness without the need for adversarial training.

Numerical Results

The proposed methods were empirically validated, showing significant gains in adversarial robustness:

• On CIFAR-10, without adversarial training, the authors achieved approximately 72% adversarial accuracy with a finetuned ImageNet-pretrained ResNet152, directly comparable to the top three models.
• On CIFAR-100, the results reached approximately 48%, representing a 5% improvement over the best current dedicated approaches.
• When adversarial training was added, these figures rose to approximately 78% for CIFAR-10 and approximately 51% for CIFAR-100, showing improvements of 5% and 9%, respectively.

These results underscore the efficacy of multi-resolution inputs and dynamic self-ensembling in achieving high-quality, robust representations against adversarial threats.

Theoretical Insights and Implications

The key insight from this paper lies in the partial decorrelation of adversarial vulnerabilities across intermediate layers. By leveraging this property and aggregating predictions dynamically, the proposed methods offer a robust defense that mitigates the potential for adversarial exploitation.

The implications of this research are multifaceted:

• Practical Applications: The techniques can be readily applied to improve the robustness of pre-trained models on various datasets, without the computational overhead of adversarial training.
• Interpretability: The resulting models exhibit human-interpretable attacks, where adversarial perturbations align more closely with meaningful image changes rather than noise-like artifacts.
• Future Research: The insights gained point towards exploring the hierarchical nature of neural representations and developing further robust aggregation mechanisms. Additionally, examining these techniques in conjunction with other defense strategies could yield even more robust models.

Conclusion

The methodologies proposed in this paper offer a robust framework for enhancing the adversarial robustness of neural networks through biologically inspired multi-resolution inputs and dynamic self-ensembling. The significant improvements in adversarial accuracy on standard benchmarks demonstrate the utility and effectiveness of these techniques. Moreover, the insights into the robustness of intermediate layer features and the resultant interpretable attacks provide valuable directions for future research. This work not only advances practical robustification of classifiers but also contributes to the broader understanding of adversarial phenomena in deep learning.