Improving robustness against common corruptions by covariate shift adaptation (2006.16971v2)

Abstract: Today's state-of-the-art machine vision models are vulnerable to image corruptions like blurring or compression artefacts, limiting their performance in many real-world applications. We here argue that popular benchmarks to measure model robustness against common corruptions (like ImageNet-C) underestimate model robustness in many (but not all) application scenarios. The key insight is that in many scenarios, multiple unlabeled examples of the corruptions are available and can be used for unsupervised online adaptation. Replacing the activation statistics estimated by batch normalization on the training set with the statistics of the corrupted images consistently improves the robustness across 25 different popular computer vision models. Using the corrected statistics, ResNet-50 reaches 62.2% mCE on ImageNet-C compared to 76.7% without adaptation. With the more robust DeepAugment+AugMix model, we improve the state of the art achieved by a ResNet50 model up to date from 53.6% mCE to 45.4% mCE. Even adapting to a single sample improves robustness for the ResNet-50 and AugMix models, and 32 samples are sufficient to improve the current state of the art for a ResNet-50 architecture. We argue that results with adapted statistics should be included whenever reporting scores in corruption benchmarks and other out-of-distribution generalization settings.

Overview of the Paper on Robustness via Covariate Shift Adaptation

This paper offers a detailed investigation into enhancing the robustness of machine vision models against common image corruptions through the employment of covariate shift adaptation. It addresses the well-documented vulnerability of current state-of-the-art machine vision algorithms to image corruptions such as blurring and compression artefacts. These vulnerabilities significantly limit the models' performance across various real-world applications. The authors propose that widely used benchmarks like ImageNet-C may often underestimate model robustness by failing to account for certain real-world scenarios where multiple unlabeled examples of corrupt images are available for adaptive purposes.

Key Findings and Numerical Outcomes

The core proposition of the research is to use the activation statistics of corrupted images to replace those estimated via batch normalization (BN) on the training set. Through this approach, the robustness of 25 different computer vision models improved consistently. Notably, ResNet-50 achieved an mCE (mean Corruption Error) of 62.2% on ImageNet-C, as opposed to the 76.7% without adaptation. The resilience of other models, including the DeepAugment+AugMix architecture, was similarly enhanced, reaching a state-of-the-art 45.4% mCE, improving from the previous best of 53.6%. The paper indicates that even minimal adaptation samples can significantly bolster robustness, as demonstrated by the ResNet-50 and AugMix models that benefited from the adaptation to a single example.

Implications and Future Directions

The paper asserts that adopting model statistics derived from corrupt images should be standard practice in future benchmarks of model robustness to corruption. This research draws pivotal connections between the fields of robustness against common corruptions and unsupervised domain adaptation, encouraging integrated approaches for more robust application capabilities.

Theoretical Underpinnings and Practical Relevance

The authors theorize that most distributional shifts between clean and corrupted images reveal themselves in the change of first and second order moments in internal representations of a deep network. Adapting BN statistics can effectively mitigate these shifts, offering a low-complexity yet impactful improvisation to existing model evaluation and deployment methodologies. This theoretical framework invites further exploration into domain adaptation techniques more specifically tailored for robustness in out-of-distribution scenarios.

Analytical and Methodological Insights

An extensive experimental setup spans a range of model architectures, datasets, and corruption scenarios. This includes evaluating models trained via Group Normalization, Fixup Initialization, and others against standard benchmarks like IN-C, IN-A, and ObjectNet. An interesting observation is the diminishing return on covariate shift adaptation for models trained on significantly larger datasets, such as the 3.5 billion image dataset, suggestive of inherent robustness through massive data pre-training.

Perspectives on Broader Impact

While this research enhances the robustness of machine vision, it also posits broader questions about reliance and interpretability in AI systems. Despite improvements, an undue trust in automation could arise, necessitating ongoing reflection on the broader societal impacts of deploying AI-driven decision systems.

In conclusion, the paper delivers compelling evidence that simple adaptation mechanisms in response to covariate shifts can significantly enhance model robustness. It not only opens the door for improved real-world applications but also lays a foundation for further paper into the adaptive capabilities of existing machine vision architectures.