Why do deep convolutional networks generalize so poorly to small image transformations? (1805.12177v4)

Abstract: Convolutional Neural Networks (CNNs) are commonly assumed to be invariant to small image transformations: either because of the convolutional architecture or because they were trained using data augmentation. Recently, several authors have shown that this is not the case: small translations or rescalings of the input image can drastically change the network's prediction. In this paper, we quantify this phenomena and ask why neither the convolutional architecture nor data augmentation are sufficient to achieve the desired invariance. Specifically, we show that the convolutional architecture does not give invariance since architectures ignore the classical sampling theorem, and data augmentation does not give invariance because the CNNs learn to be invariant to transformations only for images that are very similar to typical images from the training set. We discuss two possible solutions to this problem: (1) antialiasing the intermediate representations and (2) increasing data augmentation and show that they provide only a partial solution at best. Taken together, our results indicate that the problem of insuring invariance to small image transformations in neural networks while preserving high accuracy remains unsolved.

• The paper demonstrates that minor image perturbations, such as one-pixel shifts, can alter predictions by up to 30% in standard CNNs.
• The study reveals that conventional convolutional architectures violate classical sampling theory, leading to aliasing and poor invariance.
• Enhanced antialiasing and reduced subsampling can improve invariance but often incur significant computational costs and limited generalization.

Overview of Deep Convolutional Networks' Poor Generalization to Small Image Transformations

The paper by Aharon Azulay and Yair Weiss critically examines the widely held assumption that deep convolutional neural networks (CNNs) inherently possess invariance to minor image transformations. Contrary to popular beliefs, small translations or scalings can significantly affect the network's prediction accuracy.

Key Findings

1. Invariance Assumptions Challenged: The research demonstrates that neither the convolutional architecture's design nor data augmentation techniques are adequate to ensure the desired invariance. The convolutional structure neglects the classical sampling theorem, leading to aliasing effects, while data augmentation fails as CNNs only generalize to transformations closely resembling training set images.
2. Quantification of Invariance Failures: The paper provides a quantitative analysis of sensitivity in modern CNNs, illustrating that minor image perturbations—such as a one-pixel shift—can change predictions up to 30% of the time. This sensitivity varies across different architectures but consistently shows brittleness across common models.
3. Sampling Theorem and Subsampling: The paper explores the sampling theorem's implications, highlighting how subsampling and convolutional processes do not guarantee shiftability or invariance. The Fourier domain's insights reveal that high-frequency components introduced by nonlinearities result in vulnerability to small transformations.
4. Bias in Training Datasets: There is a significant photographer's bias in datasets like ImageNet, which influences CNNs to generalize invariance only to commonly observed configurations during training, leading to poor generalization for atypical inputs.

Proposed Solutions and Their Limitations

1. Antialiasing: Incorporating antialiasing methods to limit frequency artifacts proved partially effective. While it slightly improved invariance, it could not address the fundamental problem extensively across CNN architectures due to nonlinearities.
2. Increasing Data Augmentation: Enhanced data augmentation strategies could only achieve improved invariance for images that matched augmented training patterns closely, lacking generality to arbitrary novel cases.
3. Reducing Subsampling: Experiments suggest that reducing subsampling in CNN layers can improve translation invariance with a significant computational cost, indicating a trade-off between invariance and resource efficiency.

Implications and Future Directions

The findings emphasize the need for revised architectural considerations to bolster CNN robustness to small transformations. The implications stretch to real-world applications where slight errors from such perturbations can propagate significantly. Future explorations could focus on designing architectures or loss functions that inherently incorporate the sampling theorem or leverage adaptive filtering techniques within the CNN pipeline.

Additionally, understanding the role of dataset bias in model training may lead to the creation of more balanced datasets—potentially incorporating synthetic data generated with diverse alterations—to enhance generalization capabilities.

This research indicates that while CNNs have achieved impressive successes, the nuances of their invariance warrant further scrutiny to achieve more reliable deployment in critical tasks.