Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack (1811.09310v1)

Abstract: Recent development in the field of Deep Learning have exposed the underlying vulnerability of Deep Neural Network (DNN) against adversarial examples. In image classification, an adversarial example is a carefully modified image that is visually imperceptible to the original image but can cause DNN model to misclassify it. Training the network with Gaussian noise is an effective technique to perform model regularization, thus improving model robustness against input variation. Inspired by this classical method, we explore to utilize the regularization characteristic of noise injection to improve DNN's robustness against adversarial attack. In this work, we propose Parametric-Noise-Injection (PNI) which involves trainable Gaussian noise injection at each layer on either activation or weights through solving the min-max optimization problem, embedded with adversarial training. These parameters are trained explicitly to achieve improved robustness. To the best of our knowledge, this is the first work that uses trainable noise injection to improve network robustness against adversarial attacks, rather than manually configuring the injected noise level through cross-validation. The extensive results show that our proposed PNI technique effectively improves the robustness against a variety of powerful white-box and black-box attacks such as PGD, C & W, FGSM, transferable attack and ZOO attack. Last but not the least, PNI method improves both clean- and perturbed-data accuracy in comparison to the state-of-the-art defense methods, which outperforms current unbroken PGD defense by 1.1 % and 6.8 % on clean test data and perturbed test data respectively using Resnet-20 architecture.

An Analysis of Parametric Noise Injection for Robustness in Deep Neural Networks

The paper "Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack" introduces an innovative method aimed at fortifying Deep Neural Networks (DNNs) against adversarial attacks. The proposed technique, Parametric-Noise-Injection (PNI), addresses the increasing concerns regarding the vulnerability of DNNs to adversarial examples which can significantly compromise classification accuracy with minimal, often imperceptible perturbations.

Core Proposal: Parametric-Noise-Injection (PNI)

PNI integrates trainable Gaussian noise into neural networks, utilizing noise injection as a form of regularization to enhance robustness. Unlike traditional approaches that inject noise during training, PNI adopts a trainable noise model that optimizes noise parameters alongside network parameters via a min-max optimization problem embedded within adversarial training. This setup ensures that the noise levels are automatically adapted for optimal performance rather than manually tuned for each layer via cross-validation.

Experimental Framework and Evaluation

The paper underscores extensive experiments, highlighting the superiority of PNI in defending against a variety of sophisticated adversarial attacks such as Projected Gradient Descent (PGD), Carlini & Wagner (C&W), Fast Gradient Sign Method (FGSM), and other black-box attacks. Noteworthy results show that PNI improves both clean- and perturbed-data accuracy when evaluated against state-of-the-art defense algorithms.

Significant improvements were registered with the ResNet-20 architecture, where PNI demonstrated a 1.1% and 6.8% increase in clean and perturbed test accuracy, respectively, compared to unenhanced PGD defense methodologies. Intriguingly, the introduction of PNI resulted in an optimized balance between clean-sample performance and adversarial robustness without a substantial compromise.

Contributions and Implications for Future Work

The research presents a novel perspective on adversarial robustness by integrating noise into neural network parameters as a tunable entity rather than a predefined hyperparameter. This parametric aspect addresses an important gap in robustness research, focusing on not just defense performance but also on the practicality and scalability of noise injection as a regularizer.

While the results are promising, the efficacy of PNI against adaptive adversaries remains an area for further exploration. Future work could expand on adaptive adversarial strategies that could circumvent noise-based defenses and optimize the element-wise scaling coefficients for more diverse neural architectures or tasks beyond image classification.

Moreover, in terms of theoretical implications, there lies a compelling narrative regarding the intersection of noise augmentation and adversarial training—whether noise injection may advance theoretical understandings of robustness and model generalization, especially in high-capacity networks.

Conclusion

The Parametric-Noise-Injection technique offers an innovative solution to one of the critical challenges in deep learning: ensuring robustness against adversarial perturbations. This research suggests that trainable randomness is a viable path to fortify models, offering tangible improvements over existing hard-coded noise applications. The integration of PNI into adversarial training schemes not only enhances defense capabilities but does so without sacrificing performance on clean data, making it a noteworthy contribution to the field of adversarial machine learning.