Spectrum Extraction and Clipping for Implicitly Linear Layers (2402.16017v2)
Abstract: We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.
- Existence, stability and scalability of orthogonal convolutional neural networks. The Journal of Machine Learning Research, 23(1):15743–15798, 2022.
- Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
- Revisiting batch normalization for improving corruption robustness. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 494–503, 2021.
- Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39–57. Ieee, 2017.
- Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pp. 854–863. PMLR, 2017.
- Efficient bound of lipschitz constant for convolutional layers by gram iteration. arXiv preprint arXiv:2305.16173, 2023.
- Generalizable adversarial training via spectral normalization. arXiv preprint arXiv:1811.07457, 2018.
- Batch normalization is a cause of adversarial vulnerability. arXiv preprint arXiv:1905.02161, 2019.
- Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2):393–416, 2021.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
- Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. PMLR, 2015.
- Anil K Jain. Fundamentals of digital image processing. Prentice-Hall, Inc., 1989.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Learning multiple layers of features from tiny images. 2009.
- Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
- Hessian schatten-norm regularization for linear inverse problems. IEEE transactions on image processing, 22(5):1873–1888, 2013.
- Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
- Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
- Yousef Saad. Numerical methods for large eigenvalue problems: revised edition. SIAM, 2011.
- The singular values of convolutional layers. arXiv preprint arXiv:1805.10408, 2018.
- Towards practical control of singular values of convolutional layers. Advances in Neural Information Processing Systems, 35:10918–10930, 2022.
- Skew orthogonal convolutions. In International Conference on Machine Learning, pp. 9756–9766. PMLR, 2021.
- Improved deterministic l2 robustness on cifar-10 and cifar-100. arXiv preprint arXiv:2108.04062, 2021.
- Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
- Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXiv:2104.07167, 2021.
- Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
- Evaluating the robustness of neural networks: An extreme value theory approach. arXiv preprint arXiv:1801.10578, 2018.
- Intriguing properties of adversarial training at scale. arXiv preprint arXiv:1906.03787, 2019.
- Lot: Layer-wise orthogonal training on improving l2 certified robustness. Advances in Neural Information Processing Systems, 35:18904–18915, 2022.
- Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2403–2412, 2018.
- Constructing orthogonal convolutions in an explicit manner. In International Conference on Learning Representations, 2021.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
