2000 character limit reached
Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration (2305.16173v3)
Published 25 May 2023 in cs.LG and cs.AI
Abstract: Since the control of the Lipschitz constant has a great impact on the training stability, generalization, and robustness of neural networks, the estimation of this value is nowadays a real scientific challenge. In this paper we introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory and a new alternative to the Power iteration. Called the Gram iteration, our approach exhibits a superlinear convergence. First, we show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. Then, it proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches. Code is available at https://github.com/blaisedelattre/lip4conv.
- Sorting out lipschitz function approximation. In International Conference on Machine Learning, 2019.
- On lipschitz regularization of convolutional layers using toeplitz matrix theory. AAAI Conference on Artificial Intelligence, 2021.
- Arnoldi, W. E. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics, 9(1):17–29, 1951.
- Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, 2017.
- Bhatia, R. Matrix analysis, volume 169. Springer Science & Business Media, 2013.
- Deep layers as stochastic solvers. In International Conference on Learning Representations, 2019.
- Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, 2017.
- Generalizable adversarial training via spectral normalization. In International Conference on Learning Representations, 2019.
- Efficient and accurate estimation of lipschitz constants for deep neural networks. In Advances in Neural Information Processing Systems, 2019.
- Golub, G. H. et al. Eigenvalue computation in the 20th century. Journal of Computational and Applied Mathematics, 2000.
- Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 2021.
- Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
- The vec-permutation matrix, the vec operator and kronecker products: A review. Linear & Multilinear Algebra, 9:271–288, 1981.
- Controllable orthogonalization in training dnns. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
- Jain, A. K. Fundamentals of digital image processing. In Fundamentals of digital image processing. Englewood Cliffs, NJ: Prentice Hall, 1989.
- Exactly Computing the Local Lipschitz Constant of ReLU Networks. In Advances in Neural Information Processing Systems, 2020.
- Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.
- Kublanovskaya, V. On some algorithms for the solution of the complete eigenvalue problem. USSR Computational Mathematics and Mathematical Physics, 1(3):637–657, 1962.
- Lanczos, C. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards, 1950.
- Lipschitz constant estimation of neural networks via sparse polynomial optimization. In International Conference on Learning Representations, 2020.
- Lehoucq, R. B. et al. Deflation techniques for an implicitly restarted Arnoldi iteration. SIAM Journal on Matrix Analysis and Applications, 17(4):789–821, 1996.
- Preventing gradient attenuation in lipschitz constrained convolutional networks. In Advances in Neural Information Processing Systems. 2019.
- Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
- Bounding multivariate trigonometric polynomials. IEEE Transactions on Signal Processing, 67(3):700–707, 2019.
- Plug-and-play methods provably converge with properly trained denoisers. In International Conference on Machine Learning, 2019.
- The singular values of convolutional layers. In International Conference on Learning Representations, 2019.
- Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation. Advances in Neural Information Processing Systems, 2022.
- Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- Singla, S. et al. Fantastic four: Differentiable and efficient bounds on singular values of convolution layers. In International Conference on Learning Representations, 2021.
- Trockman, A. et al. Orthogonalizing convolutional layers with the cayley transform. In International Conference on Learning Representations, 2021.
- Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, 2018.
- Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems, 2018.
- Orthogonal convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
- Yi, X. Asymptotic singular value distribution of linear convolutional layers. arXiv preprint arXiv:2006.07117, 2021.
- Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941, 2017.
- Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective. In Advances in Neural Information Processing Systems, 2022.
- RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications. AAAI Conference on Artificial Intelligence, 2019.