Papers
Topics
Authors
Recent
Search
2000 character limit reached

Koopman-based generalization bound: New aspect for full-rank weights

Published 12 Feb 2023 in cs.LG, math.FA, and stat.ML | (2302.05825v3)

Abstract: We propose a new bound for generalization of neural networks using Koopman operators. Whereas most of existing works focus on low-rank weight matrices, we focus on full-rank weight matrices. Our bound is tighter than existing norm-based bounds when the condition numbers of weight matrices are small. Especially, it is completely independent of the width of the network if the weight matrices are orthogonal. Our bound does not contradict to the existing bounds but is a complement to the existing bounds. As supported by several existing empirical results, low-rankness is not the only reason for generalization. Furthermore, our bound can be combined with the existing bounds to obtain a tighter bound. Our result sheds new light on understanding generalization of neural networks with full-rank weight matrices, and it provides a connection between operator-theoretic analysis and generalization of neural networks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Neural network learning: Theoretical foundations. Cambridge University Press, 2009.
  2. Stronger generalization bounds for deep nets via a compression approach. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
  3. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482, 2002.
  4. Spectrally-normalized margin bounds for neural networks. In Proceedings of Advances in Neural Information Processing Systems 31 (NIPS), 2017.
  5. Smooth maximum unit: Smooth activation function for deep networks using smoothing maximum technique. In Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  6. Modern Koopman theory for dynamical systems. SIAM Review, 64(2):229–340, 2022.
  7. Applied Koopmanism. Chaos (Woodbury, N.Y.), 22:047510, 2012.
  8. Reproducing kernel Hilbert space compactification of unitary evolution groups. Applied and Computational Harmonic Analysis, 54:75–136, 2021.
  9. Optimizing neural networks via Koopman operator theory. In Proceedings of Advances in Neural Information Processing Systems 34 (NeurIPS), 2020.
  10. Extraction and prediction of coherent patterns in incompressible flows through space-time Koopman analysis. Physica D: Nonlinear Phenomena, 402:132211, 2020.
  11. Truth or backpropaganda? an empirical investigation of deep learning theory. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
  12. Size-independent sample complexity of neural networks. In Proceedings of the 2018 Conference On Learning Theory (COLT), 2018.
  13. Nearly-tight VC-dimension bounds for piecewise linear neural networks. In Proceedings of the 2017 Conference on Learning Theory (COLT), pp.  1064–1068, 2017.
  14. Krylov subspace method for nonlinear dynamical systems with random noise. Journal of Machine Learning Research, 21(172):1–29, 2020.
  15. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
  16. Boundedness of composition operators on reproducing kernel Hilbert spaces with analytic positive definite functions. Journal of Mathematical Analysis and Applications, 511(1):126048, 2022a.
  17. Koopman and Perron-–Frobenius operators on reproducing kernel Banach spaces. Chaos: An Interdisciplinary Journal of Nonlinear Science, 32(12):123143, 2022b.
  18. Isao Ishikawa. Bounded composition operators on functional quasi-banach spaces and stability of dynamical systems. Advances in Mathematics, 424:109048, 2023.
  19. Metric on nonlinear dynamical systems with Perron-Frobenius operators. In Proceedings of Advances in Neural Information Processing Systems 32 (NeurIPS), 2018.
  20. Robust fine-tuning of deep neural networks with Hessian-based generalization guarantees. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
  21. Yoshinobu Kawahara. Dynamic mode decomposition with reproducing kernels for Koopman spectral analysis. In Proceedings of Advances in Neural Information Processing Systems 30 (NIPS), 2016.
  22. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
  23. Eigendecompositions of transfer operators in reproducing kernel Hilbert spaces. Journal of Nonlinear Science, 30:283–315, 2020.
  24. Stable invariant models via Koopman spectra. Neural Networks, 165:393–405, 2023.
  25. Bernard Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences, 17(5):315–318, 1931.
  26. Imagenet classification with deep convolutional neural networks. In Proceedings of Advances in Neural Information Processing Systems 26 (NIPS), 2012.
  27. Orthogonal deep neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(04):1352–1368, 2021.
  28. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9:4950, 2017.
  29. Complex unitary recurrent neural networks using scaled Cayley transform. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), 2019.
  30. Foundations of Machine Learning. MIT press, 2018.
  31. Norm-based capacity control in neural networks. In Proceedings of the 2015 Conference on Learning Theory (COLT), 2015.
  32. A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
  33. The role of over-parametrization in generalization of neural networks. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
  34. Sensitivity and generalization in neural networks: an empirical study. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
  35. An operator theoretic view on pruning deep neural networks. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
  36. B. A. Rubin. A note on the blaschke-petkantschin formula, Riesz distributions, and Drury’s identity. Fractional Calculus and Applied Analysis, 21:1641–1650, 2018.
  37. Yoshihiro Sawano. Theory of Besov Spaces. Springer Singapore, 2018.
  38. Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
  39. Loring W. Tu. An Introduction to Manifolds. Springer New York, second edition, 2011.
  40. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  41. Data-dependent sample complexity of deep neural networks via Lipschitz augmentation. In Proceedings of Advances in Neural Information Processing Systems 33 (NeurIPS), 2019.
  42. Improved sample complexities for deep neural networks and robust classification via an all-layer margin. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
  43. The Barron space and the flow-induced function spaces for neural network models. Constructive Approximation, 55:369–406, 2022.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up