2000 character limit reached
Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures
Published 19 Feb 2024 in stat.ML and cs.LG | (2402.13285v1)
Abstract: In statistical learning theory, a generalization bound usually involves a complexity measure imposed by the considered theoretical framework. This limits the scope of such bounds, as other forms of capacity measures or regularizations are used in algorithms. In this paper, we leverage the framework of disintegrated PAC-Bayes bounds to derive a general generalization bound instantiable with arbitrary complexity measures. One trick to prove such a result involves considering a commonly used family of distributions: the Gibbs distributions. Our bound stands in probability jointly over the hypothesis and the learning sample, which allows the complexity to be adapted to the generalization gap as it can be customized to fit both the hypothesis class and the task.
- Pierre Alquier. User-friendly Introduction to PAC-Bayes Bounds. Foundations and Trends® in Machine Learning, 2024.
- On the properties of variational approximations of Gibbs posteriors. Journal of Machine Learning Research, 2016.
- An Exact Characterization of the Generalization Error for the Gibbs Algorithm. In Advances in Neural Information Processing System (NeurIPS), 2021.
- Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. Journal of Machine Learning Research, 2002.
- Occam’s Hammer. In Annual Conference on Learning Theory (COLT), 2007.
- Stability and Generalization. Journal of Machine Learning Research, 2002.
- Tightening Mutual Information-Based Bounds on Generalization Error. IEEE Journal on Selected Areas in Information Theory, 2020.
- Olivier Catoni. Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning. arXiv, abs/0712.0248, 2007.
- Diffusion for global optimization in ℝnsuperscriptℝ𝑛\mathbb{R}^{n}blackboard_R start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT. Siam Journal on Control and Optimization, 1987.
- Sharp Minima Can Generalize For Deep Nets. In International Conference on Machine Learning (ICML), 2017.
- Generalization in Adaptive Data Analysis and Holdout Reuse. In Advances in Neural Information Processing Systems (NIPS), 2015.
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In Conference on Uncertainty in Artificial Intelligence (UAI), 2017.
- Data-dependent PAC-Bayes priors via differential privacy. In Advances in Neural Information Processing System (NeurIPS), 2018.
- In search of robust measures of generalization. In Advances in Neural Information Processing System (NeurIPS), 2020.
- On the role of data in PAC-Bayes Bounds. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
- PAC-Bayesian Learning of Linear Classifiers. In International Conference on Machine Learning (ICML), 2009.
- Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
- PAC-Bayesian Analysis for a Two-Step Hierarchical Multiview Learning Approach. In Machine Learning and Knowledge Discovery in Databases (ECML-PKDD), 2017.
- Benjamin Guedj. A Primer on PAC-Bayesian Learning. arXiv, abs/1901.05353, 2019.
- Generalization Bounds via Information Density and Conditional Information Density. IEEE Journal on Selected Areas in Information Theory, 2020.
- Generalization Bounds: Perspectives from Information Theory and PAC-Bayes. arXiv, abs/2309.04381, 2023.
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning (ICML), 2015.
- Predicting the Generalization Gap in Deep Networks with Margin Distributions. In International Conference on Learning Representation (ICLR), 2019a.
- Fantastic Generalization Measures and Where to Find Them. In International Conference on Learning Representation (ICLR), 2019b.
- Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning. In NeurIPS 2020 Competition and Demonstration Track, 2021.
- Adam: A Method for Stochastic Optimization. In International Conference on Learning Representation (ICLR), 2015.
- Distribution-Dependent Analysis of Gibbs-ERM Principle. In Annual Conference on Learning Theory (COLT), 2019.
- THE MNIST DATASET of handwritten digits, 1998. URL http://yann.lecun.com/exdb/mnist/.
- Neural Complexity Measures. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networks. In Advances in Neural Information Processing System (NeurIPS), 2019.
- Tighter PAC-Bayes bounds through distribution-dependent priors. Theoretical Computer Science, 2013.
- Andreas Maurer. A Note on the PAC Bayesian Theorem. arXiv, cs.LG/0411099, 2004.
- David McAllester. Some PAC-Bayesian Theorems. In Annual Conference on Computational Learning Theory (COLT), 1998.
- Mechanism Design via Differential Privacy. In IEEE Symposium on Foundations of Computer Science (FOCS), 2007.
- Ilya Mironov. Rényi Differential Privacy. In IEEE Computer Security Foundations Symposium, 2017.
- Foundations of Machine Learning. Adaptive computation and machine learning. MIT Press, 2012.
- Uniform convergence may be unable to explain generalization in deep learning. In Advances in Neural Information Processing System (NeurIPS), 2019.
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
- PAC-Bayes Bounds with Data Dependent Priors. Journal of Machine Learning Research, 2012.
- Tighter Risk Certificates for Neural Networks. Journal of Machine Learning Research, 2021.
- Information-theoretic analysis of stability and bias of learning algorithms. In IEEE Information Theory Workshop, 2016.
- Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis. In Annual Conference on Learning Theory (COLT), 2017.
- Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds. In Advances in Neural Information Processing System (NeurIPS), 2018.
- PAC-Bayes Analysis Beyond the Usual Bounds. In Advances in Neural Information Processing System (NeurIPS), 2020.
- A PAC Analysis of a Bayesian Estimator. In Annual Conference on Computational Learning Theory (COLT), 1997.
- Striving for Simplicity: The All Convolutional Net. In International Conference on Learning Representations (ICLR) – Workshop Track, 2015.
- On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities. Theory of Probability and its Applications, 1971.
- A general framework for the practical disintegration of PAC-Bayesian bounds. Machine Learning, 2024.
- Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference on Machine Learning (ICML), 2011.
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv, abs/1708.07747, 2017.
- Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing System (NeurIPS), 2017.
- Robustness and generalization. Machine Learning, 2012.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.