2000 character limit reached
How many samples are needed to leverage smoothness?
Published 25 May 2023 in stat.ML, cs.AI, cs.LG, math.ST, and stat.TH | (2305.16014v3)
Abstract: A core principle in statistical learning is that smoothness of target functions allows to break the curse of dimensionality. However, learning a smooth function seems to require enough samples close to one another to get meaningful estimate of high-order derivatives, which would be hard in machine learning problems where the ratio between number of data and input dimension is relatively small. By deriving new lower bounds on the generalization error, this paper formalizes such an intuition, before investigating the role of constants and transitory regimes which are usually not depicted beyond classical learning theory statements while they play a dominant role in practice.
- Sobolev Spaces. Academic Press, 1975.
- Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 1950.
- Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv preprint arXiv:2303.01372, 2023a.
- Francis Bach. Learning Theory from First Principles. MIT press (announced), 2023b.
- Fast rates in structured prediction. In Conference on Learning Theory, 2021.
- Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 2006.
- William Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 1979.
- A probabilistic theory of pattern recognition. Springer, 2013.
- Minimax estimation via wavelet shrinkage. The Annals of Statistics, 26(3):879 – 921, 1998.
- Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research, 2020.
- Discriminatory analysis. Nonparametric discrimination: Consistency properties. Technical report, School of Aviation Medicine, Randolph Field, Texas, 1951.
- A Distribution-Free Theory of Nonparametric Regression. Springer, 2002.
- ε𝜀\varepsilonitalic_ε-entropy and ε𝜀\varepsilonitalic_ε-capacity of sets in functional spaces. Uspekhi Matematicheskikh Nauk, 1979.
- Just interpolate: Kernel “ridgeless” regression can generalize. The Annals of Statistics, 2020.
- The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2022.
- On the estimation of the derivatives of a function with the derivatives of an estimate. Applied and Computational Harmonic Analysis, 2022.
- Jaouad Mourtada. Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices. The Annals of Statistics, 50(4), aug 2022.
- An elementary analysis of ridge regression with random design. Comptes Rendus. Mathématique, 2022.
- Distribution-free robust linear regression. Mathematical Statistics and Learning, 2022.
- Local risk bounds in statistical aggregation. Preprint, 2023.
- Interpolation and learning with scale dependent kernels. In ArXiv, 2020.
- Jaak Peetre. New thoughts on Besov spaces. Duke University Mathematics Series, 1976.
- Consistency of interpolation with laplace kernels is a high-dimensional phenomenon. In Conference on Learning Theory, 2019.
- Gaussian Processes for Machine Learning. The MIT Press, 2005.
- Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.
- Estimating the approximation error in learning theory. Analysis and Applications, 2003.
- Learning theory estimates via integral operators and their approximations. Constructive Approximation, 2007.
- A jamming transition from under- to over-parametrization affects loss landscape and generalization. Journal of Physics A: Mathematical and Theoretical, 2019.
- Charles Stone. Consistent nonparametric regression. The Annals of Statistics, 1977.
- Hans Triebel. Interpolation Theory, Function Spaces, Differential Operators. North-Holland Publishing Co., 1978.
- Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
- Ding-Xuan Zhou. The covering number in learning theory. Journal of Complexity, 2002.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.