2000 character limit reached
Online Infinite-Dimensional Regression: Learning Linear Operators (2309.06548v3)
Published 8 Sep 2023 in stat.ML and cs.LG
Abstract: We consider the problem of learning linear operators under squared loss between two infinite-dimensional Hilbert spaces in the online setting. We show that the class of linear operators with uniformly bounded $p$-Schatten norm is online learnable for any $p \in [1, \infty)$. On the other hand, we prove an impossibility result by showing that the class of uniformly bounded linear operators with respect to the operator norm is \textit{not} online learnable. Moreover, we show a separation between sequential uniform convergence and online learnability by identifying a class of bounded linear operators that is online learnable but uniform convergence does not hold. Finally, we prove that the impossibility result and the separation between uniform convergence and learnability also hold in the batch setting.
- A new approach to collaborative filtering: Operator estimation with spectral regularization. Journal of Machine Learning Research, 10(3), 2009.
- Relative loss bounds for on-line density estimation with the exponential family of distributions. Machine learning, 43:211–246, 2001.
- On the Clarkson-McCarthy inequalities. Mathematische Annalen, 281:7–12, 1988.
- Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121–157, 2021.
- Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
- Prediction, learning, and games. Cambridge university press, 2006.
- On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050–2057, 2004.
- John B Conway. A course in functional analysis (1990). Graduate Texts in Mathematics, 1990.
- Multiclass learnability and the erm principle. In Sham M. Kakade and Ulrike von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory, volume 19 of Proceedings of Machine Learning Research, pages 207–232, Budapest, Hungary, 09–11 Jun 2011. PMLR.
- Convergence rates for learning linear operators from noisy data. SIAM/ASA Journal on Uncertainty Quantification, 11(2):480–513, 2023.
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
- Frédéric Ferraty. Nonparametric functional data analysis. Springer, 2006.
- Multiclass online learning and uniform convergence. In The Thirty Sixth Annual Conference on Learning Theory, pages 5682–5696. PMLR, 2023.
- Andreas Kirsch. An introduction to the mathematical theory of inverse problems, volume 120. Springer, 2011.
- Data-driven approximation of the koopman generator: Model reduction, system identification, and control. Physica D: Nonlinear Phenomena, 406:132416, 2020.
- Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481, 2021.
- Samuel Lanthaler. Operator learning with pca-net: upper and lower complexity bounds. arXiv preprint arXiv:2303.16317, 2023.
- Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
- Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10), 2021.
- Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem. arXiv preprint arXiv:2211.08875, 2022.
- Vc classes are adversarially robustly learnable, but only improperly. In Conference on Learning Theory, pages 2512–2530. PMLR, 2019.
- Balaubramaniam Kausik Natarajan. Some results on learning. Technical Report CMU-RI-TR-89-06, The Robotics Institute, Carnegie Mellon University, 1989.
- The random feature model for input-output maps between banach spaces. SIAM Journal on Scientific Computing, 43(5):A3212–A3243, 2021.
- Francisco Duarte Moura Neto and Antônio José da Silva Neto. An introduction to inverse problems with applications. Springer Science & Business Media, 2012.
- Online non-parametric regression. In Conference on Learning Theory, pages 1232–1264. PMLR, 2014.
- Online learning via sequential complexities. J. Mach. Learn. Res., 16(1):155–186, 2015a.
- Sequential complexities and uniform martingale laws of large numbers. Probability theory and related fields, 161:111–153, 2015b.
- Relax and randomize: From value to algorithms. Advances in Neural Information Processing Systems, 25, 2012.
- II: Fourier analysis, self-adjointness, volume 2. Elsevier, 1975.
- Generalized deep image to image regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5609–5619, 2017.
- Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, 2014.
- On the universality of online mirror descent. Advances in neural information processing systems, 24, 2011.
- Convex games in banach spaces. In COLT, pages 1–13. Citeseer, 2010.
- Learning schatten–von neumann operators. arXiv preprint arXiv:1901.10076, 2019.
- Albert Tarantola. Inverse problem theory and methods for model parameter estimation. SIAM, 2005.
- Gunther Uhlmann. Inside out: inverse problems and applications, volume 47. Cambridge University Press, 2003.
- Volodya Vovk. Competitive on-line statistics. International Statistical Review, 69(2):213–248, 2001.
- Functional data analysis. Annual Review of Statistics and its application, 3:257–295, 2016.
- Joachim Weidmann. Linear operators in Hilbert spaces, volume 68. Springer Science & Business Media, 2012.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.