Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
Gemini 2.5 Pro Pro
o3 Pro
GPT-4.1 Pro
DeepSeek R1 via Azure Pro
2000 character limit reached

CS4ML: A general framework for active learning with arbitrary data based on Christoffel functions (2306.00945v2)

Published 1 Jun 2023 in cs.LG, cs.NA, and math.NA

Abstract: We introduce a general framework for active learning in regression problems. Our framework extends the standard setup by allowing for general types of data, rather than merely pointwise samples of the target function. This generalization covers many cases of practical interest, such as data acquired in transform domains (e.g., Fourier data), vector-valued data (e.g., gradient-augmented data), data acquired along continuous curves, and, multimodal data (i.e., combinations of different types of measurements). Our framework considers random sampling according to a finite number of sampling measures and arbitrary nonlinear approximation spaces (model classes). We introduce the concept of generalized Christoffel functions and show how these can be used to optimize the sampling measures. We prove that this leads to near-optimal sample complexity in various important cases. This paper focuses on applications in scientific computing, where active learning is often desirable, since it is usually expensive to generate data. We demonstrate the efficacy of our framework for gradient-augmented learning with polynomials, Magnetic Resonance Imaging (MRI) using generative models and adaptive sampling for solving PDEs using Physics-Informed Neural Networks (PINNs).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (120)
  1. Sparse Polynomial Approximation of High-Dimensional Functions. Comput. Sci. Eng. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2022.
  2. B. Adcock and J. M. Cardenas. Near-optimal sampling strategies for multivariate function approximation on general domains. SIAM J. Math. Data Sci., 2(3):607–630, 2020.
  3. CAS4DL: Christoffel adaptive sampling for function approximation via deep learning. Sampl. Theory Signal Process. Data Anal., 20(21):1–29, 2022.
  4. Towards optimal sampling for learning sparse approximation in high dimensions. In A. Nikeghbali, P. Pardalos, A. Raigorodskii, and T. M. Rassias, editors, High Dimensional Optimization and Probability, volume 191. Springer Optim. Appl., 2022.
  5. B. Adcock and A. C. Hansen. Compressive Imaging: Structure, Sampling, Learning. Cambridge University Press, Cambridge, UK, 2021.
  6. Breaking the coherence barrier: a new theory for compressed sensing. Forum Math. Sigma, 5:e4, 2017.
  7. MoDL: Model-based deep learning architecture for inverse problems. IEEE transactions on medical imaging, 38(2):394–405, 2018.
  8. A. Alaoui and M. W. Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  9. The estimation of functional uncertainty using polynomial chaos and adjoint equations. Internat. J. Numer. Methods Fluids, 67(3):328–341, 2011.
  10. ADLGM: An efficient adaptive sampling deep learning Galerkin method. J. Comput. Phys, page 111944, 2023.
  11. Random Fourier features for kernel ridge regression: approximation bounds and statistical guarantees. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 253–262. PMLR, 2017.
  12. A Universal Sampling Method for Reconstructing Signals with Simple Fourier Transforms. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, pages 1051–1063, New York, NY, USA, 2019. Association for Computing Machinery.
  13. Foundations of Image Science. Wiley–Interscience, Hoboken, NJ, 2004.
  14. Spectral and finite difference solutions of the Burgers equation. Comput. Fluids, 14(1):23–41, 1986.
  15. A. Berk. Deep generative demixing: Error bounds for demixing subgaussian mixtures of Lipschitz signals. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4010–4014. IEEE, 2021.
  16. A coherence parameter characterizing generative compressed sensing with Fourier measurements. IEEE J. Sel. Areas Inf. Theory, 3(3):502–512, 2023.
  17. Model reduction and neural networks for parametric PDEs. J. Comput. Math., 7:121–157, 2021.
  18. An analysis of block sampling strategies in compressed sensing. IEEE Trans. Inform. Theory, 62(4):2125–2139, 2016.
  19. J. Blechschmidt and O. G. Ernst. Three ways to solve partial differential equations with neural networks—a review. GAMM-Mitteilungen, 44(2):e202100006, 2021.
  20. Compressed sensing using generative models. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 537–546. PMLR, 06–11 Aug 2017.
  21. C. A. Bouman. Foundations of Computational Imaging: A Model-Based Approach. SIAM, Philadelphia, PA, 2022.
  22. An algorithm for variable density sampling with block-constrained acquisition. SIAM J. Imaging Sci., 7(2):1080–1107, 2014.
  23. Sparse recovery in bounded Riesz systems with applications to numerical methods for PDEs. Appl. Comput. Harmon. Anal., 53:231–269, 2021.
  24. Data-Driven Science and Engineering. Cambridge University Press, 2 edition, 2022.
  25. Machine learning for partial differential equations. CoRR, abs/2303.17078, 2023.
  26. E. J. Candès and Y. Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Trans. Inform. Theory, 57(4):2342–2359, 2011.
  27. E. J. Candès and B. Recht. Exact matrix completion via convex optimization. Found. Comput. Math., 9(6):717–772, 2009.
  28. Spectral methods: Fundamentals in Single Domains. Springer, 2006.
  29. S. Chatterjee and A. S. Hadi. Influential observations, high leverage points, and outliers in linear regression. Statist. Sci., 1(3):379–415, 1986.
  30. A statistical perspective of sampling scores for linear regression. In 2016 IEEE International Symposium on Information Theory (ISIT), pages 1556–1560, 2016.
  31. Adaptive trajectories sampling for solving PDEs with deep learning methods. arXiv:2303.15704, 2023.
  32. X. Chen and E. Price. Active regression via linear-sample sparsification. In A. Beygelzimer and D. Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 663–695. PMLR, 2019.
  33. I.-Y. Chun and B. Adcock. Compressed sensing and parallel acquisition. IEEE Trans. Inform. Theory, 63(8):4860–4882, 2017.
  34. A. Cohen and G. Migliorati. Optimal weighted least-squares methods. SMAI J. Comput. Math., 3:181–203, 2017.
  35. Robust training and initialization of deep neural networks: An adaptive basis viewpoint. In Jianfeng Lu and Rachel Ward, editors, Proceedings of The First Mathematical and Scientific Machine Learning Conference, volume 107 of Proceedings of Machine Learning Research, pages 512–536, Princeton University, Princeton, NJ, USA, 2020. PMLR.
  36. Sobolev training for neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  37. Hyperbolic Cross Approximation. Adv. Courses Math. CRM Barcelona. Birkhäuser, Basel, Switzerland, 2018.
  38. M. A. Davenport and J. Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE J. Sel. Topics Signal Process., 10(4):602–622, 2016.
  39. T. De Ryck and S. Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  40. Structure preserving compressive sensing MRI reconstruction using generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 522–523, 2020.
  41. Leveraged volume sampling for linear regression. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  42. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Commun. Math. Stat., 5(4):349–380, 2017.
  43. Convergence bounds for empirical nonlinear least-squares. ESAIM Math. Model. Numer. Anal., 56(1):79–104, 2022.
  44. Multifidelity uncertainty quantification using spectral stochastic discrepancy models. In Roger Ghanem, David Higdon, and Houman Owhadi, editors, Handbook of Uncertainty Quantification, pages 991–1036. Springer, Cham, Switzerland, 2017.
  45. C. L. Epstein. Introduction to the Mathematics of Medical Imaging. Other Titles in Applied Mathematics. Society for Industrial and Applied Mathematics, 2nd edition, 2007.
  46. Fourier sparse leverage scores and approximate kernel learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 109–122. Curran Associates, Inc., 2020.
  47. Nyström landmark sampling and regularized Christoffel functions. Mach. Learn., 111:2213–2254, 2022.
  48. S. Foucart and H. Rauhut. A Mathematical Introduction to Compressive Sensing. Appl. Numer. Harmon. Anal. Birkhäuser, New York, NY, 2013.
  49. Provable active learning of neural networks for parametric PDEs. In The Symbiosis of Deep Learning and Differential Equations II, 2022.
  50. Active Learning for Single Neuron Models with Lipschitz Non-Linearities. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 4101–4113. PMLR, 2023.
  51. Failure-informed adaptive sampling for PINNs, part II: combining with re-sampling and subset simulation. arXiv:2302.01529, 2023.
  52. Failure-informed adaptive sampling for pinns. SIAM Journal on Scientific Computing, 45(4):A1971–A1994, 2023.
  53. Numerical solution of the parametric diffusion equation by deep neural networks. J. Sci. Comput., 88:22, 2021.
  54. X. Geng and L. Zeng. Gradient-enhanced deep neural network approximations. J. Mach. Learn. Model. Comput., 3(4):73–91, 2022.
  55. Learning-based compressive MRI. IEEE Trans. Med. Imag., 37(6):1394–1406, 2018.
  56. A gradient enhanced ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-minimization for sparse approximation of polynomial chaos expansions. J. Comput. Phys., 367:49–64, 2018.
  57. Constructing least-squares polynomial approximations. SIAM Rev., 62(2):483–508, 2020.
  58. Boosted optimal weighted least-squares. Math. Comp., 91:1281–1315, 2022.
  59. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med., 79(6):3055–3071, 2018.
  60. J. Hampton and A. Doostan. Coherence motivated sampling and convergence analysis of least squares polynomial chaos regression. Comput. Methods Appl. Mech. Engrg., 290:73–97, 2015.
  61. Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. U.S.A., 115(34):8505–8510, 2018.
  62. B. Hanin. Which neural net architectures give rise to exploding and vanishing gradients? Advances in Neural Information Processing Systems, pages 582–591, 2018.
  63. B. Hanin and D. Rolnick. How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems, pages 571–581. 2018.
  64. Generalization bounds for sparse random feature expansions. Applied and Computational Harmonic Analysis, 62:310–330, 2023.
  65. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  66. A neural multilevel method for high-dimensional parametric PDEs. In Advances in Neural Information Processing Systems, 2021.
  67. Robust Compressed Sensing MRI with Deep Generative Priors. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14938–14954. Curran Associates, Inc., 2021.
  68. Instance-optimal compressed sensing via posterior sampling. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4709–4720. PMLR, 18–24 Jul 2021.
  69. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. J. Comput. Phys., 426:109951, 2021.
  70. neuronets/nobrainer: 0.4.0, October 2022.
  71. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  72. D. P. Kingma and J. Ba. Adam: a method for stochastic optimization. arXiv:1412.6980, 2017.
  73. Neural operator: Learning maps between function spaces with applications to PDEs. J. Mach. Learn. Res., 24:1–97, 2023.
  74. F. Krahmer and R. Ward. Stable and robust sampling strategies for compressive imaging. IEEE Trans. Image Process., 23(2):612–622, 2013.
  75. A theoretical analysis of deep neural networks and parametric PDEs. Constr. Approx., 55:73–125, 2022.
  76. Parallel Magnetic Resonance Imaging. Phys. Med. Biol., 52(7):R15, 2007.
  77. J. B. Lasserre and E. Pauwels. The empirical Christoffel function with applications in data analysis. Adv. Comput. Math., 45:1439–1468, 2019.
  78. Orhogonal bases for polynomial regression with derivative information in uncertainty quantification. Int. J. Uncertain. Quantif., 1(4):297–320, 2011.
  79. Deep Magnetic Resonance image reconstruction: inverse problems meet neural networks. IEEE Signal Process. Mag., 37(1):141–151, 2020.
  80. Z. Liang and P. C. Lauterbur. Principles of Magnetic Resonance Imaging: A Signal Processing Perspective. IEEE Press Series on Biomedical Engineering. Wiley–IEEE Press, New York, 2000.
  81. J. Liu and Z. Liu. Non-iterative recovery from nonlinear observations using generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 233–243, 2022.
  82. B. Lockwood and D. Mavriplis. Gradient-based methods for uncertainty quantification in hypersonic flows. Comput. & Fluids, 85:27–38, 2013.
  83. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell., 3:218–229, 2021.
  84. Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Signal Process. Mag., 35(1):20–36, 2018.
  85. A statistical perspective on algorithmic leveraging. J. Mach. Learn. Res., 16:861–911, 2015.
  86. Z. Mao and X. Meng. Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving partial differential equations with sharp solutions. Applied Mathematics and Mechanics, 44(7):1069–1084, 2023.
  87. Deep generative adversarial neural networks for compressive sensing MRI. IEEE transactions on medical imaging, 38(1):167–179, 2018.
  88. M. T. McCann and M. Unser. Biomedical image reconstruction: From the foundations to deep neural networks. Foundations and Trends® in Signal Processing, 13(3):283–359, 2019.
  89. MRI: From Picture to Proton. Cambridge University Press, Cambridge, 2nd edition, 2006.
  90. G. Migliorati. Polynomial approximation by means of the random discrete L2superscript𝐿2L^{2}italic_L start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT projection and application to inverse problems for PDEs with stochastic data. PhD thesis, Politecnico di Milano, 2013.
  91. G. Migliorati. Adaptive approximation by optimal weighted least squares methods. SIAM J. Numer. Anal, 57(5):2217–2245, 2019.
  92. G. Migliorati. Multivariate approximation of functions on irregular domains by weighted least-squares methods. IMA J. Numer. Anal., 41(2):1293–1317, 2021.
  93. Convergence estimates in probability and in expectation for discrete least squares with noisy evaluations at random points. J. Multivariate Anal., 142:167–182, 2015.
  94. C. Musco and C. Musco. Recursive Sampling for the Nyström Method. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  95. A. Naderi and Y. Plan. Beyond independent measurements: General compressed sensing with gnn application. In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021.
  96. A. Narayan. Computation of induced orthogonal polynomial distributions. Electron. Trans. Numer. Anal., 50:71–97, 2018.
  97. P. Nevai. Géza Freud, orthogonal polynomials and Christoffel functions. A case study. J. Approx. Theory, 48(1):3–167, 1986.
  98. Derivative-informed neural operator: an efficient framework for high-dimensional parametric derivative learning. arXiv:2206.10745, 2022.
  99. Derivative-informed projected neural networks for high-dimensional parametric maps governed by PDEs. Comput. Methods Appl. Mech. Engrg., 388(1):114199, 2022.
  100. Deep learning techniques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory, 1(1):39–56, 2020.
  101. Generalized leverage scores: Geometric interpretation and applications. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17056–17070. PMLR, 2022.
  102. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. SIAM Rev., 60(3):550–591, 2018.
  103. On polynomial chaos expansion via gradient-enhanced l1subscript𝑙1l_{1}italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-minimization. J. Comput. Phys., 310:440–458, 2016.
  104. C. Poon. On the role of total variation in compressed sensing. SIAM J. Imaging Sci., 8(1):682–720, 2015.
  105. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378:686–707, 2019.
  106. Low rank tensor recovery via iterative hard thresholding. Linear Algebra Appl., 524:220–262, 2017.
  107. Image reconstruction: from sparsity to data-adaptive methods and machine learning. Proc. IEEE, 108(1):86–109, 2020.
  108. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev., 52(3):471–501, 2010.
  109. Data-driven discovery of partial differential equations. Sci. Adv., 3(4), 2017.
  110. Compressed sensing: from research to clinical practice with deep neural networks. IEEE Signal Process. Mag., 31(1):117–127, 2020.
  111. Learning the sampling pattern for MRI. IEEE Trans. Med. Imag., 39(12):4310–4321, 2020.
  112. J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. J. Comput. Phys., 375:1339–1364, 2018.
  113. Das: A deep adaptive sampling method for solving partial differential equations. arXiv preprint arXiv:2112.14038, 2021.
  114. T. Tang and T. Zhou. On discrete least-squares projection in unbounded domain with random evaluations and its application to parametric uncertainty quantification. SIAM J. Sci. Comput., 36(5):A2272–A2295, 2014.
  115. J. A. Tropp. User-friendly tail bounds for sums of random matrices. Found. Comput. Math., 12:389–434, 2012.
  116. M. Uecker. Parallel Magnetic Resonance Imaging. arXiv:1501.06209, 2015.
  117. M. Vidyasagar. An Introduction to Compressed Sensing. Comput. Sci. Eng. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2019.
  118. Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets. Sci. Adv., 7(40):eabi8605, 2021.
  119. D. P. Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science, 10(1-2):1–157, 2014.
  120. Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems. Comput. Methods Appl. Mech. Engrg., 393:114823, 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now