Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Let data talk: data-regularized operator learning theory for inverse problems (2310.09854v2)

Published 15 Oct 2023 in math.NA and cs.NA

Abstract: Regularization plays a pivotal role in integrating prior information into inverse problems. While many deep learning methods have been proposed to solve inverse problems, determining where to apply regularization remains a crucial consideration. Typical methods regularize neural networks via architecture, wherein neural network functions parametrize the parameter of interest or the regularization term. We introduce a novel approach, denoted as the "data-regularized operator learning" (DaROL) method, designed to address PDE inverse problems. The DaROL method trains a neural network on data, regularized through common techniques such as Tikhonov variational methods and Bayesian inference. The DaROL method offers flexibility across different frameworks, faster inverse problem-solving, and a simpler structure that separates regularization and neural network training. We demonstrate that training a neural network on the regularized data is equivalent to supervised learning for a regularized inverse map. Furthermore, we provide sufficient conditions for the smoothness of such a regularized inverse map and estimate the learning error in terms of neural network size and the number of training samples.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Learning the optimal tikhonov regularizer for inverse problems. Advances in Neural Information Processing Systems, 34:25205–25216, 2021.
  2. Giovanni Alessandrini. Stable determination of conductivity by boundary measurements. Applicable Analysis, 27(1-3):153–172, 1988.
  3. Solving inverse problems using data-driven models. Acta Numerica, 28:1–174, 2019.
  4. Random walks in frequency and the reconstruction of obstacles with cavities from multi-frequency data. arXiv e-prints, pages arXiv–2308, 2023.
  5. Unsupervised deep learning algorithm for pde-based forward and inverse problems. arXiv preprint arXiv:1904.05417, 2019.
  6. Neural network augmented inverse problems for pdes. arXiv preprint arXiv:1712.09685, 2017.
  7. Neural networks as smooth priors for inverse problems for pdes. Journal of Computational Mathematics and Data Science, 1:100008, 2021.
  8. Lasso reloaded: a variational analysis perspective with applications to compressed sensing. arXiv preprint arXiv:2205.06872, 2022.
  9. Square root f{{\{{LASSO}}\}}: well-posedness, lipschitz stability and the tuning trade off. arXiv preprint arXiv:2303.15588, 2023.
  10. Large-scale inverse problems and quantification of uncertainty. John Wiley & Sons, 2011.
  11. Optimization problems with perturbations: A guided tour. SIAM review, 40(2):228–264, 1998.
  12. Message passing neural pde solvers. arXiv preprint arXiv:2202.03376, 2022.
  13. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  14. Inverse problems: From regularization to bayesian inference. Wiley Interdisciplinary Reviews: Computational Statistics, 10(3):e1427, 2018.
  15. Decoding by linear programming. IEEE transactions on information theory, 51(12):4203–4215, 2005.
  16. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8):1207–1223, 2006.
  17. Guy Chavent. Nonlinear least squares for inverse problems: theoretical foundations and step-by-step guide for applications. Springer Science & Business Media, 2010.
  18. Friedrichs learning: Weak solutions of partial differential equations via deep learning. SIAM Journal on Scientific Computing, 45(3):A1271–A1299, 2023.
  19. Online learning in optical tomography: a stochastic approach. Inverse Problems, 34(7):075010, 2018.
  20. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA, 11(4):1203–1253, 2022.
  21. Projected stein variational newton: A fast and scalable bayesian inference method in high dimensions. Advances in Neural Information Processing Systems, 32, 2019.
  22. Learning regularization parameters for general-form tikhonov. Inverse Problems, 33(7):074004, 2017.
  23. The bayesian approach to inverse problems. Handbook of Uncertainty Quantification, pages 311–428, 2017.
  24. Coupling deep learning with full waveform inversion. arXiv preprint arXiv:2203.01799, 2022.
  25. Regularization by architecture: A deep prior approach for inverse problems. Journal of Mathematical Imaging and Vision, 62:456–470, 2020.
  26. David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289–1306, 2006.
  27. Solving traveltime tomography with deep learning. Communications in Mathematics and Statistics, 11(1):3–19, 2023.
  28. Size-independent sample complexity of neural networks. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 297–299. PMLR, 06–09 Jul 2018. URL https://proceedings.mlr.press/v75/golowich18a.html.
  29. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645–6649. Ieee, 2013.
  30. A distribution-free theory of nonparametric regression, volume 1. Springer, 2002.
  31. Regularization of inverse problems by neural networks. In Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision, pages 1065–1093. Springer, 2023.
  32. Regularization of inverse problems by neural networks. arXiv preprint arXiv:2006.03972, 2020.
  33. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  34. Model-based learning for accelerated, limited-view 3-d photoacoustic tomography. IEEE transactions on medical imaging, 37(6):1382–1393, 2018.
  35. Reinhard Heckel. Regularizing linear inverse problems with convolutional neural networks. arXiv preprint arXiv:1907.03100, 2019.
  36. Modulus of continuity for conditionally stable ill-posed problems in hilbert space. Journal of Inverse & Ill-Posed Problems, 16(6), 2008.
  37. Victor Isakov. Stability estimates for obstacles in inverse scattering. Journal of computational and applied mathematics, 42(1):79–88, 1992.
  38. Inverse problems: Tikhonov theory and algorithms, volume 22. World Scientific, 2014.
  39. Reinforced inverse scattering. arXiv preprint arXiv:2206.04186, 2022.
  40. Deep convolutional neural network for inverse problems in imaging. IEEE transactions on image processing, 26(9):4509–4522, 2017.
  41. Statistical and computational inverse problems, volume 160. Springer Science & Business Media, 2006.
  42. Total deep variation: A stable regularization method for inverse problems. IEEE transactions on pattern analysis and machine intelligence, 44(12):9163–9180, 2021.
  43. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  44. Probability in Banach Spaces: isoperimetry and processes, volume 23. Springer Science & Business Media, 1991.
  45. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. CoRR, abs/1910.03193, 2019. URL http://arxiv.org/abs/1910.03193.
  46. Peter Maass. Deep learning for trivial inverse problems. In Compressed Sensing and Its Applications: Third International MATHEON Conference 2017, pages 195–209. Springer, 2019.
  47. Neural inverse operators for solving pde inverse problems. arXiv preprint arXiv:2301.11167, 2023.
  48. Dias: A data-informed active subspace regularization framework for inverse problems. Computation, 10(3):38, 2022.
  49. Tnet: A model-constrained tikhonov network approach for inverse problems. arXiv preprint arXiv:2105.12033, 2021.
  50. Deep synthesis regularization of inverse problems. arXiv preprint arXiv:2002.00155, 2020.
  51. Integral autoencoder network for discretization-invariant learning. The Journal of Machine Learning Research, 23(1):12996–13040, 2022.
  52. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39–56, 2020.
  53. Data-driven forward-inverse problems for yajima–oikawa system using deep learning with parameter regularization. Communications in Nonlinear Science and Numerical Simulation, 118:107051, 2023.
  54. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  55. Linear inversion of band-limited reflection seismograms. SIAM journal on scientific and statistical computing, 7(4):1307–1330, 1986.
  56. Deep network approximation characterized by number of neurons. arXiv preprint arXiv:1906.05497, 2019.
  57. Deep network with approximation error being reciprocal of width to power of square root of depth. Neural Computation, 33(4):1005–1036, 2021.
  58. Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996.
  59. Low complexity regularization of linear inverse problems. Sampling Theory, a Renaissance: Compressive Sensing and Other Developments, pages 103–153, 2015.
  60. The degrees of freedom of partly smooth regularizers. Annals of the Institute of Statistical Mathematics, 69:791–832, 2017.
  61. Data-informed regularization for inverse and imaging problems. In Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision, pages 1–38. Springer, 2021.
  62. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523, 2019.
  63. Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. In Conference on learning theory, pages 639–649. PMLR, 2018.
  64. Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now