Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems (2206.00934v5)

Published 2 Jun 2022 in math.NA, cs.NA, math.AP, and stat.ML

Abstract: We study the problem of reconstructing solutions of inverse problems when only noisy measurements are available. We assume that the problem can be modeled with an infinite-dimensional forward operator that is not continuously invertible. Then, we restrict this forward operator to finite-dimensional spaces so that the inverse is Lipschitz continuous. For the inverse operator, we demonstrate that there exists a neural network which is a robust-to-noise approximation of the operator. In addition, we show that these neural networks can be learned from appropriately perturbed training data. We demonstrate the admissibility of this approach to a wide range of inverse problems of practical interest. Numerical examples are given that support the theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. G. S. Alberti and M. Santacesaria. Infinite-dimensional inverse problems with finite measurements. Archive for Rational Mechanics and Analysis, 243(1):1–31, 2022.
  2. Continuous generative neural networks. arXiv preprint arXiv:2205.14627, 2022.
  3. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning, pages 173–182. PMLR, 2016.
  4. M. Anthony and P. L. Bartlett. Neural network learning: theoretical foundations. Cambridge University Press, Cambridge, 1999.
  5. On instabilities of deep learning in image reconstruction-does AI come at a cost? arXiv preprint arXiv:1902.05300, 2019.
  6. Unsupervised medical image translation using cycle-MedGAN. In 2019 27th European Signal Processing Conference (EUSIPCO), pages 1–5. IEEE, 2019.
  7. Solving inverse problems using data-driven models. Acta Numerica, 28:1–174, 2019.
  8. V. Bacchelli and S. Vessella. Lipschitz stability for a stationary 2d inverse problem with unknown polygonal boundary. Inverse problems, 22(5):1627, 2006.
  9. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  10. The modern mathematics of deep learning. arXiv preprint arXiv:2105.04026, 2021.
  11. P. J. Bickel and B. Li. Local polynomial regression on unknown manifolds. In Complex datasets and inverse problems, volume 54, pages 177–187. Institute of Mathematical Statistics, 2007.
  12. Limitations of deep learning for inverse problems on digital hardware. arXiv preprint arXiv:2202.13490, 2022.
  13. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
  14. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning, 3(1):1–122, 2011.
  15. Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press, 2022.
  16. Nonparametric regression on low-dimensional manifolds using deep relu networks. arXiv preprint arXiv:1908.01842, 2019.
  17. C. K. Chui and H. Mhaskar. Deep nets for local manifold learning. Frontiers in Applied Mathematics and Statistics, 4:12, 2018.
  18. A. Cloninger and T. Klock. A deep network construction that adapts to intrinsic dimensionality beyond the domain. Neural Networks, 141:404–419, 2021.
  19. The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smale’s 18th problem. Proceedings of the National Academy of Sciences, 119(12):e2107151119, 2022.
  20. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  21. DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing. Technical Report 1809.07669, arXiv, 2018.
  22. Regularization of inverse problems, volume 375. Springer Science & Business Media, 1996.
  23. Solving inverse problems with deep neural networks-robustness included. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  24. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  25. The troublesome kernel: why deep learning for inverse problems is typically unstable. arXiv preprint arXiv:2001.01258, 2020.
  26. F. A. Graybill and G. Marsaglia. Idempotent matrices and quadratic forms in the general linear hypothesis. The Annals of Mathematical Statistics, 28(3):678–686, 1957.
  27. K. Gregor and Y. LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pages 399–406, 2010.
  28. P. Grohs and F. Voigtlaender. Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces. arXiv preprint arXiv:2104.02746, 2021.
  29. Expressivity of deep neural networks. arXiv preprint arXiv:2007.04759, 2020.
  30. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  31. ReLU deep neural networks and linear finite elements. arXiv preprint arXiv:1807.03973, 2018.
  32. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  33. From graphs to manifolds–weak and strong pointwise consistency of graph laplacians. In International Conference on Computational Learning Theory, pages 470–485. Springer, 2005.
  34. Squeeze-and-excitation networks. pages 7132–7141, 2018.
  35. Some investigations on robustness of deep learning in limited angle tomography. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 145–153. Springer, 2018.
  36. Iterative regularization methods for nonlinear ill-posed problems. In Iterative Regularization Methods for Nonlinear Ill-Posed Problems. de Gruyter, 2008.
  37. A. Kirsch. An introduction to the mathematical theory of inverse problems, volume 120. Springer, 2011.
  38. Estimation of a function of low local dimensionality by deep neural networks. arXiv preprint arXiv:1908.11140, 2019.
  39. M. Kontak and V. Michel. A greedy algorithm for nonlinear inverse problems with an application to nonlinear inverse gravimetry. GEM-International Journal on Geomathematics, 9(2):167–198, 2018.
  40. Imagenet classification with deep convolutional neural networks. In Adv. Neural Inf. Process. Syst. 25, pages 1097–1105. Curran Associates, Inc., 2012.
  41. Deep learning. Nature, 521(7553):436–444, 2015.
  42. J. M. Lee. Smooth manifolds. In Introduction to smooth manifolds, pages 1–31. Springer, 2013.
  43. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, 2022.
  44. Inverse problem for coefficient identification in the Euler–Bernoulli equation. Computers & Mathematics with Applications, 56(2):400–410, 2008.
  45. W. McCulloch and W. Pitts. A logical calculus of ideas immanent in nervous activity. Bull. Math. Biophys., 5:115–133, 1943.
  46. E. J. McShane. Extension of range of functions. Bulletin of the American Mathematical Society, 40(12):837–842, 1934.
  47. H. N. Mhaskar. Eignets for function approximation on manifolds. Applied and Computational Harmonic Analysis, 29(1):63–87, 2010.
  48. R. K. Miller. Volterra integral equations in a Banach space. Funkcial. Ekvac, 18(2):163–193, 1975.
  49. J. R. Munkres. Topology, volume 2. Prentice Hall Upper Saddle River, 2000.
  50. R. Nakada and M. Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174):1–38, 2020.
  51. F. Natterer. Regularisierung schlecht gestellter Probleme durch Projektionsverfahren. Numerische Mathematik, 28(3):329–341, 1977.
  52. F. Natterer. The mathematics of computerized tomography. SIAM, 2001.
  53. F. Natterer and F. Wübbeling. Mathematical methods in image reconstruction. SIAM, 2001.
  54. S. Niwas and D.C. Singhal. Aquifer transmissivity of porous media from resistivity data. Journal of Hydrology, 82(1-2):143–153, 1985.
  55. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39–56, 2020.
  56. Deep ReLU networks and high-order finite element methods. Analysis and Applications, 18(05):715–770, 2020.
  57. P. C. Petersen. Neural network theory. University of Vienna, 2020.
  58. P. C. Petersen and F. Voigtlaender. Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 180:296–330, 2018.
  59. R. Plato and G. Vainikko. On the regularization of projection methods for solving ill-posed problems. Numerische Mathematik, 57(1):63–79, 1990.
  60. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
  61. M. Razzaghi and Y. Ordokhani. Solution of nonlinear Volterra-Hammerstein integral equations via rationalized Haar functions. Mathematical Problems in Engineering, 7(2):205–219, 2001.
  62. L. Rondi. A remark on a paper by Alessandrini and Vessella. Advances in Applied Mathematics, 36(1):67–69, 2006.
  63. J. Schmidt-Hieber. Deep ReLU network approximation of functions on a manifold. arXiv preprint arXiv:1908.00695, 2019.
  64. B. Sepehrian and M. Razzaghi. Solution of nonlinear Volterra-Hammerstein integral equations via single-term Walsh series method. Mathematical Problems in engineering, 2005(5):547–554, 2005.
  65. Provable approximation properties for deep neural networks. Applied Computational Harmonic Analysis, 44(3):537–557, 2018.
  66. A. Singer. From graph to manifold Laplacian: The convergence rate. Applied and Computational Harmonic Analysis, 21(1):128–134, 2006.
  67. J. Sirignano and K. Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018.
  68. Deep admm-net for compressive sensing MRI. Advances in neural information processing systems, 29, 2016.
  69. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  70. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pages 945–948. IEEE, 2013.
  71. D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Netw., 94:103–114, 2017.
  72. W. W-G Yeh. Review of parameter identification procedures in groundwater hydrology: The inverse problem. Water Resources Research, 22(2):95–108, 1986.
  73. Image reconstruction by domain-transform manifold learning. Nature, 555(7697):487–492, 2018.
  74. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now