Papers
Topics
Authors
Recent
Search
2000 character limit reached

Score Operator Newton transport

Published 16 May 2023 in math.ST, cs.LG, cs.NA, math.NA, and stat.TH | (2305.09792v3)

Abstract: We propose a new approach for sampling and Bayesian computation that uses the score of the target distribution to construct a transport from a given reference distribution to the target. Our approach is an infinite-dimensional Newton method, involving a linear PDE, for finding a zero of a ``score-residual'' operator. We prove sufficient conditions for convergence to a valid transport map. Our Newton iterates can be computed by exploiting fast solvers for elliptic PDEs, resulting in new algorithms for Bayesian inference and other sampling tasks. We identify elementary settings where score-operator Newton transport achieves fast convergence while avoiding mode collapse.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
  2. A Nash-Moser approach to KAM theory. Hamiltonian partial differential equations and applications, pages 255–284, 2015.
  3. Learning elliptic partial differential equations with randomized linear algebra. Foundations of Computational Mathematics, 23(2):709–739, 2023.
  4. Nisha Chandramoorthy. https://doi.org/10.5281/zenodo.10307262, December 2023. URL https://doi.org/10.5281/zenodo.10307262.
  5. Efficient computation of linear response of chaotic attractors with one-dimensional unstable manifolds. SIAM Journal on Applied Dynamical Systems, 21(2):735–781, 2022. doi:10.1137/21M1405599. URL https://doi.org/10.1137/21M1405599.
  6. Projected Stein variational gradient descent. Advances in Neural Information Processing Systems, 33:1947–1958, 2020.
  7. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  8. SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence. Advances in Neural Information Processing Systems, 33:2098–2109, 2020.
  9. Philippe G Ciarlet. The finite element method for elliptic problems. SIAM, 2002.
  10. Vortex methods: theory and practice, volume 8. Cambridge university press Cambridge, 2000.
  11. Tensor-sparsity of solutions to high-dimensional elliptic partial differential equations. Found Comput Math, 16:813–874, 2016. doi:10.1007/s10208-015-9265-9.
  12. A Stein variational Newton method. Advances in Neural Information Processing Systems, 31, 2018.
  13. On the geometry of Stein variational gradient descent. arXiv preprint arXiv:1912.00894, 2019.
  14. Semyon Dyatlov. Lecture notes for 18.155: distributions, elliptic regularity, and applications to pdes. https://math.mit.edu/~dyatlov/18.155/155-notes.pdf, 2022. [Online; accessed Jan 20th, 2023].
  15. Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Society, 2022.
  16. A. Galántai. The theory of newton’s method. Journal of Computational and Applied Mathematics, 124(1):25–44, 2000. ISSN 0377-0427. doi:https://doi.org/10.1016/S0377-0427(00)00435-0. URL https://www.sciencedirect.com/science/article/pii/S0377042700004350. Numerical Analysis 2000. Vol. IV: Optimization and Nonlinear Equations.
  17. Jürgen Gerlach. Accelerated convergence in newton’s method. Siam Review, 36(2):272–276, 1994.
  18. Elliptic partial differential equations of second order, volume 224. Springer, 1977.
  19. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  20. Stein variational adaptive importance sampling. arXiv preprint arXiv:1704.05201, 2017.
  21. Lars Hörmander. Elliptic boundary problems, pages 242–274. Springer Berlin Heidelberg, Berlin, Heidelberg, 1963. ISBN 978-3-642-46175-0. doi:10.1007/978-3-642-46175-0_10. URL https://doi.org/10.1007/978-3-642-46175-0_10.
  22. Convex potential flows: Universal probability distributions with optimal transport and convex optimization. arXiv preprint arXiv:2012.05942, 2020.
  23. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
  24. Adaptive regularisation for ensemble kalman inversion. Inverse Problems, 37(2):025008, 2021.
  25. The variational formulation of the Fokker–Planck equation. SIAM Journal on Mathematical Analysis, 29(1):1–17, 1998. doi:10.1137/S0036141096303359. URL https://doi.org/10.1137/S0036141096303359.
  26. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12(4):307–392, 2019.
  27. On the limitations of first-order approximation in gan dynamics. In International Conference on Machine Learning, pages 3005–3013. PMLR, 2018.
  28. A stochastic version of Stein variational gradient descent for efficient sampling. Communications in Applied Mathematics and Computational Science, 15(1):37–63, 2020a.
  29. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020b.
  30. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
  31. Machine learning for elliptic pdes: Fast rate generalization bound, neural scaling law and minimax optimality. arXiv preprint arXiv:2110.06897, 2021a.
  32. A priori generalization analysis of the deep ritz method for solving high dimensional elliptic partial differential equations. In Conference on learning theory, pages 3196–3241. PMLR, 2021b.
  33. q-paths: Generalizing the geometric annealing path using power means. In Uncertainty in Artificial Intelligence, pages 1938–1947. PMLR, 2021.
  34. Joseph J Monaghan. Smoothed particle hydrodynamics and its diverse applications. Annual Review of Fluid Mechanics, 44:323–346, 2012.
  35. Jurgen Moser. A new technique for the construction of solutions of nonlinear differential equations. Proceedings of the National Academy of Sciences, 47(11):1824–1831, 1961.
  36. Angxiu Ni. Fast linear response algorithm for differentiating chaos, 2020. URL https://arxiv.org/abs/2009.00595.
  37. Interpolating between bsdes and pinns: deep learning for elliptic and parabolic boundary value problems, 2023.
  38. Operator-Adapted Wavelets, Fast Solvers, and Numerical Homogenization: From a Game Theoretic Approach to Numerical Approximation and Algorithm Design, volume 35. Cambridge University Press, 2019.
  39. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617–2680, 2021.
  40. Mpart: Monotone parameterization toolkit. Journal of Open Source Software, 7(80):4843, 2022.
  41. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  42. Sebastian Reich. A dynamical systems framework for intermittent data assimilation. BIT Numerical Mathematics, 51:235–249, 2011.
  43. Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015.
  44. Kernel techniques: from machine learning to meshless methods. Acta numerica, 15:543–639, 2006.
  45. Sparse recovery of elliptic solvers from matrix-vector products. arXiv preprint arXiv:2110.05351, 2021.
  46. Steve Smale. On the efficiency of algorithms of analysis. Bulletin of the American Mathematical Society, 13(2):87–121, 1985.
  47. Denoising diffusion implicit models. ArXiv, abs/2010.02502, 2020a. URL https://api.semanticscholar.org/CorpusID:222140788.
  48. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  49. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  50. Catastrophic forgetting and mode collapse in gans. In 2020 international joint conference on neural networks (ijcnn), pages 1–10. IEEE, 2020.
  51. JF Traub and H Woźniakowski. Convergence and complexity of interpolatory—newton iteration in a banach space. Computers & Mathematics with Applications, 6(4):385–400, 1980.
  52. HIPPYlib: an extensible software framework for large-scale inverse problems governed by PDEs. ACM Transactions on Mathematical Software (TOMS), 47(2):1–34, 2021.
  53. A weak galerkin finite element method for second-order elliptic problems. Journal of Computational and Applied Mathematics, 241:103–115, 2013.
  54. Holger Wendland. Scattered data approximation, volume 17. Cambridge university press, 2004.
  55. Andre Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem. In Conference on Learning Theory, pages 2093–3027. PMLR, 2018.
  56. Convergence in kl divergence of the inexact Langevin algorithm with application to score-based generative models, 2022. URL https://arxiv.org/abs/2211.01512.
  57. Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018.
  58. Label-free learning of elliptic partial differential equation solvers with generalizability across boundary value problems. Computer Methods in Applied Mechanics and Engineering, page 116214, 2023. ISSN 0045-7825. doi:https://doi.org/10.1016/j.cma.2023.116214. URL https://www.sciencedirect.com/science/article/pii/S0045782523003389.
  59. Meshless methods based on collocation with radial basis functions. Computational mechanics, 26:333–343, 2000.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now