Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processes (2402.11179v1)

Published 17 Feb 2024 in cs.LG, math.ST, physics.comp-ph, and stat.TH

Abstract: The application of neural network models to scientific machine learning tasks has proliferated in recent years. In particular, neural network models have proved to be adept at modeling processes with spatial-temporal complexity. Nevertheless, these highly parameterized models have garnered skepticism in their ability to produce outputs with quantified error bounds over the regimes of interest. Hence there is a need to find uncertainty quantification methods that are suitable for neural networks. In this work we present comparisons of the parametric uncertainty quantification of neural networks modeling complex spatial-temporal processes with Hamiltonian Monte Carlo and Stein variational gradient descent and its projected variant. Specifically we apply these methods to graph convolutional neural network models of evolving systems modeled with recurrent neural network and neural ordinary differential equations architectures. We show that Stein variational inference is a viable alternative to Monte Carlo methods with some clear advantages for complex neural network models. For our exemplars, Stein variational interference gave similar uncertainty profiles through time compared to Hamiltonian Monte Carlo, albeit with generally more generous variance.Projected Stein variational gradient descent also produced similar uncertainty profiles to the non-projected counterpart, but large reductions in the active weight space were confounded by the stability of the neural network predictions and the convoluted likelihood landscape.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence. Technical report, U.S. Department of Energy Office of Science (SC), 2019.
  2. Predicting the mechanical response of oligocrystals with deep learning. Computational Materials Science, 169:109099, 2019.
  3. Prediction of the evolution of the stress field of polycrystals undergoing elastic-plastic deformation with a hybrid neural network model. Machine Learning: Science and Technology, 2020.
  4. Mesh-based graph convolutional neural networks for modeling materials with microstructure. Journal of Machine Learning for Modeling and Computing, 3(1), 2022.
  5. Deep learning and multi-level featurization of graph representations of microstructural data. Computational Mechanics, 72(1):57–75, 2023.
  6. Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1):932–955, 2018.
  7. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  8. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
  9. A survey on physics informed reinforcement learning: Review and open problems. arXiv preprint arXiv:2309.01909, 2023.
  10. Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced kalman filter. Computational Mechanics, 72(1):95–124, 2023.
  11. Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning. Computer Methods in Applied Mechanics and Engineering, 416:116304, 2023.
  12. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018.
  13. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2):742–769, 2018.
  14. An improved analysis of training over-parameterized deep neural networks. Advances in neural information processing systems, 32, 2019.
  15. James O. Berger. Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics. Springer New York, New York, NY, 1985.
  16. D. S. Sivia and J. Skilling. Data Analysis: A Bayesian Tutorial. Oxford University Press, New York, NY, 2nd edition, 2006.
  17. Udo Von Toussaint. Bayesian inference in physics. Reviews of Modern Physics, 83:943–999, 2011.
  18. David J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Computation, 4(3):448–472, 1992.
  19. Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag New York, New York, NY, 1996.
  20. Alex Graves. Practical Variational Inference for Neural Networks. In Advances in Neural Information Processing Systems 24 (NIPS 2011), pages 2348–2356, Granada, Spain, 2011.
  21. Weight Uncertainty in Neural Networks. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1613–1622, 2015.
  22. Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
  23. An Introduction to MCMC for Machine Learning. Machine Learning, 50:5–43, 2003.
  24. Handbook of Markov Chain Monte Carlo. Chapman-Hall/CRC, 2011.
  25. Hybrid Monte Carlo. Physics Letters B, 195(2):216–222, sep 1987.
  26. Qi He and Jack Xin. Hybrid deterministic-stochastic gradient Langevin dynamics for Bayesian learning. Communications in Information and Systems, 12(3):221–232, 2012.
  27. Stochastic Gradient Hamiltonian Monte Carlo. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pages 1683–1691, Beij, 2014.
  28. Variational Hamiltonian Monte Carlo via Score Matching. Bayesian Analysis, 13(2):485–506, jun 2018.
  29. Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017.
  30. Advances in Variational Inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):2008–2026, 2019.
  31. Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm. In Advances in Neural Information Processing Systems 29 (NIPS 2016), pages 2378–2386, Barcelona, Spain, 2016.
  32. Message passing Stein variational gradient descent. In 35th International Conference on Machine Learning, ICML 2018, volume 13, pages 9629–9638, 2018.
  33. Stein variational message passing for continuous graphical models. In 35th International Conference on Machine Learning, ICML 2018, volume 12, pages 8278–8289, 2018.
  34. Projected stein variational gradient descent. In Advances in Neural Information Processing Systems, 2020.
  35. A rotorcraft in-flight ice detection framework using computational aeroacoustics and Bayesian neural networks. Structural and Multidisciplinary Optimization, 66:197, 2023.
  36. Jeremiah Hauth. Advances in Intuitive Priors and Scalable Algorithms for Bayesian Deep Neural Network Models in Scientific Applications. PhD thesis, University of Michigan, 2024.
  37. Phase-field simulations of intergranular fission gas bubble behavior in U3Si2 nuclear fuel. Journal of Nuclear Materials, 541:152415, 2020.
  38. Modeling mesoscale fission gas behavior in uo2 by directly coupling the phase field method to spatially resolved cluster dynamics. Materials Theory, 6(1):7, 2022.
  39. Convolutional lstm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
  40. Bi-directional convlstm u-net with densley connected convolutions. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  41. Self-attention convlstm for spatiotemporal prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 34:07, pages 11531–11538, 2020.
  42. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  43. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  44. Neural ordinary differential equations. In Advances in neural information processing systems, pages 6571–6583, 2018.
  45. Augmented neural odes. Advances in neural information processing systems, 32, 2019.
  46. Solving ordinary differential equations i: Non-stiff problems, volume 8 of springer series in comput. mathematics, 1993.
  47. Solving ordinary differential equations II: Stiff and Differential-Algebraic Problems, volume 375. Springer Berlin Heidelberg New York, 1996.
  48. Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, volume 31. Springer Science & Business Media, 2006.
  49. Michael Betancourt. A conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434, 2017.
  50. Stein variational gradient descent with matrix-valued kernels. Advances in neural information processing systems, 32, 2019.
  51. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.
  52. Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5–30, 2006.
  53. C. Soize and R. Ghanem. Data-driven probability concentration and sampling on manifold. Journal of Computational Physics, 321:242–258, 2016.
  54. Localization models for the plastic response of polycrystalline materials using the material knowledge systems framework. Modelling and Simulation in Materials Science and Engineering, 27(7):074008, 2019.
  55. Prediction of the evolution of the stress field of polycrystals undergoing elastic-plastic deformation with a hybrid neural network model. Machine Learning: Science and Technology, 1(3):035005, 2020.
  56. Sobolev training of thermodynamic-informed neural networks for interpretable elasto-plasticity models with level set hardening. Computer Methods in Applied Mechanics and Engineering, 377:113695, 2021.
  57. Predicting plastic anisotropy using crystal plasticity and bayesian neural network surrogate models. Materials Science and Engineering: A, 833:142472, 2022.
  58. Geometric learning for computational mechanics part ii: Graph embedding for interpretable multiscale plasticity. Computer Methods in Applied Mechanics and Engineering, 404:115768, 2023.
  59. Paul R Dawson. Computational crystal plasticity. International journal of solids and structures, 37(1-2):115–130, 2000.
  60. Overview of constitutive laws, kinematics, homogenization and multiscale methods in crystal plasticity finite-element modeling: Theory, experiments, applications. Acta Materialia, 58(4):1152–1211, 2010.
  61. UF Kocks. Laws for work-hardening and low-temperature creep. Journal of engineering materials and technology, 98(1):76–85, 1976.
  62. Hardening, recovery, and creep in fcc mono-and polycrystals. In Presented at the 4th Intern. Conf. on Strength of Metals and Alloys, Nancy, 30 Aug.-3 Sep. 1976, 1976.
  63. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  64. Gilbert Strang. On the construction and comparison of difference schemes. SIAM journal on numerical analysis, 5(3):506–517, 1968.
  65. Alexandre Joel Chorin. Numerical solution of the Navier-Stokes equations. Mathematics of computation, 22(104):745–762, 1968.
  66. Explicit reversible integrators for extended systems dynamics. Molecular Physics, 87(5):1117–1157, 1996.
  67. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  68. Hamiltonian Monte Carlo with energy conserving subsampling. Journal of machine learning research, 20, 2019.
  69. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008.
  70. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396, 2003.
  71. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35(6):2769, 2007.
  72. Brownian distance covariance. The annals of applied statistics, pages 1236–1265, 2009.
  73. Projected Stein variational gradient descent. Advances in Neural Information Processing Systems, 33:1947–1958, 2020.
  74. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
  75. Graph neural networks in tensorflow and keras with spektral [application notes]. IEEE Computational Intelligence Magazine, 16(1):99–106, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Jeremiah Hauth (1 paper)
  2. Cosmin Safta (24 papers)
  3. Xun Huan (30 papers)
  4. Ravi G. Patel (12 papers)
  5. Reese E. Jones (23 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now