Papers
Topics
Authors
Recent
Search
2000 character limit reached

Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers

Published 16 Jun 2023 in cs.LG | (2306.09863v1)

Abstract: Recent work has shown that renormalisation group theory is a useful framework with which to describe the process of pruning neural networks via iterative magnitude pruning. This report formally describes the link between RG theory and IMP and extends previous results around the Lottery Ticket Hypothesis and Elastic Lottery Hypothesis to Hamiltonian Neural Networks for solving differential equations. We find lottery tickets for two Hamiltonian Neural Networks and demonstrate transferability between the two systems, with accuracy being dependent on integration times. The universality of the two systems is then analysed using tools from an RG perspective.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, jan 2015. doi: 10.1016/j.neunet.2014.09.003.
  2. Big neural networks waste capacity, 2013.
  3. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018. doi: 10.48550/ARXIV.1803.03635. URL https://arxiv.org/abs/1803.03635.
  4. Methods for pruning deep neural networks, 2021.
  5. Snip: Single-shot network pruning based on connection sensitivity, 2019.
  6. Hessian-aware pruning and optimal neural implant, 2021.
  7. Robust pruning at initialization, 2021.
  8. Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey. https://arxiv.org/abs/2005.04275, 2020.
  9. Dimensionality reduced training by pruning and freezing parts of a deep neural network: a survey. Artificial Intelligence Review, may 2023. doi: 10.1007/s10462-023-10489-1.
  10. The state of sparsity in deep neural networks, 2019. URL https://arxiv.org/abs/1902.09574.
  11. The lottery ticket hypothesis for pre-trained bert networks, 2020.
  12. Considering layerwise importance in the lottery ticket hypothesis, 2023.
  13. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models, 2021a.
  14. Are sixteen heads really better than one?, 2019.
  15. On the existence of universal lottery tickets, 2022.
  16. Playing lottery tickets in style transfer models, 2022.
  17. Rahul Mehta. Sparse transfer learning via winning lottery tickets, 2019.
  18. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers, 2019.
  19. The elastic lottery ticket hypothesis, 2021b.
  20. Universality of winning tickets: A renormalization group perspective. 2021. doi: 10.48550/ARXIV.2110.03210. URL https://arxiv.org/abs/2110.03210.
  21. Nigel Goldenfeld. Lectures on Phase Transitions and the Renormalization Group. Frontiers in Physics. Westview Press, 1992. ISBN 9780201554090.
  22. On the predictability of pruning across scales, 2020. URL https://arxiv.org/abs/2006.10621.
  23. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics, 5(4):349–380, nov 2017. doi: 10.1007/s40304-017-0117-6. URL https://doi.org/10.1007%2Fs40304-017-0117-6.
  24. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2018.08.029.
  25. dnnsolve: an efficient nn-based pde solver, 2021.
  26. Adaptive deep neural networks methods for high-dimensional partial differential equations. Journal of Computational Physics, 463:111232, 2022. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2022.111232. URL https://www.sciencedirect.com/science/article/pii/S0021999122002947.
  27. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 11 2018. ISSN 0021-9991. doi: 10.1016/j.jcp.2018.10.045.
  28. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, mar 2021. doi: 10.1038/s42256-021-00302-5.
  29. Shaper: Can you hear the shape of a jet?, 2023.
  30. Solving differential equations using deep neural networks. Neurocomputing, 399:193–212, 2020. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2020.02.015. URL https://www.sciencedirect.com/science/article/pii/S0925231220301909.
  31. Deep euler method: solving odes by approximating the local truncation error of the euler method, 2020.
  32. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, mar 2016. doi: 10.1073/pnas.1517384113.
  33. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987–1000, 1998. doi: 10.1109/72.712178.
  34. Akshunna S. Dogra. Dynamical systems and neural networks, 2020.
  35. Hamiltonian neural networks for solving equations of motion. Physical Review E, 105(6), jun 2022. doi: 10.1103/physreve.105.065305. URL https://doi.org/10.1103%2Fphysreve.105.065305.
  36. Hamiltonian neural networks, 2019.
  37. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28–41, 2018. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2018.06.056. URL https://www.sciencedirect.com/science/article/pii/S092523121830794X.
  38. Kenneth G. Wilson. Renormalization group and critical phenomena. i. renormalization group and the kadanoff scaling picture. Phys. Rev. B, 4:3174–3183, Nov 1971. doi: 10.1103/PhysRevB.4.3174.
  39. Kevin Costello. Renormalization and Effective Field Theory, volume 170 of Mathematical Surveys and Monographs. American Mathematical Society, 2011. ISBN 9781470413972.
  40. Ben Simons. Phase transitions and collective phenomena lecture notes, 1997.
  41. E Sullivan. Universality caused: the case of renormalization group explanation. 9, 2019. URL https://doi.org/10.1007/s13194-019-0260-x.
  42. A comprehensive survey on transfer learning, 2020.
  43. Optimizing neural networks via koopman operator theory. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
  44. Lottery tickets in linear models: An analysis of iterative magnitude pruning, 2021.
  45. Towards understanding iterative magnitude pruning: Why lottery tickets win, 2021.
  46. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, aug 2018. doi: 10.1073/pnas.1718942115.
  47. A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations. Computer Methods in Applied Mechanics and Engineering, 390:114474, feb 2022. doi: 10.1016/j.cma.2021.114474.
  48. Space-time deep neural network approximations for high-dimensional partial differential equations, 2020.
  49. John R. Taylor. Classical Mechanics. University Science Books Mill Valley, 2005.
  50. https://github.com/MathePhysics/M4R_23_epg.
  51. Neural network differential equation solvers allow unsupervised error analysis and correction. arXiv, 2023.
  52. A. S. Dogra. v tangent kernels. arXiv, 2023.
  53. On the predictability of pruning across scales. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9075–9083. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/rosenfeld21a.html.
  54. The applicability of the third integral of motion: Some numerical experiments. 69:73, February 1964. doi: 10.1086/109234.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up