Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The autoregressive neural network architecture of the Boltzmann distribution of pairwise interacting spins systems (2302.08347v3)

Published 16 Feb 2023 in cond-mat.dis-nn, cond-mat.stat-mech, cs.LG, and stat.ML

Abstract: Generative Autoregressive Neural Networks (ARNNs) have recently demonstrated exceptional results in image and language generation tasks, contributing to the growing popularity of generative models in both scientific and commercial applications. This work presents an exact mapping of the Boltzmann distribution of binary pairwise interacting systems into autoregressive form. The resulting ARNN architecture has weights and biases of its first layer corresponding to the Hamiltonian's couplings and external fields, featuring widely used structures such as the residual connections and a recurrent architecture with clear physical meanings. Moreover, its architecture's explicit formulation enables the use of statistical physics techniques to derive new ARNNs for specific systems. As examples, new effective ARNN architectures are derived from two well-known mean-field systems, the Curie-Weiss and Sherrington-Kirkpatrick models, showing superior performance in approximating the Boltzmann distributions of the corresponding physics model compared to other commonly used architectures. The connection established between the physics of the system and the neural network architecture provides a means to derive new architectures for different interacting systems and interpret existing ones from a physical perspective.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities., Proceedings of the National Academy of Sciences 79, 2554 (1982), https://www.pnas.org/doi/pdf/10.1073/pnas.79.8.2554 .
  2. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Spin-glass models of neural networks, Phys. Rev. A 32, 1007 (1985a).
  3. Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature 521, 436 (2015).
  4. G. Carleo and M. Troyer, Solving the quantum many-body problem with artificial neural networks, Science 355, 602 (2017), https://www.science.org/doi/pdf/10.1126/science.aag2302 .
  5. E. P. L. van Nieuwenburg, Y.-H. Liu, and S. D. Huber, Learning phase transitions by confusion, Nature Physics 13, 435 (2017).
  6. J. Carrasquilla and R. G. Melko, Machine learning phases of matter, Nature Physics 13, 431 (2017).
  7. D. Wu, L. Wang, and P. Zhang, Solving Statistical Mechanics Using Variational Autoregressive Networks, Physical Review Letters 122, 1 (2019), 1809.10606 .
  8. L. Zdeborová and F. Krzakala, Statistical physics of inference: thresholds and algorithms, Advances in Physics 65, 453 (2016), https://doi.org/10.1080/00018732.2016.1211393 .
  9. H. C. Nguyen, R. Zecchina, and J. Berg, Inverse statistical problems: from the inverse ising problem to data science, Advances in Physics, Advances in Physics 66, 197 (2017).
  10. C. Nash and C. Durkan, Autoregressive energy machines, in Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Salakhutdinov (PMLR, 2019) pp. 1735–1744.
  11. D. Wu, R. Rossi, and G. Carleo, Unbiased monte carlo cluster updates with autoregressive neural networks, Phys. Rev. Res. 3, L042024 (2021).
  12. Z. Wang and E. J. Davis, Calculating rényi entropies with neural autoregressive quantum states, Phys. Rev. A 102, 062413 (2020).
  13. T. D. Barrett, A. Malyshev, and A. I. Lvovsky, Autoregressive neural-network wavefunctions for ab initio quantum chemistry, Nature Machine Intelligence 4, 351 (2022).
  14. E. M. Inack, S. Morawetz, and R. G. Melko, Neural annealing and visualization of autoregressive neural networks in the newman-moore model, Condensed Matter 7, 10.3390/condmat7020038 (2022).
  15. L. P. Kadanoff, Statistical physics: statics, dynamics and renormalization (World Scientific, 2000).
  16. D. Sherrington and S. Kirkpatrick, Solvable model of a spin-glass, Phys. Rev. Lett. 35, 1792 (1975).
  17. T. nobel committee for Physics, For groundbreaking contributions to our understanding of complex physical systems. [nobel to g. parisi] (2021).
  18. G. Parisi, Toward a mean field theory for spin glasses, Physics Letters A 73, 203 (1979a).
  19. G. Parisi, Infinite number of order parameters for spin-glasses, Phys. Rev. Lett. 43, 1754 (1979b).
  20. E. Gardner, Maximum storage capacity in neural networks, Europhysics Letters 4, 481 (1987).
  21. D. J. Amit, H. Gutfreund, and H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55, 1530 (1985b).
  22. M. Mézard, G. Parisi, and R. Zecchina, Analytic and algorithmic solution of random satisfiability problems, Science , 812 (2002), https://www.science.org/doi/pdf/10.1126/science.1073287 .
  23. G. Parisi and F. Zamponi, Mean-field theory of hard sphere glasses and jamming, Rev. Mod. Phys. 82, 789 (2010).
  24. Z. C. Lipton, J. Berkowitz, and C. Elkan, A critical review of recurrent neural networks for sequence learning (2015).
  25. M. Mezard, G. Parisi, and M. Virasoro, Spin Glass Theory and Beyond (1986).
  26. H. Nishimori, Statistical physics of spin glasses and information processing: an introduction (Clarendon Press, 2001).
  27. M. Talagrand, The parisi formula, Annals of Mathematics 163, 221 (2006).
  28. G. Parisi, A sequence of approximated solutions to the s-k model for spin glasses, Journal of Physics A: Mathematical and General 13, L115 (1980).
  29. D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980  (2014).
  30. I. Biazzo, h2arnn, http://github.com/ocadni/h2arnn (2023), [Online; accessed 9-march-2023].
  31. A. P. Young, Direct determination of the probability distribution for the spin-glass order parameter, Phys. Rev. Lett. 51, 1206 (1983).
Citations (5)

Summary

We haven't generated a summary for this paper yet.