Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 71 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 12 tok/s Pro
GPT-5 High 21 tok/s Pro
GPT-4o 81 tok/s Pro
Kimi K2 231 tok/s Pro
GPT OSS 120B 435 tok/s Pro
Claude Sonnet 4 33 tok/s Pro
2000 character limit reached

Synchronization on circles and spheres with nonlinear interactions (2405.18273v1)

Published 28 May 2024 in math.OC, cs.LG, and math.DS

Abstract: We consider the dynamics of $n$ points on a sphere in $\mathbb{R}d$ ($d \geq 2$) which attract each other according to a function $\varphi$ of their inner products. When $\varphi$ is linear ($\varphi(t) = t$), the points converge to a common value (i.e., synchronize) in various connectivity scenarios: this is part of classical work on Kuramoto oscillator networks. When $\varphi$ is exponential ($\varphi(t) = e{\beta t}$), these dynamics correspond to a limit of how idealized transformers process data, as described by Geshkovski et al. (2024). Accordingly, they ask whether synchronization occurs for exponential $\varphi$. In the context of consensus for multi-agent control, Markdahl et al. (2018) show that for $d \geq 3$ (spheres), if the interaction graph is connected and $\varphi$ is increasing and convex, then the system synchronizes. What is the situation on circles ($d=2$)? First, we show that $\varphi$ being increasing and convex is no longer sufficient. Then we identify a new condition (that the Taylor coefficients of $\varphi'$ are decreasing) under which we do have synchronization on the circle. In so doing, we provide some answers to the open problems posed by Geshkovski et al. (2024).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Expander graphs are globally synchronising. Oct. 2022.
  2. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008. ISBN 978-0-691-13298-3.
  3. The Kuramoto model: A simple paradigm for synchronization phenomena. Reviews of Modern Physics, 77(137):137–185, 2005. doi:10.1103/revmodphys.77.137.
  4. R. Berthier. Incremental learning in diagonal linear networks. Journal of Machine Learning Research, 24(171):1–26, 2023. URL http://jmlr.org/papers/v24/22-1395.html.
  5. N. Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023. doi:10.1017/9781009166164. URL https://www.nicolasboumal.net/book.
  6. Manopt, a Matlab toolbox for optimization on manifolds. Journal of Machine Learning Research, 15(42):1455–1459, 2014. URL https://www.manopt.org.
  7. E. A. Canale and P. Monzón. Exotic equilibria of Harary graphs and a new minimum degree lower bound for synchronization. Chaos, 25(2), 2015. doi:10.1063/1.4907952.
  8. A nonlinear model of opinion formation on the sphere. Discrete & Continuous Dynamical Systems, 35(9):4241–4268, 2015. doi:10.3934/dcds.2015.35.4241.
  9. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf.
  10. H. Cohn. Global equilibrium theory of charges on a circle. The American Mathematical Monthly, 67(4):338–343, 1960. doi:10.1080/00029890.1960.11989502.
  11. H. Cohn and A. Kumar. Universally optimal distribution of points on spheres. Journal of the American Mathematical Society, 20(1):99–148, 2007.
  12. F. Dörfler and F. Bullo. Synchronization in complex networks of phase oscillators: A survey. Automatica, 50(6):1539–1564, 2014. ISSN 0005-1098. doi:10.1016/j.automatica.2014.04.012.
  13. Redesigning the transformer architecture with insights from multi-particle dynamical systems. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5531–5544. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/2bd388f731f26312bfc0fe30da009595-Paper.pdf.
  14. W. E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1–11, March 2017. doi:10.1007/s40304-017-0103-z.
  15. T. Erber and G. Hockney. Equilibrium configurations of N equal charges on a sphere. Journal of Physics A: Mathematical and General, 24(23):L1369–L1377, 1991. doi:10.1088/0305-4470/24/23/008.
  16. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794, 2024.
  17. E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, dec 2017. doi:10.1088/1361-6420/aa9a90.
  18. R. Horn and C. Johnson. Topics in Matrix Analysis. Cambridge University Press, 1991.
  19. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933, 2022. URL https://arxiv.org/abs/2106.15933.
  20. Low-rank optimization on the cone of positive semidefinite matrices. SIAM Journal on Optimization, 20(5):2327–2351, 2010. doi:10.1137/080731359.
  21. Sufficiently dense Kuramoto networks are globally synchronizing. Chaos, 31(7), 2021. doi:10.1063/5.0057659.
  22. S. Krantz and H. Parks. The Implicit Function Theorem. Springer New York, 2013. doi:10.1007/978-1-4614-5981-1.
  23. Y. Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In Proceedings of the International Symposium on Mathematical Problems in Theoretical Physics, pages 420–422, Kyoto, Japan, Jan. 1975. doi:10.1007/bfb0013365.
  24. C. Lageman and Z. Sun. Consensus on spheres: Convergence analysis and perturbation theory. In Proceedings of the IEEE 55th Conference on Decision and Control (CDC), Las Vegas, NV, USA, Dec. 2016. doi:10.1109/cdc.2016.7798240.
  25. W. Li and M. W. Spong. Unified cooperative control of multiple agents on a sphere for different spherical patterns. IEEE Transactions on Automatic Control, 59(5):1283–1289, 2014. doi:10.1109/tac.2013.2286897.
  26. On the landscape of synchronization networks: A perspective from nonconvex optimization. SIAM Journal on Optimization, 29(3):1879–1907, 2019. doi:10.1137/18m1217644.
  27. S. Łojasiewicz. Ensembles semi-analytiques. Lecture Notes IHES (Bures-sur-Yvette), 1965.
  28. Understanding and improving transformer from a multi-particle dynamic system point of view. International Conference on Learning Representations 2020 Workshop on Integration of Deep Neural Models and Differential Equations, arXiv 1906.02762, 2020.
  29. J. Markdahl. Synchronization on Riemannian manifolds: Multiply connected implies multistable. IEEE Transactions on Automatic Control, 66(9):4311–4318, 2021. doi:10.1109/tac.2020.3030849.
  30. Almost global consensus on the n𝑛nitalic_n-sphere. IEEE Transactions on Automatic Control, 63(6):1664–1675, 2018. doi:10.1109/tac.2017.2752799.
  31. A. McRae and N. Boumal. Benign landscapes of low-dimensional relaxations for orthogonal synchronization on general graphs. SIAM Journal on Optimization, 34(2):1427–1454, 2024. doi:10.1137/23M1584642.
  32. R. Olfati-Saber. Swarms on sphere: A programmable swarm with synchronous behaviors like oscillator networks. In Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, CA, USA, Dec. 2006. doi:10.1109/cdc.2006.376811.
  33. S. Pesme and N. Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 7475–7505. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/17a9ab4190289f0e1504bbb98d1d111a-Paper-Conference.pdf.
  34. Sinkformers: Transformers with doubly stochastic attention. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 3515–3530. PMLR, 28–30 Mar 2022. URL https://proceedings.mlr.press/v151/sander22a.html.
  35. A. Sarlette and R. Sepulchre. Consensus optimization on manifolds. SIAM Journal on Control and Optimization, 48(1):56–76, 2009a.
  36. A. Sarlette and R. Sepulchre. Synchronization on the circle. arXiv preprint arXiv:0901.2408, 2009b.
  37. D. Sclosa. Kuramoto networks with infinitely many stable equilibria. SIAM Journal on Applied Dynamical Systems, 22(4):3267–3283, 2023. doi:10.1137/23m155400x.
  38. Stabilization of planar collective motion: All-to-all communication. IEEE Transactions on Automatic Control, 52(5):811–824, 2007. doi:10.1109/tac.2007.898077.
  39. M. Shub. Global Stability of Dynamical Systems. Springer New York, 1987. doi:10.1007/978-1-4757-1947-5.
  40. R. Taylor. There is no non-zero stable fixed point for dense networks in the homogeneous Kuramoto model. Journal of Physics A: Mathematical and Theoretical, 45(5):055102, 2012. doi:10.1088/1751-8113/45/5/055102.
  41. Dense networks that do not synchronize and sparse ones that do. Chaos, 30(8), 2020. doi:10.1063/5.0018322.
  42. The size of the sync basin. Chaos, 16, 2006. doi:10.1063/1.2165594.
  43. The lower bound of the network connectivity guaranteeing in-phase synchronization. Chaos, 31(6), 2021. doi:10.1063/5.0054271.
Citations (2)

Summary

We haven't generated a summary for this paper yet.

Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 1 post and received 1 like.