Papers
Topics
Authors
Recent
Search
2000 character limit reached

Symmetry & Critical Points

Published 26 Aug 2024 in cs.LG, cs.NA, math.NA, math.OC, and stat.ML | (2408.14445v1)

Abstract: Critical points of an invariant function may or may not be symmetric. We prove, however, that if a symmetric critical point exists, those adjacent to it are generically symmetry breaking. This mathematical mechanism is shown to carry important implications for our ability to efficiently minimize invariant nonconvex functions, in particular those associated with neural networks.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Y. Arjevani, “Optimization and symmetry,” In preparation, 2025.
  2. L. Qi, “Eigenvalues of a real supersymmetric tensor,” Journal of symbolic computation, vol. 40, no. 6, pp. 1302–1324, 2005.
  3. E. Bierstone, “Generic equivariant maps,” Real and Complex Singularities, Oslo, pp. 127–161, 1976.
  4. A. M. Garsia and D. Stanton, “Group actions on stanley-reisner rings and invariants of permutation groups,” Advances in Mathematics, vol. 51, no. 2, pp. 107–201, 1984.
  5. L. Smith, Polynomial invariants of finite groups. AK Peters/CRC Press, 1995.
  6. Y. Arjevani, “Symmetry & critical points ii,” In preparation, 2024.
  7. B. Simsek, A. Bendjeddou, W. Gerstner, and J. Brea, “Should under-parameterized student networks copy or average teacher weights?,” Advances in Neural Information Processing Systems, vol. 36, 2024.
  8. I. Safran, G. Vardi, and J. D. Lee, “On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias,” Advances in Neural Information Processing Systems, vol. 35, pp. 32667–32679, 2022.
  9. Y. Arjevani and M. Field, “Analytic study of families of spurious minima in two-layer relu neural networks: A tale of symmetry ii,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  10. Y. Arjevani, “Deep symmetry breaking,” In preparation, 2024.
  11. Y. Arjevani and M. Field, “On the principle of least symmetry breaking in shallow relu models,” arXiv preprint arXiv:1912.11939, 2019.
  12. Y. Arjevani, J. Bruna, M. Field, J. Kileel, M. Trager, and F. Williams, “Symmetry breaking in symmetric tensor decomposition,” arXiv preprint arXiv:2103.06234, 2021.
  13. D. Saad and S. A. Solla, “On-line learning in soft committee machines,” Physical Review E, vol. 52, no. 4, p. 4225, 1995.
  14. S. Goldt, M. Advani, A. M. Saxe, F. Krzakala, and L. Zdeborová, “Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup,” Advances in neural information processing systems, vol. 32, 2019.
  15. Y. Cho and L. K. Saul, “Kernel methods for deep learning,” in Advances in neural information processing systems, pp. 342–350, 2009.
  16. Y. Arjevani, “Hidden minima in two-layer relu networks,” arXiv preprint arXiv:2312.16819, 2023.
  17. Springer Science & Business Media, 2012.
  18. C. B. Thomas, Representations of finite and Lie groups. World Scientific, 2004.
  19. W. Fulton and J. Harris, “Representation theory, volume 129 of,” Graduate Texts in Mathematics, 1991.
  20. Y. Arjevani and G. Vinograd, “Symmetry & critical points for symmetric tensor decompositions problems,” arXiv preprint arXiv:2306.07886, 2023.
  21. Y. Arjevani and M. Field, “Analytic characterization of the hessian in shallow relu models: A tale of symmetry,” Advances in Neural Information Processing Systems, vol. 33, 2020.
  22. C. J. Hillar and L.-H. Lim, “Most tensor problems are np-hard,” Journal of the ACM (JACM), vol. 60, no. 6, pp. 1–39, 2013.
  23. S. Hu, “Relations of the nuclear norm of a tensor and its matrix flattenings,” Linear Algebra and its Applications, vol. 478, pp. 188–199, 2015.
  24. Y. Arjevani and M. Field, “Annihilation of spurious minima in two-layer relu networks,” Advances in Neural Information Processing Systems, vol. 35, pp. 37510–37523, 2022.
  25. A. B. Netto, “Jet-detectable extrema,” Proceedings of the American Mathematical Society, vol. 92, no. 4, pp. 604–608, 1984.
  26. T.-C. Kuo, “On c0-sufficiency of jets of potential functions,” Topology, vol. 8, no. 2, pp. 167–171, 1969.
  27. T.-C. Kuo, “Characterizations of v-sufficiency of jets,” Topology, vol. 11, no. 1, pp. 115–131, 1972.
  28. M. Buchner, J. Marsden, and S. Schecter, “Applications of the blowing-up construction and algebraic geometry to bifurcation problems,” Journal of Differential Equations, vol. 48, no. 3, pp. 404–433, 1983.
  29. M. A. S. Ruas, “Old and new results on density of stable mappings,” in Handbook of Geometry and Topology of Singularities III, pp. 1–80, Springer, 2022.
  30. Y. Arjevani and M. Field, “Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of s k,” Nonlinearity, vol. 35, no. 6, p. 2809, 2022.
  31. D. Cartwright and B. Sturmfels, “The number of eigenvalues of a tensor,” Linear algebra and its applications, vol. 438, no. 2, pp. 942–952, 2013.
  32. Y. Arjevani and M. Field, “Symmetry & critical points for a model shallow neural network,” Physica D: Nonlinear Phenomena, vol. 427, p. 133014, 2021.
  33. L. Van den Dries, “A generalization of the tarski-seidenberg theorem, and some nondefinability results,” 1986.
  34. L. Sagun, L. Bottou, and Y. LeCun, “Eigenvalues of the hessian in deep learning: Singularity and beyond,” arXiv preprint arXiv:1611.07476, 2016.
  35. P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina, “Entropy-sgd: Biasing gradient descent into wide valleys,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 12, p. 124018, 2019.
  36. V. Papyan, “The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size,” arXiv preprint arXiv:1811.07062, 2018.
  37. Springer, 2006.
  38. J. N. Mather, “Stability of c-infinity mappings: V, transversality,” Advances in Mathematics, vol. 4, no. 3, pp. 301–336, 1970.
  39. D. J. Trotman, “Stability of transversality to a stratification implies whitney (a)-regularity,” Inventiones mathematicae, vol. 50, no. 3, pp. 273–277, 1978.
  40. J. Mather, “Notes on topological stability,” Bulletin of the American Mathematical Society, vol. 49, no. 4, pp. 475–506, 2012.
  41. J. N. Mather, “Stratifications and mappings,” in Dynamical systems, pp. 195–232, Elsevier, 1973.
  42. Springer, 2020.
  43. L. Van den Dries and C. Miller, “Geometric categories and o-minimal structures,” 1996.
  44. T. L. Loi, “Lecture 2: Stratifications in o-minimal structures,” in The Japanese-Australian Workshop on Real and Complex Singularities: JARCS III, vol. 43, pp. 31–40, Australian National University, Mathematical Sciences Institute, 2010.
  45. A. Némethi and A. Zaharia, “Milnor fibration at infinity,” Indagationes Mathematicae, vol. 3, no. 3, pp. 323–335, 1992.
  46. T. Lê Loi and A. Zaharia, “Bifurcation sets of functions definable in o𝑜oitalic_o-minimal structures,” Illinois Journal of Mathematics, vol. 42, no. 3, pp. 449–457, 1998.
  47. A. H. Durfee, “The index of gradf (x, y),” Topology, vol. 37, no. 6, pp. 1339–1361, 1998.
  48. World Scientific, 2016.
  49. M. Golubitsky, “An introduction to catastrophe theory and its applications,” Siam Review, vol. 20, no. 2, pp. 352–387, 1978.
  50. G. W. Schwarz, “Smooth functions invariant under the action of a compact lie group,” Topology, vol. 14, no. 1, pp. 63–68, 1975.
  51. J. N. Mather, “Stability of c-infinty mappings, iii. finitely determined map-germs,” Publications Mathématiques de l’IHÉS, vol. 35, pp. 127–156, 1968.
  52. H. R. Parks and S. Krantz, A primer of real analytic functions. Birkhäuser Verlag Boston (MA), 1992.
  53. M. Field and R. Richardson, “Symmetry breaking and branching patterns in equivariant bifurcation theory i,” Archive for rational mechanics and analysis, vol. 120, pp. 147–190, 1992.
  54. H. Whitney, “Elementary structure of real algebraic varieties,” Hassler Whitney Collected Papers, pp. 456–467, 1992.
  55. H. Derksen and G. Kemper, Computational invariant theory. Springer, 2015.
  56. M. Göbel, “Computing bases for rings of permutation-invariant polynomials,” Journal of Symbolic Computation, vol. 19, no. 4, pp. 285–291, 1995.
  57. E. Bierstone, “General position of equivariant maps,” Transactions of the American Mathematical Society, vol. 234, no. 2, pp. 447–466, 1977.
  58. M. Field, “Transversality in g-manifolds,” Transactions of the American Mathematical Society, vol. 231, no. 2, pp. 429–450, 1977.
  59. M. Field, “Equivariant bifurcation theory and symmetry breaking,” Journal of Dynamics and Differential equations, vol. 1, pp. 369–421, 1989.
  60. F. D. Murnaghan, “The analysis of the kronecker product of irreducible representations of the symmetric group,” American journal of mathematics, vol. 60, no. 3, pp. 761–784, 1938.
  61. Königliche Akademie der Wissenschaften, 1900.
  62. F. D. Murnaghan, “The characters of the symmetric group,” American Journal of Mathematics, vol. 59, no. 4, pp. 739–753, 1937.
  63. D. Ruelle, “Bifurcations in the presence of a symmetry group,” 1973.
  64. M. Field and R. Richardson, “Symmetry breaking and branching patterns in equivariant bifurcation theory ii,” Archive for rational mechanics and analysis, vol. 120, pp. 147–190, 1992.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 1 tweet with 2 likes about this paper.

Sign Up for Free