Goldstone Modes: How Physics Unlocks Deep Network Trainability
This presentation explores how spontaneous symmetry breaking in neural networks creates Goldstone-like excitations that enable robust information propagation across depth and time. Drawing from equilibrium statistical mechanics, the work demonstrates that internal layer equivariance under continuous symmetry groups such as U(1) and O(k) provides a protected channel for signal transmission, independent of task symmetries. Through rigorous mean-field analysis and experiments on feedforward, recurrent, and convolutional architectures, the authors show how symmetry-protected modes improve trainability in very deep networks and long-sequence tasks without normalization or skip connections.Script
Deep networks have a hidden enemy: information vanishes as signals pass through hundreds of layers, making training nearly impossible. But physics offers an unexpected solution through spontaneous symmetry breaking.
When networks operate with internal U(1) or O(k) equivariance, they exhibit a phase transition. Below a critical weight variance, activations decay to zero and erase all input information. Above this threshold, the symmetry spontaneously breaks, activations stabilize at nonzero magnitude, and a protected channel emerges that carries information across arbitrary depth.
Mean-field analysis reveals that in the broken symmetry phase, the phase component of complex activations propagates unchanged through every layer. This Goldstone mode acts like a non-dissipative channel, allowing gradient signals to flow backward and input correlations to persist forward without exponential decay.
Experiments confirm the theory. 100 layer feedforward networks with U(1) or O(4) equivariance train successfully in the broken phase with no normalization or skip connections, reaching over 85% test accuracy on Fashion MNIST where generic linear architectures collapse below 20%. Recurrent models show even sharper gains, solving copy tasks with delays exceeding 100 steps where vanilla RNNs and GRUs fail completely.
In convolutional recurrent models, the symmetry breaking manifests as topological defects. Vortices in the phase field emerge during training and persist for hundreds of sequence steps, suggesting a novel mechanism for spatial memory storage. These defects appear robust, annihilating only on direct contact, and their density scales with hidden state capacity.
Spontaneous symmetry breaking offers a physics-grounded design principle that works independently of task structure, enabling stable training in regimes where edge of chaos heuristics break down. If you want to explore how symmetry protects information in your own models, visit EmergentMind.com to dive deeper into this work and create videos on the research that matters to you.
