Architecture Induces Structural Invariant Manifolds of Neural Network Training Dynamics

Abstract: While architecture is recognized as key to the performance of deep neural networks, its precise effect on training dynamics has been unclear due to the confounding influence of data and loss functions. This paper proposed an analytic framework based on the geometric control theory to characterize the dynamical properties intrinsic to a model's parameterization. We prove that the Structural Invariant Manifolds (SIMs) of an analytic model $F(\mathbf{\theta})(\mathbf{x})$--submanifolds that confine gradient flow trajectories independent of data and loss--are unions of orbits of the vector field family ${\nabla_{\mathbf{\theta}} F(\cdot)(\mathbf{x})\mid\mathbf{x}\in\mathbb{R}^d}$. We then prove that a model's symmetry, e.g., permutation symmetry for neural networks, induces SIMs. Applying this, we characterize the hierarchy of symmetry-induced SIMs in fully-connected networks, where dynamics exhibit neuron condensation and equivalence to reduced-width networks. For two-layer networks, we prove all SIMs are symmetry-induced, closing the gap between known symmetries and all possible invariants. Overall, by establishing the framework for analyzing SIMs induced by architecture, our work paves the way for a deeper analysis of neural network training dynamics and generalization in the near future.