--- title: Deep Neural Network Catapult Phase url: https://www.emergentmind.com/topics/deep-neural-network-catapult-phase type: topic --- # Deep Neural Network Catapult Phase A deep neural network enters the "catapult phase" when trained with a learning rate above the traditional stability bound for local linearized dynamics but below a second, architecture-dependent divergence threshold. In this regime, gradient descent temporarily drives up the loss and sharply reduces the curvature of the loss landscape before settling into a minimum that is significantly flatter than those found using small learning rates. The catapult phase has been observed robustly in deep linear and nonlinear networks, under both full-batch and stochastic gradient descent, and is strongly associated with superior generalization performance across a range of realistic tasks. This nonperturbative phenomenon fills the critical gap between linearized, infinite-width theory and the entirely nonlinear, large-learning-rate dynamics prevalent in practical training scenarios [2003.02218, 2306.04815, 2205.11787, 2301.07737, 2011.12547]. ## 1. Formal Definition and Learning-Rate Regimes Let $f(\theta;x)$ be a neural network with parameters $\theta$, and let $L(\theta)$ denote the mean squared error (MSE) loss. The training dynamics exhibit three distinct regimes as the learning rate $\eta$ is varied: - **NTK (lazy) phase ($\eta<\eta_1$):** Training evolves close to linearly around initialization, governed by the Neural Tangent Kernel (NTK). The top eigenvalue of the Hessian or NTK, $\lambda_0$, remains constant. All parameter-space directions are contracted. - **Catapult phase ($\eta_1<\eta<\eta_2$):** For $\eta_1=2/\lambda_0 < \eta < \eta_2$, the linearized model predicts divergence, but the finite-width nonlinear network displays non-monotonic dynamics: the training loss initially spikes (the "catapult"), then quickly drops as the Hessian curvature $\lambda_t$ decreases dramatically, stabilizing when $\lambda_t < 2/\eta$. The minimum reached is much flatter, with $\lambda_{\mathrm{final}} \ll \lambda_0$. - **Divergent phase ($\eta > \eta_2$):** Training is unstable and diverges completely, or the network collapses (e.g., to dead ReLU units). Empirically, in practical ReLU networks, $\eta_2 \approx c/\lambda_0$ with $c \approx 12$; the theoretical toy model yields $c=4$ for two-layer linear networks [2003.02218, 2301.07737]. ## 2. Catapult Dynamics: Theory and Toy Models The canonical toy model utilizes a one-hidden-layer linear network with $f(x)=n^{-1/2} v^T u x$, tracked by the squared loss $L = \tfrac12 f^2$. Recursion relations derived for the output $f_t$ and an NTK-like measure $\lambda_t$ clarify phase transitions: \[ f_{t+1} = (1 - \eta \lambda_t + \eta^2 f_t^2 / n) f_t \] \[ \lambda_{t+1} = \lambda_t + (\eta f_t^2 / n)\big( \eta \lambda_t - 4 \big) \] In the catapult phase, unstable dynamics inflate $|f_t|$ and the leading curvature drops via nonlinear feedback until convergence resumes. The essence of the catapult is captured even in low-dimensional quadratic models, which demonstrate the universality and analytic tractability of this transition [2205.11787, 2301.07737]. | Regime | Learning Rate $\eta$ | Behavior | |------------------|------------------------------|-------------------------------------------------| | NTK/Lazy Phase | $0 < \eta < 2/\lambda_0$ | Monotonic loss decay, constant curvature | | Catapult Phase | $2/\lambda_0 < \eta < c/\lambda_0$ | Loss spike, rapid curvature drop, flat minima | | Divergence | $\eta > c/\lambda_0$ | Loss explodes, no convergence | ## 3. Empirical Manifestations: NTK, Loss, and Feature Learning Across architectures (MLPs, CNNs, WideResNets) and datasets (MNIST, CIFAR, SVHN), the catapult phase is evidenced by: - **Training loss:** Characteristic spike followed by rapid decay, localized to the subspace spanned by leading NTK/Hessian eigenvectors. - **Curvature:** Top NTK or Hessian eigenvalue $\lambda_t$ rapidly diminishes to $\sim2/\eta$, confirming migration to a flatter region. - **Generalization:** Test accuracy typically peaks in the catapult window, outperforming both smaller and excessively large learning rates [2003.02218, 2306.04815, 2205.11787]. Practical SGD, even with small batch sizes, produces catapults whenever $\eta$ momentarily exceeds $n/\lambda_1$ for the current batch. Smaller batch sizes introduce more variance in NTK eigenvalues, thereby inciting more frequent catapult events and facilitating superior feature learning via alignment with the average gradient outer product (AGOP) of the true predictor [2306.04815]. ## 4. Mechanistic Interpretation and Generalization Impact The critical mechanism underlying the catapult is nonlinear feedback: in the unstable phase, loss growth amplifies leading eigenmodes, provoking a sharp reduction in the NTK/Hessian spectrum (i.e., global curvature). Convergence resumes in a much flatter valley, which is empirically associated with improved generalization. This is a deterministic contraction to flat regions, in contrast to stochastic flattening via noise. Repeated catapults (e.g., via multiple learning-rate increases or frequent SGD batch transitions) drive enhanced feature alignment and learning, as measured by AGOP alignment. The magnitude of generalization improvements is most pronounced for low-rank target functions or tasks [2306.04815]. ## 5. Extensions: Losses, Architectures, and Theory The bulk of analytic understanding is for MSE loss and full-batch gradient descent. Catapult boundaries are shifted by momentum and adaptive optimizers (for example, with momentum $\mu$, instability in SGD requires $\eta > 2(1+\mu)/\lambda_0$). Cross-entropy loss, with its non-constant Hessian, complicates direct analysis, though catapult-like dynamics are observed empirically. Quadratic models (second-order Taylor approximations) replicate the essential catapult phenomenon and generalization trends of neural networks, while linear/NTK models do not, indicating the nonlinearity is the minimal requirement for catapult behavior [2205.11787]. Extensions to nonlinear, homogeneous two-layer networks confirm the universality of the phase structure [2301.07737]. ## 6. Training Protocols, Limitations, and Open Challenges Practical guidelines for exploiting the catapult phase include: - Estimating the initial curvature ($\lambda_0$ via Lanczos or similar methods) - Selecting $\eta \in [2/\lambda_0, c/\lambda_0]$, with $c\approx6$–$12$ for deep ReLU networks - Employing learning-rate warmup to stabilize initial catapult transitions - Employing learning-rate decay to refine convergence after entry into the flat basin Key limitations and open questions include: - Precise analytic characterization of the catapult phase for general nonlinear and finite-width networks remains incomplete - The dependence of the upper bound constant $c$ on architecture, depth, and normalization is not known in closed form - The global picture for cross-entropy loss and adaptive optimization schemes is not yet systematically developed - While strong empirical and toy-model evidence exists for improved generalization via catapulting, a fully rigorous, general theory is an unsolved problem [2003.02218, 2306.04815, 2011.12547] ## 7. Broader Implications and Universality The catapult phase elucidates why empirically successful large learning rates in deep-network training can exceed the NTK regime's purported stability limit. By leveraging initial instability, networks traverse out of sharp, initialization-dominated basins and into broad, flat minima, translating into improved test-time behavior. This phenomenon has been robustly identified across architectures, tasks, and optimization settings, and forms a central piece in the modern understanding of implicit bias, curvature manipulation, and learning dynamics in deep neural networks [2003.02218, 2306.04815, 2205.11787, 2301.07737, 2011.12547]. Source: https://www.emergentmind.com/topics/deep-neural-network-catapult-phase