Catapult Convergence in Neural Networks

• Catapult convergence is a training regime where super-critical learning rates trigger a temporary increase in loss before nonlinear dynamics reduce curvature and stabilize the model.
• The phenomenon is analyzed using quadratic models and neural quadratic approximations that capture the nonlinear feedback mechanisms missing in linearized NTK theory.
• This regime bridges the gap between lazy (NTK) convergence and divergence, providing insights into achieving stable training dynamics and improved generalization.

Catapult convergence denotes the convergent but non-monotonic regime of gradient descent that appears at large constant learning rates in finite-width neural networks and related quadratic models. In this regime, the loss initially increases—often exponentially—after the step size crosses the usual linear-stability threshold, yet training does not diverge; instead, the local curvature or tangent-kernel norm decreases until the effective dynamics re-enter a stable region, after which the loss rapidly decreases to zero or to a flatter minimum. In the literature, this behavior is placed between the lazy or NTK regime and outright divergence, and is used to explain why super-critical learning rates can remain stable and can be associated with improved generalization (Lewkowycz et al., 2020, Zhu et al., 2022).

1. Phase structure and defining thresholds

Gradient descent is written in the usual form

$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t).$

In the lazy or NTK regime, gradient descent on a sufficiently wide network or its linearization converges monotonically if

$\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$

where $H$ is the loss Hessian at initialization. The catapult phase begins when the learning rate becomes super-critical, so that the initial linearized dynamics would be unstable, but remains below a larger threshold $\eta_{\max}$ for which the full nonlinear dynamics still converges (Zhu et al., 2022).

In solvable large-width models, this phase structure is explicit. For the one-hidden-layer linear network analyzed in "The large learning rate phase of deep learning: the catapult mechanism," the small-learning-rate threshold is $\eta_1=2/\lambda_0$ and the large-learning-rate upper threshold is $\eta_2=4/\lambda_0$, where $\lambda_0$ is the initial NTK curvature. The three regimes are then: monotone decay for $\eta<\eta_1$, initial growth followed by convergence for $\eta_1<\eta<\eta_2$, and divergence for $\eta>\eta_2$. In general networks with ReLU nonlinearity, the same work reports an empirical upper bound $\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$0 with $\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$1 (Lewkowycz et al., 2020).

A central conceptual point is that this regime is not captured by purely linear models. In the quadratic-model analysis, the catapult phase occurs for

$\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$2

whereas the linearized model $\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$3 either underfits or diverges beyond the critical threshold. This identifies catapult convergence as a genuinely nonlinear feature of wide but finite networks rather than a consequence of classical NTK theory alone (Zhu et al., 2022).

2. Dynamical mechanism: instability, curvature collapse, and return to stability

The canonical mechanism is most transparent in the warmup model of (Lewkowycz et al., 2020). Let $\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$4 denote the network output error and let

$\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$5

be the NTK curvature. The exact gradient-descent recursions are

$\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$6

$\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$7

When $\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$8, one initially has $\eta<\frac{2}{\lambda_{\max}(H(\theta_0))},$9, so the linearized update is unstable and $H$0 and the loss grow exponentially for a short time. Once $H$1 becomes $H$2, the finite-width term in the $H$3 update drives $H$4 downward until $H$5. At that point, the multiplicative factor in the $H$6 update falls below unity in magnitude, and training enters a decaying regime that converges to a minimum with $H$7 (Lewkowycz et al., 2020).

The same mechanism appears in the quadratic-model treatment. For a single example with squared loss, if $H$8 is the model output and

$H$9

then the dynamics satisfy

$\eta_{\max}$0

where $\eta_{\max}$1 and $\eta_{\max}$2 are curvature corrections of order $\eta_{\max}$3. The top eigenvalue of the tangent kernel decreases during the catapult, and this kernel shrinkage is observed both in neural quadratic models and in finite-width networks (Zhu et al., 2022).

This mechanism is also the basis for the phrase “self-tuning.” The learning rate is initially too large for the starting curvature, but the dynamics itself reduces that curvature until the effective stability condition is restored. A plausible implication is that large-step training does not merely accelerate movement along a fixed landscape; it actively reshapes the local geometry encountered by the iterates.

3. Quadratic models and reduced low-dimensional descriptions

A major analytic simplification is the Neural Quadratic Model (NQM), obtained by taking the second-order Taylor expansion of a network around initialization:

$\eta_{\max}$4

Despite its fixed Hessian, this model still exhibits catapult dynamics. That observation is significant because it shows that full Hessian evolution is not necessary for the phenomenon; second-order parameter-space curvature already suffices (Zhu et al., 2022).

For the single-example dynamics, the paper introduces

$\eta_{\max}$5

and derives the reduced recurrences

$\eta_{\max}$6

$\eta_{\max}$7

These equations separate the regimes sharply. For sub-critical $\eta_{\max}$8, one has $\eta_{\max}$9 exponentially and $\eta_1=2/\lambda_0$0 remains near its initial value, reproducing lazy dynamics. For super-critical $\eta_1=2/\lambda_0$1, $\eta_1=2/\lambda_0$2 grows exponentially at first, then the negative term $\eta_1=2/\lambda_0$3 drives $\eta_1=2/\lambda_0$4 below $\eta_1=2/\lambda_0$5, after which $\eta_1=2/\lambda_0$6 collapses to $\eta_1=2/\lambda_0$7. For $\eta_1=2/\lambda_0$8, the dynamics runs away (Zhu et al., 2022).

The same work analyzes fixed points of the reduced dynamics and finds that the only stable steady state has $\eta_1=2/\lambda_0$9 with $\eta_2=4/\lambda_0$0, whereas any nonzero $\eta_2=4/\lambda_0$1 is unstable. This places catapult convergence within a conventional stability framework: the transient is violent, but the terminal state is a linearly stable low-curvature regime rather than a marginal or metastable artifact.

Multiple-example experiments reveal a further refinement. In uni-dimensional data, NQMs can display sequential catapult, first in the top eigendirection and then in a second eigendirection as $\eta_2=4/\lambda_0$2 grows. This suggests that catapult behavior can be spectrally layered rather than tied to a single scalar instability (Zhu et al., 2022).

4. Convergence theory beyond the warmup model

The paper "Catapult Dynamics and Phase Transitions in Quadratic Nets" develops convergence results for a broad class of quadratic models and two-layer homogeneous neural nets. In a two-mode reduction based on the top NTK eigendirection, the squared weight norm satisfies a recurrence of the form

$\eta_2=4/\lambda_0$3

with a coupled update for $\eta_2=4/\lambda_0$4. The analysis yields the sufficient interval

$\eta_2=4/\lambda_0$5

for catapult dynamics, while higher-order corrections sharpen the upper end to $\eta_2=4/\lambda_0$6 in practice. Theorem 3.1 states that there exists a critical loss level $\eta_2=4/\lambda_0$7 such that whenever $\eta_2=4/\lambda_0$8 one has $\eta_2=4/\lambda_0$9, preventing escape to infinite norm. Theorem 3.2 then gives eventual convergence to a global minimizer $\lambda_0$0 with $\lambda_0$1, so that once the iterates land in the stable regime the loss decays at a linear rate (Meltzer et al., 2023).

A complementary line of work places catapult convergence inside a more general theory of large-learning-rate implicit bias. "Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult" studies nonconvex objectives of the form

$\lambda_0$2

for which all global minima lie on the hyperbola $\lambda_0$3. The paper defines the degree of regularity

$\lambda_0$4

and shows that good regularity, corresponding to low dor, is the condition under which catapult, edge-of-stability behavior, and balancing are most likely to occur (Wang et al., 2023).

For the good-regularity case $\lambda_0$5, with initial point satisfying

$\lambda_0$6

and large constant step size

$\lambda_0$7

gradient descent converges to a minimizer on $\lambda_0$8 and obeys the non-asymptotic estimate

$\lambda_0$9

The paper interprets the transient through the one-step factor

$\eta<\eta_1$0

showing that if the initial sharpness exceeds the linear-stability bound $\eta<\eta_1$1, then $\eta<\eta_1$2 and the first step overshoots to the other side of the minimum, causing an initial loss increase. Good regularity keeps the subsequent dynamics controlled and yields a two-phase pattern of escape and contraction (Wang et al., 2023).

5. Scope, exceptions, and dependence on regularity

Catapult convergence is not a universal feature of large-step optimization. In deep linear networks with logistic loss, the phase structure depends strongly on the data geometry. For linearly separable data, there is no true catapult phase because the global solution lies at infinity. For non-separable data in the fully convex setting, there is again no intermediate regime: one has monotone convergence for $\eta<\eta_1$3 and divergence for $\eta<\eta_1$4. The intermediate catapult interval appears only in the degenerate special case and in the one-hidden-layer linear network under Assumption 3, where there exists an interval $\eta<\eta_1$5 such that the loss increases for all $\eta<\eta_1$6 and decreases monotonically to the global minimum for all $\eta<\eta_1$7 (Huang et al., 2020).

In that one-hidden-layer linear network, the top eigenvalue of the NTK decreases monotonically for $\eta<\eta_1$8, and Corollary 3.4 identifies the same nonzero spectrum at convergence for the Hessian and the NTK. This is one of the clearest formal statements of the flattening picture: catapult convergence is accompanied by monotone collapse of local curvature after the transient (Huang et al., 2020).

Regularity provides a second axis of non-universality. In the dor framework of (Wang et al., 2023), different losses and activations yield different effective objective regularities:

Loss × activation	dor
Huber × tanh	0
Huber × ReLU	1
Huber × ReLU$\eta<\eta_1$9	3
MSE × tanh	0
MSE × ReLU	2
MSE × ReLU$\eta_1<\eta<\eta_2$0	6

With large $\eta_1<\eta<\eta_2$1, the authors observe catapult, edge-of-stability behavior, and balancing only when dor$\eta_1<\eta<\eta_2$2, and no such phenomena when dor$\eta_1<\eta<\eta_2$3. Batch normalization is analyzed as a mechanism that globally bounds pre-activations and forces $\eta_1<\eta<\eta_2$4, thereby restoring these large-learning-rate phenomena even in cases that would otherwise have bad regularity (Wang et al., 2023).

A common misconception is that any sufficiently large learning rate should induce catapult behavior. The available analyses do not support that claim. The phenomenon depends on spectral thresholds, nonlinear curvature feedback, and regularity properties of the objective; outside those conditions, large-step training may simply diverge or remain in an ordinary convex regime.

6. Implicit bias, generalization, and practical interpretation

The empirical appeal of catapult convergence lies in its link to flatter minima and better test performance. In the quadratic-model study, the best test performance of the original network $\eta_1<\eta<\eta_2$5 and its quadratic approximation $\eta_1<\eta<\eta_2$6 is systematically better in the catapult regime than under sub-critical $\eta_1<\eta<\eta_2$7, whereas the linearized model $\eta_1<\eta<\eta_2$8 either underfits or diverges. The same experiments show a sharp drop in the top eigenvalue of the tangent kernel during the catapult, consistent with kernel shrinkage as the mechanism returning training to stability (Zhu et al., 2022).

The solvable-model and deep-network experiments of (Lewkowycz et al., 2020) reinforce this picture. In the warmup model, $\eta_1<\eta<\eta_2$9 remains flat in the lazy phase, rapidly drops to a stable plateau $\eta>\eta_2$0 in the catapult phase, and diverges above the upper threshold. On MNIST and CIFAR-10, early-time NTK spectra match these predictions, and test accuracy peaks inside the large-learning-rate interval $\eta>\eta_2$1. The paper interprets this as convergence to minima with lower local curvature, independent of SGD-noise effects (Lewkowycz et al., 2020).

For logistic-loss deep linear networks, the generalization story is more specific. On a CIFAR-10 “cars vs dogs” subset, test accuracy as a function of $\eta>\eta_2$2 peaks in the catapult interval $\eta>\eta_2$3. When the learning rate is later annealed from the catapult regime into the lazy regime, the resulting training combines the flat-minimum bias on the non-separable component with margin maximization on the separable component and obtains the best test accuracy in the reported experiments (Huang et al., 2020).

Several works also extract practical heuristics from the theory. One recommendation is to estimate the top NTK eigenvalue at initialization and choose $\eta>\eta_2$4 slightly above the kernel threshold; another is to monitor $\eta>\eta_2$5 or $\eta>\eta_2$6 during training and regard the condition $\eta>\eta_2$7 as the onset of the stable regime. In ReLU networks, very large $\eta>\eta_2$8 can still be stable when aided by gradient clipping or small batch noise, and in that setting the activation map becomes increasingly sparse as the learning rate grows (Meltzer et al., 2023).

Taken together, these results portray catapult convergence as a large-step optimization regime in which temporary instability is not a failure mode but part of the mechanism. The initial loss spike expels the iterates from a sharp basin, curvature then collapses through nonlinear feedback, and the dynamics settles into a flatter stable region. This suggests that the primary explanatory gap between small-step NTK theory and practical deep-network training lies not in whether stability exists, but in how stability is dynamically recovered after an intentionally super-critical initialization of the learning rate.