---
title: Edge-of-Stability Regime in Neural Networks
url: https://www.emergentmind.com/topics/edge-of-stability-eos-regime-bbef1f0e-36e2-469a-ab08-62bb5f7af7ae
type: topic
---

# Edge-of-Stability Regime in Neural Networks

The Edge-of-Stability (EoS) regime refers to a ubiquitous and distinctive phase in the training of overparameterized neural networks and related models, characterized by systematic oscillatory behavior of sharpness (typically quantified as the largest Hessian eigenvalue or related curvature proxy) near a precise instability threshold dictated by the optimization step size. Contrary to classical smooth optimization theory, which mandates that gradient descent step sizes remain below a stability cutoff, empirical and theoretical analysis reveal that optimization frequently transits and then self-regulates precisely at—or in specific cases, slightly below—this boundary, enabling efficient descent despite the apparent local instability.

## 1. Definition and Unifying Theoretical Principles

The EoS regime is precisely delineated by the coupling between the optimizer step size $\eta$ and curvature/“sharpness” metrics:

- **Sharpness**: Typically $\lambda_{\max}(\nabla^2L(w))$, the largest eigenvalue of the Hessian of the loss $L$.
- **Stability threshold**: For classical gradient descent (GD) on a quadratic model, monotonic convergence requires $\eta \cdot \lambda_{\max} < 2$. Violation of this induces oscillatory, but not divergent, descent [2205.09745, 2210.03294].
- **Edge of Stability**: The regime where $\lambda_{\max} \gtrsim 2/\eta$, yet training loss continues to decrease on average, interleaved with non-monotonic local oscillations [2604.20446].

The emergence of this phenomenon is universal: full-batch GD, stochastic optimization, preconditioned/adaptive methods, and even zeroth-order algorithms all exhibit sharpness self-regulation at boundaries dictated by step size, optimizer geometry, and problem structure [2603.05002, 2207.14484, 2604.14669].

## 2. Phases and Dynamics of Edge-of-Stability

A canonical EoS trajectory unfolds in discernible phases (not all present in every model):

- **Phase I: Progressive Sharpening** — Early iterations see sharpness increase monotonically toward the stability threshold, driven by gradient flow-like dynamics [2207.12678, 2503.02809]. Output layer scaling and data-covariance properties tightly predict this upward movement.
- **Phase II: Instability/Transition** — On reaching and marginally exceeding $2/\eta$, the trajectory exhibits transient overshoots and reversal events in the leading curvature direction, causing small but distinct increases in loss and norm adjustments in network weights [2210.03294, 2207.12678].
- **Phase III: Self-Stabilization** — Oscillatory interactions between top modes, output scaling, and geometrical alignment force sharpness to hover near the threshold, with the trajectory prevented from persistent instability via dynamic readjustments [2604.20446, 2503.02809].
- **Cycle Repeats** — Over training, these oscillatory cycles repeat, underpinning slow but steady optimization progress despite formal local instability.

The EoS is robust to the presence of nonconvexity, high depth, overparameterization, and intricate neural architectures [2502.20531, 2604.02653, 2412.08025].

## 3. Model-Specific Instantiations and Extensions

EoS appears across a broad spectrum of optimizers and architectures, each with unique stability signatures:

| Setting            | Sharpness Constraint    | Regularized Quantity             | Reference                     |
|--------------------|------------------------|----------------------------------|-------------------------------|
| Full-batch GD      | $\lambda_{\max}(H) \rightarrow 2/\eta$      | Top Hessian eigenvalue           | [2205.09745, 2604.20446]      |
| SGD (mini-batch)   | $\text{Batch Sharpness} \rightarrow 2/\eta$ | Expected mini-batch directional  | [2412.20553, 2604.14108]      |
| Adam, Adagrad      | $\lambda_{\max}(P^{-1} H)\rightarrow{\rm thresh.}$           | Preconditioned Hessian eigenvalue   | [2207.14484]                  |
| Zeroth-Order (ZO)  | $\operatorname{Tr}(H)\rightarrow 2/\eta$   | Hessian trace                    | [2604.14669]                  |

For momentum and stochastic cases, sharpness plateaus interpolate between lower ($2(1-\beta)/\eta$) and higher ($2(1+\beta)/\eta$) thresholds depending on batch size and noise, delineating deterministic and noise-dominated EoS [2604.14108].

In kernel (NTK) or deep linear matrix factorization models, EoS governs both eigenvalue spectra and eigenvector evolution, with feature learning and target alignment dynamics tightly controlled by sharpness cycles [2507.12837, 2502.20531].

## 4. Analytical Mechanisms and Bifurcations

EoS is fundamentally linked to the spectral and geometric structure of the loss landscape and the discrete-time update map:

- **Edge coupling functional**: A symmetric “action” on consecutive iterate pairs, with criticality fixed by $\mathcal{A}_\eta(x,y) = L(x) + L(y) - \frac{1}{2\eta}\|x-y\|^2$, organizes all dynamics. Differencing its optimality condition yields a recurrence whose linear stability boundary is $2/\eta$ [2604.20446].
- **Bifurcation theory**: Nonlinear period-doubling and center manifold expansions reveal that, above threshold, training settles into one- or multi-dimensional periodic orbits (period-2 cycles) pinning sharpness at or just below $2/\eta$. The side and nature of bifurcations depend on higher-order derivatives and product-stability conditions of the loss [2206.04172, 2210.03294, 2604.02653].
- **Mean-value localization**: Taylor and telescoping arguments localize observed sharpness oscillations to actual Hessian values at interior points along the GD step, yielding exact edge-forcing of the true spectrum [2604.20446].
- **Non-Euclidean and adaptive geometries**: EoS generalizes to arbitrary norm geometries and adaptive algorithms via a sharpness quantity $S^{\|\cdot\|}(w)$, with the GD stability edge universally at $2/S^{\|\cdot\|}$ [2603.05002, 2207.14484].

## 5. Implicit Regularization and Generalization

Operating at the EoS induces strong implicit biases in model selection:

- **Classical GD**: Large $\eta$ drives the model into regions of low top Hessian eigenvalue, biasing toward flatter minima in the direction of sharpest curvature [2205.09745, 2503.02809].
- **ZO methods**: The mean-square EoS places the constraint on trace($H$), hence regularization acts on the “bulk” of the spectrum and not merely its edge [2604.14669].
- **Variational/injective frameworks**: By dynamically tuning the effective EoS threshold, methods like variational learning force diffusion toward lower-sharpness minima and enable controlled generalization improvements [2506.12903].
- **SGD and EoSS**: Small batch size exaggerates mini-batch sharpness and, due to convexity arguments (Jensen's gap), self-regularizes the full-batch sharpness to even flatter regions than full-batch GD would; empirically this is tightly coupled to improved generalization [2412.20553, 2604.19740].
- **Sharpness Dimension**: Generalization in the EoS regime correlates with the fractal dimension of the long-run attractor set generated by the optimizer dynamics, which, under EoS, is often substantially lower than the ambient parameter dimension [2604.19740].

## 6. Extensions, Singular Cases, and Limitations

Several advanced and singular regimes have been mathematically characterized:

- **Overparameterized quadratics**: Even with quadratic loss, if parameterization is nonlinear (e.g., depth-2 diagonal nets), EoS arises and leads to structured oscillatory convergence and bias away from minimum-norm interpolants [2412.08025].
- **Loss-specific phenomena**: Logistic regression, but not exponential loss, possesses global convergence and classical max-margin bias at EoS for arbitrary stepsizes—even beyond stability—highlighting loss-dependent mechanisms [2305.11788].
- **PDE and numerical viewpoints**: EoS coincides with bounded, restrained numerical instabilities in gradient-flow PDEs for deep nets, where nonlinearity dynamically suppresses instability growth, explaining why divergence does not occur even when classical conditions are violated [2206.02001].
- **Differential Privacy**: DP noise and clipping slow or inhibit the attainment of EoS, lowering sharpness plateaus and biasing toward even flatter minima; practical schedules must account for this in tuning learning rate and privacy parameters [2512.19019].
- **Limitations**: EoS theory for multidimensional, non-factorized, highly nonlinear, or dynamic-learning-rate settings is still evolving. Local analysis often does not fully explain the global approach to EoS from random initialization, and convergence guarantees may be limited to neighborhoods of stable fixed points or cycles [2604.02653, 2210.03294].

## 7. Practical Considerations and Applications

EoS governs both optimization efficiency and generalization trade-offs, with significant repercussions for practical deep learning:

- **Step size tuning**: Efficient convergence often requires pushing $\eta$ up to—but not far beyond—the EoS threshold; excessive overshoot can induce prolonged instability or slow convergence along flat directions [2207.12678, 2503.02809].
- **Batch size and momentum selection**: Batch sharpness monitoring and adaptive schedule interventions can safely maintain the system at the EoSS or EoS plateau, maximizing speed while avoiding divergence [2604.14108, 2412.20553].
- **Regularization and model robustness**: Mechanisms such as weight decay, architectural choices, variational “temperature”, noise, and gradient clipping modify the effective EoS threshold and thereby allow explicit control of the trajectory’s implicit bias and flatter-regions exploration [2506.12903, 2512.19019].
- **Feature learning and NTK adaptation**: Periodic sharpness cycles induce rotation and realignment of leading kernel eigenvectors with targets, undergirding feature learning properties beyond the lazy NTK regime [2507.12837].

In totality, the Edge-of-Stability regime constitutes a unifying principle in modern deep learning optimization, explaining why and how large learning rates yield both efficient convergence and robust generalization: the optimizer, network, and data geometry together induce self-regulation at sharpness thresholds dictated by the training protocol, dynamically balancing local instabilities and global descent [2604.20446, 2205.09745, 2604.19740, 2603.05002, 2412.20553, 2207.14484, 2604.14669].

Source: https://www.emergentmind.com/topics/edge-of-stability-eos-regime-bbef1f0e-36e2-469a-ab08-62bb5f7af7ae