Edge of Stability in Deep Learning

• Edge of Stability is a regime where optimization dynamics operate near the critical curvature threshold, leading to oscillatory behavior and long-term loss reduction.
• The phenomenon reveals self-stabilization mechanisms where higher-order gradients redirect updates to reduce sharpness and prevent divergence.
• The concept extends across various optimizers and geometries, influencing stability, implicit bias, and generalization in deep learning applications.

In contemporary optimization theory for deep learning, the Edge of Stability (EoS) denotes the regime in which discrete-time training dynamics operate near the curvature threshold at which a local quadratic model would cease to predict one-step loss decrease. For full-batch gradient descent (GD) with step size $\eta$, this threshold is set by the largest Hessian eigenvalue, or sharpness, and empirically training often exhibits progressive sharpening until $\lambda_{\max}(\nabla^2 L)$ approaches $2/\eta$, after which the loss becomes non-monotone on short time scales yet continues to decrease over longer horizons (Cohen et al., 2021). Subsequent work has recast EoS as a broader phenomenon that depends on optimizer geometry, stochasticity, curvature control, and data structure, rather than as a peculiarity of vanilla GD alone (Damian et al., 2022, Islamov et al., 5 Mar 2026, Liao et al., 22 Apr 2026).

1. Classical definition and discrete-time stability boundary

For GD,

$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$

the local quadratic approximation yields the classical stability condition

$0<\eta \lambda_i<2$

along each Hessian eigenmode, so the sharpest mode governs the threshold:

$$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$

In the empirical regime identified in neural-network training, sharpness rises during early optimization and then hovers near this value, while the loss shows oscillations or spikes at the iterate scale but still decreases over long horizons (Cohen et al., 2021).

The same logic extends to momentum. For heavy-ball momentum on a scalar quadratic mode,

$x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$

the divergence boundary becomes

$\eta \lambda = 2(1+\beta),$

so the corresponding EoS threshold is $\lambda_{\max}\approx (2+2\beta)/\eta$ (Iordan et al., 2023). Empirical studies in supervised learning and off-policy reinforcement learning use precisely this threshold to identify whether the dynamics have entered an EoS regime.

Two features are central to the modern use of the term. First, EoS is a discrete-time phenomenon: gradient flow does not depend on $\eta$ and therefore cannot express the threshold $\lambda_{\max}(\nabla^2 L)$0. Second, the defining symptom is not immediate divergence but a sustained near-critical regime in which curvature remains close to the nominal stability boundary while optimization still progresses (Cohen et al., 2021, Marion, 16 Jun 2026).

2. Mechanistic accounts of why training does not simply diverge

A major theoretical development is the self-stabilization picture. In this view, once GD becomes unstable along the top Hessian eigenvector $\lambda_{\max}(\nabla^2 L)$1, third-order structure pushes the trajectory in a sharpness-reducing direction. For simple top eigenvalue,

$\lambda_{\max}(\nabla^2 L)$2

so the same cubic tensor that is absent from a purely quadratic local model controls the first-order evolution of sharpness itself (Damian et al., 2022). This leads to a reduced two-dimensional description in coordinates aligned with $\lambda_{\max}(\nabla^2 L)$3 and $\lambda_{\max}(\nabla^2 L)$4, and to the interpretation that GD near EoS implicitly follows projected gradient descent on the constraint

$\lambda_{\max}(\nabla^2 L)$5

rather than unconstrained descent on $\lambda_{\max}(\nabla^2 L)$6 alone (Damian et al., 2022).

A distinct but complementary explanation is given by the edge coupling

$\lambda_{\max}(\nabla^2 L)$7

Its partial criticality condition reproduces the GD update, and differencing that condition yields the exact step recurrence

$\lambda_{\max}(\nabla^2 L)$8

while a second-order expansion yields the loss-change identity

$\lambda_{\max}(\nabla^2 L)$9

Summing this identity telescopically forces a weighted average of directional curvatures toward $2/\eta$0, and a mean-value argument localizes those directional curvatures to actual Hessian values along each step segment (Litman, 22 Apr 2026). In that formulation, EoS is not merely a local stability heuristic but the consequence of an exact global identity along the GD trajectory.

Continuous-time effective descriptions have also been proposed. Edge Flow decomposes the dynamics into a slowly drifting center $2/\eta$1, a unit oscillation direction $2/\eta$2, and an oscillation magnitude $2/\eta$3, with

$2/\eta$4

$2/\eta$5

$2/\eta$6

The amplitude equation makes the threshold explicit, while the symmetrized center dynamics creates a negative feedback loop that reduces sharpness when oscillations grow (Marion, 16 Jun 2026). A related free-energy description instead tracks the slow mean trajectory and the covariance $2/\eta$7 of fast oscillations via

$2/\eta$8

which is dissipated when $2/\eta$9 (Courcel, 3 Jun 2026).

3. Optimizer- and geometry-dependent generalizations

The threshold $\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$0 is not universal in its Euclidean GD form. Several works reformulate EoS so that the relevant curvature quantity matches the optimizer.

Setting	Stability quantity	Edge condition
GD	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$1	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$2
Heavy-ball momentum	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$3	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$4
SAM	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$5 with gradient dependence	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$6
Non-Euclidean GD	generalized sharpness $\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$7	$\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$8
Adam-type methods	preconditioned sharpness $\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),$9	empirical AEoS at $0<\eta \lambda_i<2$0 in a fixed setting
Zeroth-order GD	mean-square stability depends on full spectrum	trace-based band around $0<\eta \lambda_i<2$1

For Sharpness-Aware Minimization (SAM), the update

$0<\eta \lambda_i<2$2

modifies the stability boundary itself. Under the same local quadratic model used for GD, the SAM edge becomes

$0<\eta \lambda_i<2$3

so unlike the GD threshold it depends on the gradient norm and the perturbation radius $0<\eta \lambda_i<2$4 (Long et al., 2023). This dependence implies that SAM’s admissible curvature typically tightens as training progresses and $0<\eta \lambda_i<2$5 shrinks.

For non-Euclidean descent, the relevant curvature is

$0<\eta \lambda_i<2$6

equivalently the induced operator norm $0<\eta \lambda_i<2$7 in the symmetric positive-semidefinite case. This recovers the standard Hessian spectral norm in the Euclidean geometry, but it also covers $0<\eta \lambda_i<2$8-descent, block coordinate descent, spectral GD, and normalized methods such as SignGD and Muon without momentum, all of which empirically exhibit progressive sharpening and oscillations near $0<\eta \lambda_i<2$9 when sharpness is measured in the optimizer’s own geometry (Islamov et al., 5 Mar 2026).

For adaptive methods, the relevant quantity is not raw Hessian sharpness but preconditioned sharpness. In Adam, with diagonal preconditioner $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$0, the stability-relevant matrix is $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$1. In one empirical setting, non-private full-batch Adam equilibrates near a threshold $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$2 with $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$3, while raw $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$4 can continue to increase after the preconditioned sharpness has plateaued (Hussain et al., 22 Dec 2025).

For zeroth-order methods based on the two-point estimator, mean-square stability depends on the entire Hessian spectrum rather than only on $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$5. For ZO-GD, the critical step size is the unique $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$6 satisfying

$$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$7

with $$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$8, and it is bounded by

$$\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.$$9

Empirically, full-batch ZO-GD, ZO-GDM, and ZO-Adam stabilize near the corresponding trace-based mean-square stability bands rather than a purely top-eigenvalue threshold (Song et al., 16 Apr 2026).

4. Stochasticity, minibatches, and domain-specific manifestations

Mini-batch SGD does not simply replicate the full-batch EoS picture. A stochastic self-stabilization analysis predicts that full-batch sharpness should stabilize below $x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$0, with the gap

$x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$1

where $x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$2 is the progressive-sharpening rate, $x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$3 is the self-stabilization strength, and $x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$4 is the gradient-noise variance projected onto the top Hessian eigenvector (Liao et al., 22 Apr 2026). This recovers full-batch GD when $x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$5 and predicts flatter solutions as batch size decreases.

In reinforcement learning, EoS is present but loss-dependent. In off-policy Q-learning, DQN with a Huber loss shows clear offline EoS: the leading Hessian eigenvalue rises to the heavy-ball threshold $x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$6 and fluctuates around it. C51, which uses a cross-entropy loss over return distributions, instead shows weak or absent offline EoS and, in online training, early threshold overshoot followed by a later sharpness decrease (Iordan et al., 2023). This differentiates the role of regression-like and cross-entropy curvature geometries in RL.

In differentially private training, clipping and Gaussian noise modify both the trajectory and the stability regime. Private GD and private Adam generally show reduced sharpness, delayed breakeven, and damped oscillations relative to their non-private counterparts, but at large learning rates and large privacy budgets they can still approach or exceed the corresponding non-private boundaries (Hussain et al., 22 Dec 2025). This suggests that privacy mechanisms alter, but do not eliminate, edge-like regimes.

For deep linear networks, training beyond EoS has been analyzed with unusually fine resolution. In deep matrix factorization, loss oscillations follow a period-doubling route to chaos, and the oscillatory subspace dimension is determined exactly by the learning rate. If

$x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$7

then the top-$x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$8 singular directions oscillate when

$x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},$9

In that regime, oscillations are confined to the leading feature subspace, and the balancing gap between layer singular values decays monotonically to zero beyond EoS (Ghosh et al., 27 Feb 2025).

5. Selectivity, implicit bias, and generalization

EoS is not only a global statement about an optimizer’s trajectory; it can selectively redistribute learning across the data distribution. A recent causal branching analysis shows that groups benefit from remaining at EoS only if two conditions hold jointly: their aggregate gradient must align with the top Hessian eigendirection, and their gradient magnitude must remain non-vanishing over time (Kwag et al., 2 Jun 2026). The relevant per-group proxy is

$\eta \lambda = 2(1+\beta),$0

which combines alignment and magnitude.

That work further shows that geometric atypicality alone is insufficient. Input-outliers constructed with coherent displacement along the class-centroid difference $\eta \lambda = 2(1+\beta),$1 can dominate the unstable mode and benefit under EoS, whereas random orthogonal displacement at the same distance destroys directional coherence and removes the advantage (Kwag et al., 2 Jun 2026). Under cross-entropy, confidently correct groups undergo gradient saturation, so even if alignment remains high, their curvature influence can collapse; output-outliers with persistent gradients then become the principal beneficiaries.

A separate line of work connects large-step EoS dynamics to feature learning. In a simplified two-layer ReLU mean model for sparse coding classification, there is a sharp phase transition at

$\eta \lambda = 2(1+\beta),$2

Below this threshold, the limiting bias remains near zero; above it, the model enters an EoS regime and learns substantially negative first-layer biases, i.e. threshold-like neurons (Ahn et al., 2022). The analysis identifies a concrete mechanism by which operation near the stability boundary can alter inductive bias rather than merely accelerate training.

For SAM, the optimizer’s smaller, gradient-dependent edge provides an explicit curvature-control mechanism. Across MNIST, CIFAR10, and a small Transformer LLM, the Hessian operator norm tracks SAM’s predicted edge and remains substantially below the GD threshold $\eta \lambda = 2(1+\beta),$3, while training loss stays comparable and curvature is markedly flatter (Long et al., 2023). This is consistent with, though not by itself sufficient to prove, the commonly stated link between flatter solutions and improved generalization.

6. Broader formulations and open questions

The EoS concept has continued to broaden beyond full-batch Euclidean GD. In reservoir computing, the ES$\eta \lambda = 2(1+\beta),$4N architecture combines a nonlinear reservoir with an orthogonal linear reservoir,

$\eta \lambda = 2(1+\beta),$5

and its Jacobian spectrum is provably confined to an annulus around the circle of radius $\eta \lambda = 2(1+\beta),$6. For small $\eta \lambda = 2(1+\beta),$7, the maximum local Lyapunov exponent satisfies $\eta \lambda = 2(1+\beta),$8, so the dynamics evolve close to the edge-of-chaos regime by design while retaining the Echo State Property under $\eta \lambda = 2(1+\beta),$9 (Ceni et al., 2023).

In kernel associative memory, the phrase is used in an information-geometric sense. There, the Ridge of Optimization is identified with an EoS at which the Fisher Information Matrix

$\lambda_{\max}\approx (2+2\beta)/\eta$0

approaches singularity, with $\lambda_{\max}\approx (2+2\beta)/\eta$1 and a strongly concentrated spectrum. The apparent antagonism of large Euclidean gradients and stable optimization is then interpreted as Dual Equilibrium: the Riemannian natural gradient is small even when Euclidean gradients are large (Tamamori, 28 Nov 2025).

Several open problems recur across the literature. One is to prove that optimizers such as SAM operate at their predicted edges under broad, realistic conditions rather than only in local quadratic approximations (Long et al., 2023). Another is to extend deterministic self-stabilization theory to richer stochastic settings, multiple unstable eigenvalues, negative curvature, and adaptive preconditioners (Damian et al., 2022, Liao et al., 22 Apr 2026). Mini-batch optimization remains particularly incomplete: the empirical distinction between full-batch sharpness, batch sharpness, and geometry-aware sharpness is now clear, but a unified theory across SGD, adaptive methods, and non-Euclidean descent is still unsettled (Islamov et al., 5 Mar 2026, Liao et al., 22 Apr 2026). Large-scale domains are similarly open: selectivity across data groups has been established on controlled CIFAR-10 prototypes, but whether analogous mechanisms dominate in ImageNet-scale vision or modern NLP systems remains unresolved (Kwag et al., 2 Jun 2026).

The common thread across these formulations is that EoS is not simply a failure of smoothness assumptions. It is a structured near-critical regime in which discrete-time instability, higher-order geometry, and optimizer-specific curvature interact to produce persistent oscillations, constrained sharpness, and, in several settings, a distinctive implicit bias over features, groups, or solution geometry (Cohen et al., 2021, Litman, 22 Apr 2026).