---
title: Edge of Stability in Deep Learning
url: https://www.emergentmind.com/topics/edge-of-stability
type: topic
---

# Edge of Stability in Deep Learning

In contemporary optimization theory for deep learning, the **Edge of Stability** (EoS) denotes the regime in which discrete-time training dynamics operate near the curvature threshold at which a local quadratic model would cease to predict one-step loss decrease. For full-batch gradient descent (GD) with step size $\eta$, this threshold is set by the largest Hessian eigenvalue, or sharpness, and empirically training often exhibits **progressive sharpening** until $\lambda_{\max}(\nabla^2 L)$ approaches $2/\eta$, after which the loss becomes non-monotone on short time scales yet continues to decrease over longer horizons [2103.00065]. Subsequent work has recast EoS as a broader phenomenon that depends on optimizer geometry, stochasticity, curvature control, and data structure, rather than as a peculiarity of vanilla GD alone [2209.15594], [2603.05002], [2604.21016].

## 1. Classical definition and discrete-time stability boundary

For GD,
$$
\theta_{t+1}=\theta_t-\eta \nabla L(\theta_t),
$$
the local quadratic approximation yields the classical stability condition
$$
0<\eta \lambda_i<2
$$
along each Hessian eigenmode, so the sharpest mode governs the threshold:
$$
\lambda_{\max}(\nabla^2 L(\theta_t)) \approx \frac{2}{\eta}.
$$
In the empirical regime identified in neural-network training, sharpness rises during early optimization and then hovers near this value, while the loss shows oscillations or spikes at the iterate scale but still decreases over long horizons [2103.00065].

The same logic extends to momentum. For heavy-ball momentum on a scalar quadratic mode,
$$
x_{t+1}=(1-\eta\lambda+\beta)x_t-\beta x_{t-1},
$$
the divergence boundary becomes
$$
\eta \lambda = 2(1+\beta),
$$
so the corresponding EoS threshold is $\lambda_{\max}\approx (2+2\beta)/\eta$ [2307.04210]. Empirical studies in supervised learning and off-policy reinforcement learning use precisely this threshold to identify whether the dynamics have entered an EoS regime.

Two features are central to the modern use of the term. First, EoS is a **discrete-time** phenomenon: gradient flow does not depend on $\eta$ and therefore cannot express the threshold $\eta \lambda_{\max}\approx 2$. Second, the defining symptom is not immediate divergence but a sustained near-critical regime in which curvature remains close to the nominal stability boundary while optimization still progresses [2103.00065], [2606.18080].

## 2. Mechanistic accounts of why training does not simply diverge

A major theoretical development is the **self-stabilization** picture. In this view, once GD becomes unstable along the top Hessian eigenvector $u(\theta)$, third-order structure pushes the trajectory in a sharpness-reducing direction. For simple top eigenvalue,
$$
\nabla S(\theta)=\nabla^3 L(\theta)(u(\theta),u(\theta)),
$$
so the same cubic tensor that is absent from a purely quadratic local model controls the first-order evolution of sharpness itself [2209.15594]. This leads to a reduced two-dimensional description in coordinates aligned with $u$ and $\nabla S$, and to the interpretation that GD near EoS implicitly follows projected gradient descent on the constraint
$$
S(\theta)\le \frac{2}{\eta}
$$
rather than unconstrained descent on $L$ alone [2209.15594].

A distinct but complementary explanation is given by the **edge coupling**
$$
A_\eta(x,y)=L(x)+L(y)-\frac{1}{2\eta}\|x-y\|^2.
$$
Its partial criticality condition reproduces the GD update, and differencing that condition yields the exact step recurrence
$$
d_{k+1}=(I-\eta \bar H_k)d_k,
$$
while a second-order expansion yields the loss-change identity
$$
L(w_{k+1})-L(w_k)= -\frac{\|d_k\|^2}{2\eta}(2-\eta \tilde r_k).
$$
Summing this identity telescopically forces a weighted average of directional curvatures toward $2/\eta$, and a mean-value argument localizes those directional curvatures to actual Hessian values along each step segment [2604.20446]. In that formulation, EoS is not merely a local stability heuristic but the consequence of an exact global identity along the GD trajectory.

Continuous-time effective descriptions have also been proposed. **Edge Flow** decomposes the dynamics into a slowly drifting center $\bar w_t$, a unit oscillation direction $u_t$, and an oscillation magnitude $x_t$, with
$$
\frac{d\bar w_t}{dt}= -\frac12\big[\nabla L(\bar w_t+x_tu_t)+\nabla L(\bar w_t-x_tu_t)\big],
$$
$$
\frac{du_t}{dt}=2\big[\nabla^2L(\bar w_t)-(u_t^\top \nabla^2L(\bar w_t)u_t)I\big]u_t,
$$
$$
\frac{dx_t}{dt}=\big[S(\bar w_t)-2/\eta\big]x_t.
$$
The amplitude equation makes the threshold explicit, while the symmetrized center dynamics creates a negative feedback loop that reduces sharpness when oscillations grow [2606.18080]. A related free-energy description instead tracks the slow mean trajectory and the covariance $\Sigma$ of fast oscillations via
$$
F(\theta,\Sigma)=E(\theta)+\frac{\eta}{2}\nabla^2E(\theta):\Sigma,
$$
which is dissipated when $\|\nabla^2E\|_\infty\le 2/\eta$ [2606.05326].

## 3. Optimizer- and geometry-dependent generalizations

The threshold $2/\eta$ is not universal in its Euclidean GD form. Several works reformulate EoS so that the relevant curvature quantity matches the optimizer.

| Setting | Stability quantity | Edge condition |
|---|---|---|
| GD | $\lambda_{\max}(\nabla^2 L)$ | $\lambda_{\max}\approx 2/\eta$ |
| Heavy-ball momentum | $\lambda_{\max}(\nabla^2 L)$ | $\lambda_{\max}\approx (2+2\beta)/\eta$ |
| SAM | $\|H\|_{op}$ with gradient dependence | $\|H\|_{op}< \frac{\|g\|}{2\rho}\!\left(\sqrt{1+\frac{8\rho}{\eta\|g\|}}-1\right)$ |
| Non-Euclidean GD | generalized sharpness $S^{\|\cdot\|}$ | $S^{\|\cdot\|}\approx 2/\eta$ |
| Adam-type methods | preconditioned sharpness $\lambda_{\max}(P^{-1}H)$ | empirical AEoS at $c/\eta$ in a fixed setting |
| Zeroth-order GD | mean-square stability depends on full spectrum | trace-based band around $2/\eta$ |

For **Sharpness-Aware Minimization** (SAM), the update
$$
w_{t+1}=w_t-\eta \nabla \ell\!\left(w_t+\rho \frac{\nabla \ell(w_t)}{\|\nabla \ell(w_t)\|}\right)
$$
modifies the stability boundary itself. Under the same local quadratic model used for GD, the SAM edge becomes
$$
\|H\|_{op}< \frac{\|g\|}{2\rho}\left(\sqrt{1+\frac{8\rho}{\eta\|g\|}}-1\right),
$$
so unlike the GD threshold it depends on the gradient norm and the perturbation radius $\rho$ [2309.12488]. This dependence implies that SAM’s admissible curvature typically tightens as training progresses and $\|g\|$ shrinks.

For **non-Euclidean descent**, the relevant curvature is
$$
S^{\|\cdot\|}(w)=\max_{d\neq 0}\frac{d^\top \nabla^2L(w)d}{\|d\|^2},
$$
equivalently the induced operator norm $\|\nabla^2L(w)\|_{\|\cdot\|\to \|\cdot\|_*}$ in the symmetric positive-semidefinite case. This recovers the standard Hessian spectral norm in the Euclidean geometry, but it also covers $\ell_\infty$-descent, block coordinate descent, spectral GD, and normalized methods such as SignGD and Muon without momentum, all of which empirically exhibit progressive sharpening and oscillations near $2/\eta$ when sharpness is measured in the optimizer’s own geometry [2603.05002].

For **adaptive methods**, the relevant quantity is not raw Hessian sharpness but **preconditioned sharpness**. In Adam, with diagonal preconditioner $P_t=\operatorname{diag}(\sqrt{\hat v_t}+\epsilon)$, the stability-relevant matrix is $P_t^{-1}H(\theta_t)$. In one empirical setting, non-private full-batch Adam equilibrates near a threshold $c/\eta$ with $c=38$, while raw $\lambda_{\max}(H)$ can continue to increase after the preconditioned sharpness has plateaued [2512.19019].

For **zeroth-order methods** based on the two-point estimator, mean-square stability depends on the entire Hessian spectrum rather than only on $\lambda_{\max}$. For ZO-GD, the critical step size is the unique $\eta$ satisfying
$$
\sum_{i=1}^d \frac{\eta \lambda_i}{2(1-\eta \lambda_i)}=1
$$
with $\eta \lambda_{\max}(H)<1$, and it is bounded by
$$
\frac{2}{\operatorname{Tr}(H)+2\lambda_{\max}(H)} \le \eta^\star_{ms} \le \frac{2}{\operatorname{Tr}(H)}.
$$
Empirically, full-batch ZO-GD, ZO-GDM, and ZO-Adam stabilize near the corresponding trace-based mean-square stability bands rather than a purely top-eigenvalue threshold [2604.14669].

## 4. Stochasticity, minibatches, and domain-specific manifestations

Mini-batch SGD does not simply replicate the full-batch EoS picture. A stochastic self-stabilization analysis predicts that full-batch sharpness should stabilize **below** $2/\eta$, with the gap
$$
\Delta S = \frac{\eta \beta \sigma_u^2}{4\alpha},
$$
where $\alpha=-\langle \nabla L,\nabla S\rangle$ is the progressive-sharpening rate, $\beta=\|P_u^\perp \nabla^3L(u,u)\|^2$ is the self-stabilization strength, and $\sigma_u^2=u^\top \Sigma_b u$ is the gradient-noise variance projected onto the top Hessian eigenvector [2604.21016]. This recovers full-batch GD when $\sigma_u^2=0$ and predicts flatter solutions as batch size decreases.

In **reinforcement learning**, EoS is present but loss-dependent. In off-policy Q-learning, DQN with a Huber loss shows clear offline EoS: the leading Hessian eigenvalue rises to the heavy-ball threshold $(2+2\beta)/\eta$ and fluctuates around it. C51, which uses a cross-entropy loss over return distributions, instead shows weak or absent offline EoS and, in online training, early threshold overshoot followed by a later sharpness decrease [2307.04210]. This differentiates the role of regression-like and cross-entropy curvature geometries in RL.

In **differentially private training**, clipping and Gaussian noise modify both the trajectory and the stability regime. Private GD and private Adam generally show reduced sharpness, delayed breakeven, and damped oscillations relative to their non-private counterparts, but at large learning rates and large privacy budgets they can still approach or exceed the corresponding non-private boundaries [2512.19019]. This suggests that privacy mechanisms alter, but do not eliminate, edge-like regimes.

For **deep linear networks**, training beyond EoS has been analyzed with unusually fine resolution. In deep matrix factorization, loss oscillations follow a **period-doubling route to chaos**, and the oscillatory subspace dimension is determined exactly by the learning rate. If
$$
S_i=L\sigma_{*,i}^{2-2/L},
$$
then the top-$p$ singular directions oscillate when
$$
\frac{2}{S_p}<\eta<\frac{2}{S_{p+1}}.
$$
In that regime, oscillations are confined to the leading feature subspace, and the balancing gap between layer singular values decays monotonically to zero beyond EoS [2502.20531].

## 5. Selectivity, implicit bias, and generalization

EoS is not only a global statement about an optimizer’s trajectory; it can selectively redistribute learning across the data distribution. A recent causal branching analysis shows that groups benefit from remaining at EoS only if two conditions hold jointly: their aggregate gradient must align with the top Hessian eigendirection, and their gradient magnitude must remain non-vanishing over time [2606.04212]. The relevant per-group proxy is
$$
C_j=(g_j^\top v_1)^2=\|g_j\|^2 \cos^2(g_j,v_1),
$$
which combines alignment and magnitude.

That work further shows that **geometric atypicality alone is insufficient**. Input-outliers constructed with coherent displacement along the class-centroid difference $v_{\mathrm{diff}}$ can dominate the unstable mode and benefit under EoS, whereas random orthogonal displacement at the same distance destroys directional coherence and removes the advantage [2606.04212]. Under cross-entropy, confidently correct groups undergo gradient saturation, so even if alignment remains high, their curvature influence can collapse; output-outliers with persistent gradients then become the principal beneficiaries.

A separate line of work connects large-step EoS dynamics to feature learning. In a simplified two-layer ReLU mean model for sparse coding classification, there is a sharp phase transition at
$$
\eta_c=\frac{8\pi}{d^2}.
$$
Below this threshold, the limiting bias remains near zero; above it, the model enters an EoS regime and learns substantially negative first-layer biases, i.e. threshold-like neurons [2212.07469]. The analysis identifies a concrete mechanism by which operation near the stability boundary can alter inductive bias rather than merely accelerate training.

For SAM, the optimizer’s smaller, gradient-dependent edge provides an explicit curvature-control mechanism. Across MNIST, CIFAR10, and a small Transformer language model, the Hessian operator norm tracks SAM’s predicted edge and remains substantially below the GD threshold $2/\eta$, while training loss stays comparable and curvature is markedly flatter [2309.12488]. This is consistent with, though not by itself sufficient to prove, the commonly stated link between flatter solutions and improved generalization.

## 6. Broader formulations and open questions

The EoS concept has continued to broaden beyond full-batch Euclidean GD. In **reservoir computing**, the ES$^2$N architecture combines a nonlinear reservoir with an orthogonal linear reservoir,
$$
x[t]=\beta \phi(\rho W_r x[t-1]+\omega W_{in}u[t])+(1-\beta)Ox[t-1],
$$
and its Jacobian spectrum is provably confined to an annulus around the circle of radius $1-\beta$. For small $\beta$, the maximum local Lyapunov exponent satisfies $\Lambda\approx -\beta$, so the dynamics evolve close to the edge-of-chaos regime by design while retaining the Echo State Property under $\|\rho W_r\|<1$ [2308.02902].

In **kernel associative memory**, the phrase is used in an information-geometric sense. There, the Ridge of Optimization is identified with an EoS at which the Fisher Information Matrix
$$
G(\alpha)=K D(\alpha) K
$$
approaches singularity, with $\lambda_{\min}(G)\to 0$ and a strongly concentrated spectrum. The apparent antagonism of large Euclidean gradients and stable optimization is then interpreted as **Dual Equilibrium**: the Riemannian natural gradient is small even when Euclidean gradients are large [2511.23083].

Several open problems recur across the literature. One is to prove that optimizers such as SAM operate at their predicted edges under broad, realistic conditions rather than only in local quadratic approximations [2309.12488]. Another is to extend deterministic self-stabilization theory to richer stochastic settings, multiple unstable eigenvalues, negative curvature, and adaptive preconditioners [2209.15594], [2604.21016]. Mini-batch optimization remains particularly incomplete: the empirical distinction between full-batch sharpness, batch sharpness, and geometry-aware sharpness is now clear, but a unified theory across SGD, adaptive methods, and non-Euclidean descent is still unsettled [2603.05002], [2604.21016]. Large-scale domains are similarly open: selectivity across data groups has been established on controlled CIFAR-10 prototypes, but whether analogous mechanisms dominate in ImageNet-scale vision or modern NLP systems remains unresolved [2606.04212].

The common thread across these formulations is that EoS is not simply a failure of smoothness assumptions. It is a structured near-critical regime in which discrete-time instability, higher-order geometry, and optimizer-specific curvature interact to produce persistent oscillations, constrained sharpness, and, in several settings, a distinctive implicit bias over features, groups, or solution geometry [2103.00065], [2604.20446].

Source: https://www.emergentmind.com/topics/edge-of-stability