---
title: Sharpness-Aware Learnability (SAL)
url: https://www.emergentmind.com/topics/sharpness-aware-learnability-sal
type: topic
---

# Sharpness-Aware Learnability (SAL)

Sharpness-Aware Learnability, or **SAL** (*Editor’s term*), denotes an interpretive research perspective in which learnability is studied through the local geometry induced by sharpness-aware optimization: whether learning dynamics, generalization, adaptation, or forgetting improve when training is made robust to parameter perturbations or when the degree of sharpness-awareness is itself learned, scheduled, or decomposed. The cited literature does not use SAL as a standardized name, but it repeatedly addresses closely related questions in supervised learning, self-supervision, meta-learning, continual learning, machine unlearning, hybrid modeling, and spiking neural networks [2408.08222; 2305.10252; 2206.03996].

## 1. Scope and conceptual status

SAL is best treated as an umbrella over several technical programs rather than as a single canonical framework. One line starts from **Sharpness-Aware Minimization (SAM)** and asks how neighborhood-robust objectives alter optimization and generalization. A second line asks how the **strength, frequency, or parameterization** of sharpness-aware perturbations should be chosen. A third line studies whether sharpness-aware objectives improve **downstream learnability** in settings where the target is not ordinary ERM, such as linear evaluation after self-supervised pretraining, bilevel meta-learning, or retain-versus-forget optimization [2408.08222; 2203.09962; 2305.10252; 2206.03996].

A recurring feature of this literature is that “sharpness-awareness” is operationalized through local perturbation robustness, but the object whose learnability is at stake varies by domain. In self-supervised learning it is the downstream classifier induced by a representation; in meta-learning it is the adapted task predictor after few-shot updates; in unlearning it is retained utility under active forget interference; in hybrid models it is the recoverability of scientific parameters; and in spiking networks it is the transfer from surrogate-forward training to hard-spike deployment [2305.10252; 2206.03996; 2506.13715; 2602.06837; 2603.18039].

| Direction | Representative mechanism | Papers |
|---|---|---|
| Canonical sharpness-aware training | Neighborhood worst-case loss and normalized gradient perturbation | [2408.08222], [2309.12488] |
| Adaptive sharpness control | Learned radius, randomized schedules, adaptive stepsizes | [2408.08222], [2203.09962], [2505.24399], [2011.05348] |
| Learnability theory under sharpness | Downstream SSL bounds, bilevel bounds, noise-learning separation | [2305.10252], [2206.03996], [2310.07269] |
| Task-specific SAL instantiations | CL decomposition, unlearning, hybrid models, SNN transfer | [2601.07636], [2506.13715], [2602.06837], [2603.18039] |

This suggests that SAL is less a settled doctrine than a family of questions about how local robustness criteria reshape what is learnable, how much signal is needed, and which directions of parameter change are beneficial or harmful.

## 2. Foundational objectives and sharpness notions

The common starting point is the SAM min-max problem
\[
\min_{\theta} \max_{\|\epsilon\| \leq \rho } L(D^{tr}; \theta + \epsilon),
\]
with practical update
\[
\theta_{t+1} = \theta_t - \eta \nabla L\!\left( D^{tr}; \theta_t + \rho \frac{\nabla L(D^{tr};\theta_t)} {\|\nabla L(D^{tr};\theta_t)\|} \right),
\]
so the perturbation radius \(\rho\) determines the neighborhood over which local sharpness is probed [2408.08222]. The literature repeatedly treats \(\rho\) as the critical control variable: if it is too small, sharpness is weakly penalized; if too large, optimization can be distorted or over-regularized [2408.08222].

A more dynamical interpretation is given by the SAM edge-of-stability analysis. Under a local quadratic model, gradient descent has the usual stability threshold \(\|H\|_{op} \approx 2/\eta\), whereas SAM yields a moving threshold that depends on \(\eta\), \(\rho\), and the current gradient norm \(\|g\|\). The resulting “SAM-edge” shrinks as \(\|g\|\) decreases, so the admissible curvature becomes progressively stricter during training [2309.12488]. In SAL terms, this makes sharpness-awareness a trajectory constraint rather than merely a property of final minima.

Several later papers argue that the fixed-radius formulation is itself not unique. “Loss-Equated SAM” replaces fixed parameter-space radius by a fixed loss-space budget \(\sigma\), using
\[
\rho_t = \frac{\sigma}{\|g_c\|_2+\varrho},
\]
and derives the surrogate
\[
L_S(w+\tilde\epsilon_t) \approx L_S(w) + \sigma + \frac{1}{2}\tilde\epsilon_t^\top H(w)\tilde\epsilon_t.
\]
The explicit claim is that standard SAM is dominated by the first-order term \(\rho\|\nabla L_S(w)\|_2\), whereas LE-SAM removes that variable first-order contribution and shifts the effective signal toward curvature [2605.10183]. This suggests that SAL depends not only on whether perturbations are used, but on how sharpness is parameterized.

A different criticism comes from the analysis of SAM’s approximation gap. “1st-Order Magic” argues that more precise approximations of the original SAM objective can degrade generalization performance, and that practical benefits may arise from the first-order boundary-point approximation rather than faithful optimization of the exact worst-case neighborhood loss [2411.01714]. That claim weakens any simple identification of SAL with exact local max-loss minimization. Under this view, the learnability effect may come from the algorithmic surrogate actually optimized, not only from the idealized min-max objective.

## 3. Learning or controlling the sharpness mechanism

A major SAL theme is that sharpness-awareness need not be fixed. In **LETS**, the perturbation radius itself becomes a learned hyperparameter through bilevel optimization:
\[
\underset{\rho \in (0, \infty)}{\min}\quad \frac{1}{2}\left(L(D^{vl};\theta^\star(\rho)) - L(D^{tr};\theta^\star(\rho))\right)^2
\]
subject to a SAM lower-level problem [2408.08222]. LETS learns a single global radius, enforces positivity via \(\rho=\exp(\nu)\), and alternates weight updates with radius updates. On CIFAR-100 with WRN-28-10 it reports \(83.42 \to 84.78\) for LETS-SAM over SAM, and on ImageNet with ResNet-50 it reports \(77.47 \to 77.67\) [2408.08222]. This is one of the clearest direct embodiments of SAL as “learning how much sharpness-awareness should be enforced.”

A second route is to adapt the **frequency** of sharpness-aware steps rather than the radius. **Randomized Sharpness-Aware Training (RST)** performs a Bernoulli trial at each iteration and chooses SGD or SAM with probability \(p(t)\), yielding expected propagation count \(1+p_t\) per iteration [2203.09962]. Its generalized form, **G-RST**, decouples update frequency from effective sharpness regularization and is reported to outperform SAM in most cases while saving \(50\%\) extra computation cost [2203.09962]. This suggests that sharpness-aware learnability may be stage-dependent and intermittent rather than tied to “SAM every step.”

A third route is to adapt optimizer scale directly. **SALR** computes a local sharpness proxy and sets
\[
\eta_k = \eta_0 \frac{S}{\operatorname{Median}\{\widehat{S}_i\}_{i=1}^k},
\]
so learning rate increases in sharp regions and decreases in flatter ones [2011.05348]. The method is reported to improve generalization, converge faster, and drive solutions to significantly flatter regions [2011.05348]. **LightSAM** pursues a related goal from a different angle: it uses AdaGrad-Norm, AdaGrad, and Adam inside SAM so that perturbation radius and learning rate are adaptive, and claims that under weak assumptions it can converge ideally with any choices of perturbation radius and learning rate, achieving parameter-agnostic behavior [2505.24399].

| Method | Controlled quantity | Reported implication |
|---|---|---|
| LETS | Global radius \(\rho\) via squared generalization gap | Radius becomes learnable rather than hand-tuned [2408.08222] |
| RST / G-RST | Probability \(p(t)\) of sharpness-aware steps | Intermittent sharpness-awareness can preserve gains with lower cost [2203.09962] |
| SALR | Learning rate from local sharpness | Escapes sharper regions by enlarging steps there [2011.05348] |
| LightSAM | Adaptive perturbation radius and learning rate | SAM becomes parameter-agnostic under the paper’s assumptions [2505.24399] |

Taken together, these methods broaden SAL from a fixed objective into a control problem over **extent**, **timing**, and **optimization scale**.

## 4. Theoretical formulations of learnability under sharpness

The sharpest formal SAL statement in self-supervision appears in **Sharpness & Shift-Aware Contrastive Learning**. For contrastive SSL followed by linear evaluation, the paper proves that downstream supervised loss is upper-bounded by a sharpness-aware empirical InfoNCE term, together with complexity terms and a positive-pair distribution shift term [2305.10252]. Its practical method, SSA-CLR, combines SAM-like perturbation with Fourier-based augmentation and reports linear-evaluation gains from \(93.04\%\) to \(94.08\%\) on CIFAR-10 and from \(67.90\%\) to \(71.90\%\) on CIFAR-100 over SimCLR [2305.10252]. In SAL language, that result ties flatter self-supervised encoders to better downstream linear learnability.

In bilevel learning, **Sharp-MAML** imports SAM into both the lower adaptation and upper meta-update. For the one-step MAML objective \(F(\theta)\), the paper proves
\[
\frac1T\sum_{t=1}^T \mathbb E\big[\|\nabla F(\theta^t)\|^2\big] = \mathcal O\!\left(\frac1{\sqrt T}\right),
\]
and gives a PAC-Bayes-style bound of the form
\[
F(\hat{\theta};\mathcal P) \le \max_{\|\epsilon\|_2\le \alpha}F(\hat{\theta}+\epsilon;\mathcal D) +\gamma_A +\text{complexity term},
\]
so meta-generalization depends explicitly on neighborhood empirical performance and lower-level stability [2206.03996]. Empirically, Mini-Imagenet 5-way 1-shot improves from \(47.13\%\) for reproduced MAML to \(50.28\%\) for Sharp-MAML\(_{\rm both}\), and 5-shot improves from \(62.20\%\) to \(65.04\%\) [2206.03996]. This makes SAL concrete for meta-learning: a learnable initialization is one that remains good under local perturbations throughout bilevel adaptation.

A different theoretical line studies sharpness-aware learnability in a noisy feature-learning regime. For a two-layer convolutional ReLU network with label-flip noise, the paper “Why Does Sharpness-Aware Minimization Generalize Better Than SGD?” shows that SGD achieves low test error only when
\[
n\|\bmu\|_2^4 \ge C_1 d P^4 \sigma_p^4,
\]
whereas SAM can still achieve low test error in the weaker-signal regime
\[
\Omega(1)\le \|\bmu\|_2 \le C_3 d^{1/4}n^{-1/4}P\sigma_p
\]
[2310.07269]. Its mechanism is not Hessian smoothing but perturbation-induced deactivation of noise-fitting neurons in the early stage of training. This implies that SAL can be a theory of **feature selection under perturbation**, not merely of converging to flatter minima.

These results share a common structure: local robustness enters the theory not only as a geometric regularizer, but as a constraint on which signals can accumulate reliably during learning.

## 5. Domain-specific instantiations

In continual learning, **FLAD** decomposes sharpness-aware perturbations into a gradient-aligned component and a stochastic-noise component, then keeps only the noise component [2601.07636]. The method defines perturbations
\[
\delta_0=\rho\frac{\hat{g}_{B_t}-\sigma m_t}{\| \hat{g}_{B_t}-\sigma m_t\|+c},
\qquad
\delta_1=\rho\frac{\nabla\|\hat{g}_{B_t}\|-\sigma n_t}{\| \nabla\|\hat{g}_{B_t}\|-\sigma n_t\|+c},
\]
and uses the update \(w=w-\eta(g_0+\gamma g_1)\) [2601.07636]. Full FLAD requires \(2\) forward and \(4\) backward passes per iteration, but the paper reports that applying FLAD in only \(10\%-20\%\) of epochs already yields substantial gains over vanilla SGD, and that using FLAD for only \(30\%\) of epochs reduces overhead by at least \(50\%\) relative to full-method training [2601.07636]. SAL here becomes directional: not every perturbation direction is equally beneficial for continual generalization.

In hybrid scientific machine learning, the paper “Learning Deep Hybrid Models with Sharpness-Aware Minimization” applies SAM only to the ML parameters \(\phi\) in
\[
h_{\theta,\phi}(x)=g_\phi(x,f_\theta(x)),
\]
leaving scientific parameters \(\theta\) unperturbed [2602.06837]. The induced objective is effectively
\[
\min_{\theta,\phi}\max_{\|\epsilon_\phi\|\le \rho_\phi} L_S(\theta,\phi+\epsilon_\phi).
\]
The reported effect is improved scientific-parameter recoverability without bespoke architecture-specific regularizers. On the wind-tunnel task, \(\theta\)-error moves from \(69.6\) for ERM to \(2.15\) for SAM and \(1.90\) for FSAM; on the light-tunnel task, cosine similarity moves from \(0.76\) for ERM to \(0.98\) for SAM, ASAM, and FSAM [2602.06837]. This suggests a variant of SAL in which flatness in the neural subspace improves learnability of the intended scientific decomposition.

In spiking neural networks, **SAST** applies SAM to a smooth surrogate-forward SNN so that the training gradient is exact for the optimized auxiliary model. The surrogate objective
\[
\tilde L_S(w) = \frac{1}{n}\sum_{i=1}^n \ell\big(\tilde f_w(x^{(i)}_{1:T}),y^{(i)}\big)
\]
is then trained with the SAM objective
\[
\tilde L_{\mathrm{SAM}}(w) = \max_{\|\epsilon\|_2\le\rho}\tilde L_S(w+\epsilon)
\]
and update
\[
w_{k+1} = w_k - \eta\, g_{B_k'}(w_k+\epsilon_k)
\]
using an independent second minibatch [2603.18039]. The clearest empirical effect is transfer-gap reduction: on N-MNIST, hard-spike accuracy rises from \(65.7\%\) to \(94.7\%\) with best \(\rho=0.30\); on DVS Gesture, it improves from \(31.8\%\) to \(63.3\%\) with best \(\rho=0.40\) [2603.18039]. Here SAL is tied to learnability of a deployed hard-spike model through flatness of a smooth surrogate training objective.

Machine unlearning produces perhaps the strongest interference-based SAL picture. One paper selects parameters for unlearning via the diagonal Hessian of the original learning loss, arguing that the least-sharp directions are the best coordinates to update; on CIFAR-10 targeted poisoning, “SA 1st order” reaches \(86.71 \pm 1.2\) versus \(85.34 \pm 0.74\) for last-layer first-order unlearning and \(82.12 \pm 0.91\) for fine-tuning [2504.06398]. A second paper, “Sharpness-Aware Machine Unlearning,” shows that under retain/forget interference SAM enlarges the signal regime in which retain information remains learnable, but loses its denoising effect on the forget set itself. It then proposes **Sharp MinMax**, which learns retain signals with SAM and unlearns forget signals with sharpness maximization, and reports the best performance across forget-set difficulty levels measured by memorization [2506.13715]. These results make SAL explicitly asymmetric: flatness helps retention and generalization, while sharper fitting can help stringent sample-specific forgetting.

## 6. Empirical picture, controversies, and open directions

Across the literature, SAL is associated with lower generalization gaps, flatter landscapes, or lower curvature proxies, but these benefits are neither uniform nor uncontested. In LETS, smaller validation-versus-training discrepancy is tied to better performance and flatter loss landscapes, yet the method depends on validation data and learns only a single global scalar radius [2408.08222]. In RST and FLAD, intermittent sharpness-awareness often preserves gains with lower overhead, indicating that sharpness control may be important mainly at certain stages rather than continuously [2203.09962; 2601.07636].

A major controversy concerns **what mechanism actually explains SAM**. One line argues that the practical first-order approximation itself is the source of benefit and that more accurate optimization of the original max-loss objective can degrade generalization [2411.01714]. Another line argues that fixed-radius SAM is mismatched to the second-order nature of flatness and should be replaced by fixed loss-budget perturbations, as in LE-SAM [2605.10183]. These positions are compatible only at a high level: both reject a simplistic reading of SAL as mere faithful minimization of the canonical SAM objective.

A second controversy concerns **memorization and privacy**. “On Memorization and Privacy Risks of Sharpness Aware Minimization” reports that SAM’s gains are concentrated on atypical, low-influence-entropy test points that rely more heavily on memorized training support, and that best single-query membership attack accuracy rises under SAM: on CIFAR-100, from \(76.68\%\) for SGD to \(79.09\%\) for SAM; on CIFAR-10, from \(59.05\%\) to \(61.32\%\) [2310.00488]. This implies that sharpness-aware learnability can improve accuracy while redistributing learnability toward memorization-heavy or privacy-sensitive examples.

A third controversy concerns **what aspect of a perturbation is useful**. FLAD argues that the stochastic-noise component, rather than the gradient-aligned component, is the beneficial part of SAM/GAM-style perturbations in continual learning [2601.07636]. The unlearning literature goes further and shows that different subsets of parameters or different sharpness biases may be desirable for retain and forget objectives [2504.06398; 2506.13715]. This weakens any one-size-fits-all notion of sharpness-aware training.

The open directions are correspondingly consistent across papers. Repeatedly identified problems include learning richer sharpness-control structures beyond a single scalar radius, reducing or eliminating dependence on held-out validation signals, characterizing the effect of crude Hessian approximations, adapting schedules online rather than fixing them a priori, and deriving stronger theory that links learned sharpness-control variables to true generalization rather than only to optimization convergence or surrogate criteria [2408.08222; 2203.09962]. A broader implication is that SAL is likely to remain a plural concept: some settings favor flatness, some favor selective sharpness, and many require learning not only model parameters but the geometry of the training rule itself.

Source: https://www.emergentmind.com/topics/sharpness-aware-learnability-sal