---
title: Extragradient-Inspired Sharpness-Aware Minimization
url: https://www.emergentmind.com/topics/extragradient-inspired-sharpness-aware-minimization-eisam
type: topic
---

# Extragradient-Inspired Sharpness-Aware Minimization

Searching arXiv for the target paper and closely related SAM literature to ground the article in current preprints.
Using the arXiv search tool for "2607.06151 Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning" and related "Sharpness-Aware Minimization".
Extragradient-Inspired Sharpness-Aware Minimization (EISAM) is an optimization method for deep learning that extends Sharpness-Aware Minimization (SAM) with an extragradient-inspired two-step update designed to improve generalization, reduce sensitivity to the perturbation radius, and bias training toward flatter minima. It is introduced in "Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning" [2607.06151]. The method combines a prediction step that probes the local loss geometry with a sharpness-aware correction step implemented through a base optimizer such as SGD or Adam. The paper positions EISAM as a modification of SAM rather than a replacement for standard first-order training pipelines, and supports the proposal with theoretical bounds, benchmark experiments, and tuning guidance.

## 1. Optimization objective and conceptual motivation

EISAM is defined against the same sharpness-aware objective that motivates SAM. Given a training set $S$ and empirical risk
$$
F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),
$$
SAM seeks solutions of
$$
\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),
$$
with the inner maximization approximated by
$$
\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.
$$
Within this formulation, the perturbation radius $\rho$ controls how aggressively the optimizer penalizes local sharpness.

EISAM modifies this scheme by inserting a prediction step before the sharpness-aware correction. The stated purpose of this extra lookahead is to “look ahead” in the negative-gradient direction, investigate the geometry of the loss landscape, reduce sensitivity to $\rho$, and bias optimization toward flatter minima. The abstract further states that this yields better generalization performance than SAM and simplifies tuning across diverse settings [2607.06151].

This construction places EISAM at the intersection of two established ideas: sharpness-aware training and extragradient-style lookahead. In the paper’s framing, the lookahead is lightweight rather than a separate higher-order procedure. A plausible implication is that EISAM should be understood less as a new objective than as a new update rule for approximately solving a sharpness-aware objective under improved hyperparameter robustness.

## 2. Algorithmic structure

At iteration $t$, with parameters $w_t\in\mathbb{R}^d$, minibatch $B_t$, prediction step size $s>0$, learning rate $\eta$, and perturbation radius $\rho$, EISAM performs three operations.

First, it computes a prediction step:
$$
y_t \;=\; w_t \;-\; s\,\nabla F_{B_t}(w_t).
$$

Second, it forms a sharpness perturbation at the predicted point:
$$
\epsilon_t \;=\;\rho\;\frac{\nabla F_{B_t}(y_t)}{\|\nabla F_{B_t}(y_t)\|},
$$
which is often approximated for efficiency by using $\nabla F_{B_t}(w_t)$. The perturbed point is then $y_t+\epsilon_t$.

Third, it applies a correction step with the base optimizer:
$$
w_{t+1}
\;=\;
y_t \;-\;\eta\,\nabla F_{B_t}(y_t+\epsilon_t).
$$
The paper states that the final parameter update is carried out by a base optimizer such as SGD or Adam.

The provided pseudocode makes the approximation explicit. One first stores $g_t=\nabla F_{B_t}(w_t)$, computes $y_t=w_t-s\cdot g_t$, sets $\epsilon_t=\rho\cdot g_t/\|g_t\|$, evaluates the sharpness-aware gradient $g_t^{sam}=\nabla F_{B_t}(y_t+\epsilon_t)$, and finally updates with `opt.update(w_t, g_t^{sam}, \eta)`. This sequence clarifies that EISAM retains the operational pattern of SAM while shifting the perturbation and correction to a predicted point.

A central limiting case is also stated explicitly: when $s=0$, EISAM reduces to SAM, recovering SAM’s high $\rho$-sensitivity. This is important both conceptually and diagnostically, because it isolates the extragradient-inspired prediction step as the distinctive mechanism.

## 3. Theoretical analysis

The theoretical development is given under standard convex-optimization assumptions: strong convexity, smoothness, Lipschitz continuity, and Hessian Lipschitzness. Specifically, $f(w,z)$ is assumed to be $\mu$-strongly convex in $w$, $\nabla f$ is $L$-Lipschitz, $|f(w)-f(v)|\le G\|w-v\|$, and
$$
\|\nabla^2F_{B_t}(w)-\nabla^2F_{B_t}(v)\|\le K\|w-v\|.
$$
The paper states that these assumptions guarantee bounded curvature and control higher-order remainders [2607.06151].

For uniform stability, the analysis considers two runs on datasets $S$ and $S'$ differing in one sample, with trajectories $\{w_t\}$ and $\{v_t\}$ and gap $\delta_t=\|w_t-v_t\|$. Lemmas 4.1–4.2 establish contraction for identical minibatches and a small drift for differing ones, leading to Theorem 4.1:
$$
\varepsilon_{\rm gen}
\;=\;
\mathbb{E}\big[F_\mathcal{D}(w_T) - F_S(w_T)\big]
\;\le\;
\frac{2G^2(\mu+L)}{n\,\mu\,L\,(1 - s\mu)^2}
\;+\;
O\Big(\bigl(\tfrac{\eta\,G^2(\mu+L)}{n\,\mu\,L\,(1 - s\mu)^2}\bigr)^{3/2}\Big).
$$
Compared to vanilla SAM, whose leading factor is stated to be $1$ in place of $(1-s\mu)^2$, EISAM tightens the leading term by $(1-s\mu)^2$.

For optimization error, Lemma 4.3 gives one-step progress under strong convexity:
$$
F_{B_t}(w_{t+1}) - F_{B_t}(w^*)
\;\le\;
\bigl(1 - 2\eta\mu(1 - s\mu)\bigr)\bigl[F_{B_t}(w_t) - F_{B_t}(w^*)\bigr]
+ \tfrac{\eta^2LG^2}{2}
+ O(\eta\,s^2G^3 + \eta\,\rho G^2).
$$
Theorem 4.2 then yields
$$
\varepsilon_{\rm opt}
=\;
\mathbb{E}[F_S(w_T) - F_S(w_S^*)]
\;\le\;
\frac{\eta\,L\,G^2}{4\,\mu\,(1 - s\mu)}
+ O(\eta\,s^2G^3 + \eta\,\rho G^2).
$$

Combining the generalization and optimization terms, Theorem 4.3 bounds the expected excess risk:
$$
\varepsilon_{\rm exc}
\;=\;\varepsilon_{\rm gen} + \varepsilon_{\rm opt}
\;\le\;
\underbrace{\frac{2G^2(\mu+L)}{n\,\mu\,L\,(1 - s\mu)^2}}_{\rm gen.}
\;+\;
\underbrace{\frac{\eta\,L\,G^2}{4\,\mu\,(1 - s\mu)}}_{\rm opt.}
+\;O\!\Bigl(\bigl(\tfrac{\eta G^2(\mu+L)}{n\mu L(1-s\mu)^2}\bigr)^{3/2}
+\eta s^2G^3+\eta\rho G^2\Bigr).
$$
The paper interprets this as showing that a moderate $s$ improves generalization with only a mild effect on the optimization rate, while $\rho$ appears only in higher-order terms. This is the formal basis for the claim that EISAM has reduced sensitivity to the perturbation radius.

A nonconvex extension is also given. Under $L$-smoothness and bounded gradient variance $\sigma^2$, Theorem 4.4 states for the averaged iterate $\bar w_T=\tfrac1T\sum_{t=1}^T w_t$ that
$$
\mathbb{E}[F(\bar w_T)-F(w^*)]
=O\!\Bigl(\frac{F(w_1)-F^*}{\eta T(1 - sL - \rho L)}\Bigr)
+O\bigl(\eta L\sigma^2 + s^2L\sigma^2 + \rho^2L + \sigma\sqrt{\tfrac{d\log T}{n}}\bigr).
$$
This extends the argument beyond the strongly convex setting, but the theoretical guarantees remain conditional on the assumptions stated in the paper.

## 4. Empirical performance across tasks

The experimental program spans image classification, vision transformers, natural language processing, object detection, and segmentation. The reported baselines are SGD, Adam, SAM, ASAM, GSAM, FSAM, and EISAM. The setup includes CIFAR-10/100, ImageNet-1K, ViT with patch-sizes 8/16, BOOLQ with T5-small, COCO-2017 and LVIS v1.0 with Faster R-CNN w/ ResNet-50, and ISIC2018 with U-Net [2607.06151].

The CIFAR experiments report mean test accuracy over 3 seeds. On CIFAR-10, EISAM attains **96.84±0.09** on ResNet-18, **97.13±0.06** on ResNet-50, and **97.63±0.12** on WideRes-28-10, exceeding the corresponding reported SGD, SAM, ASAM, GSAM, and FSAM values. On CIFAR-100, EISAM reports **82.22±0.14** on ResNet-18 and **83.64±0.17** on ResNet-50, again above the listed baselines.

On ImageNet-1K, the reported top-1 accuracies are **70.89±0.15** for ResNet-18 and **78.01±0.07** for ResNet-50, which are higher than the corresponding figures given for SGD, SAM, ASAM, GSAM, and FSAM. For ViT-S-8 with CutMix on CIFAR-10, EISAM reaches **86.42±0.05**, compared with SAM at 86.26±0.19 and Adam at 86.30±0.25.

The paper also reports results beyond image classification. On BOOLQ, EISAM gives validation/test accuracy **77.19/78.64**, compared with Adam at 76.17/75.18, SAM at 77.14/76.02, and FSAM at 76.19/78.41. For object detection, the COCO AP progression is reported as SGD 22.23, SAM 27.53, and EISAM **27.65**; on LVIS AP it is SGD 10.37, SAM 11.74, and EISAM **11.76**. For segmentation on ISIC2018, the U-Net score is reported as Adam 1.2698, FSAM 1.2973, and EISAM 1.2735, with best val **89.11%**.

| Setting | Baseline values reported | EISAM |
|---|---|---|
| CIFAR-10, ResNet-50 | SAM 96.86±0.05; GSAM 97.11±0.12; FSAM 97.08±0.24 | **97.13±0.06** |
| CIFAR-100, ResNet-50 | SAM 83.09±0.37; FSAM 83.23±0.46 | **83.64±0.17** |
| ImageNet-1K, ResNet-50 | SAM 77.81±0.04; ASAM 77.63±0.18 | **78.01±0.07** |
| BOOLQ test | Adam 75.18; SAM 76.02; FSAM 78.41 | **78.64** |
| COCO AP | SGD 22.23; SAM 27.53 | **27.65** |
| LVIS AP | SGD 10.37; SAM 11.74 | **11.76** |

These results support the paper’s claim that EISAM consistently outperforms SGD, Adam, and SAM in test accuracy and training efficiency across various architectures. At the same time, the reported segmentation metric shows that task-specific interpretation remains necessary: the table lists EISAM as having a U-Net score of 1.2735, whereas FSAM is 1.2973, and the paper separately notes best validation at 89.11%. This suggests that aggregate conclusions should be read in the context of the metric being emphasized for a given task.

## 5. Hyperparameter behavior and implementation guidance

The paper includes explicit hyperparameter ranges for different regimes. For CIFAR, the training schedule is 200 epochs with cosine decay, learning rate in $\{0.01,0.05,0.1\}$, weight decay in $\{5\mathrm{e}{-4},1\mathrm{e}{-3},2\mathrm{e}{-3}\}$, $\rho\in\{0.05,0.1,0.2\}$, and $s\in\{1\mathrm{e}{-3},5\mathrm{e}{-3}\}$. For ImageNet, the schedule is 90 epochs with $\mathrm{lr}=0.1\to 0$, weight decay $=1\mathrm{e}{-4}$, $\rho\in\{0.01,0.05,0.1\}$, and $s=1\mathrm{e}{-3}$. For BOOLQ, the configuration is 30 epochs with warmup+cosine, learning rate in $\{1\mathrm{e}{-4},5\mathrm{e}{-4},1\mathrm{e}{-3}\}$, weight decay in $\{1\mathrm{e}{-5},5\mathrm{e}{-5},5\mathrm{e}{-4}\}$, $\rho\in\{1\mathrm{e}{-3}\ldots 1\mathrm{e}{-1}\}$, and $s\in\{5\mathrm{e}{-5}\ldots 5\mathrm{e}{-4}\}$ [2607.06151].

The implementation guidance gives a broader tuning range: perturbation radius $\rho$ from 0.001 to 0.2 with default 0.05, and prediction step size $s$ from $1\mathrm{e}{-5}$ to $1\mathrm{e}{-2}$ with default $1\mathrm{e}{-3}$. Smaller $s$ is described as yielding more accurate geometry probing, while larger $s$ can accelerate convergence but risks overshoot. Base optimizer hyperparameters such as $\eta$, weight decay, and batch size are to be tuned as usual.

Sensitivity studies are central to the empirical characterization. Sweeping $\rho\in[0.001,0.25]$ on CIFAR-100 ResNet-50, the paper reports that training and test loss remain low and stable for $\rho\in[0.005,0.1]$, but degrade for $\rho>0.2$. Sweeping $s\in[0.001,0.02]$, the loss curves are described as almost overlapping, indicating robustness to $s$. A joint contour over $(\rho,s)$ shows that EISAM’s loss surface is much flatter along $\rho$ than SAM, and when $s=0$ the method reduces to SAM and recovers its high $\rho$-sensitivity. This is one of the clearest experimental differentiators claimed for the method.

The implementation procedure is also spelled out: store $g=\nabla F(w)$, perform $w\leftarrow w-s\cdot g$, add $\epsilon=\rho\cdot g/\|g\|$, run a forward/backward pass to obtain $g_{sam}=\nabla F(w+\epsilon)$, and then call `base_opt.step(w,gₛₐₘ)`. The paper further recommends mixed-precision or gradient checkpointing for large models and notes that cosine scheduling of $s$ in parallel with learning-rate decay often improves stability.

## 6. Computational profile, scope, and interpretation

EISAM is not presented as a zero-cost modification. Each iteration requires two gradient evaluations—prediction and correction—the same as SAM, and the approximation $\nabla F(y_t)\approx\nabla F(w_t)$ is used to avoid a third. In the LVIS detection benchmark, the reported per-batch time is 0.2673 s for SGD, 0.5519 s for SAM, and 0.5615 s for EISAM; total time is 4515 s, 9286 s, and 9449 s, respectively; peak memory is 18040 MB, 18702 MB, and 18540 MB; and FLOPs per batch for all SAM variants are 17.8656 T. The paper therefore characterizes the additional overhead relative to SAM as marginal rather than negligible [2607.06151].

Several interpretive points follow directly from the evidence provided. First, EISAM should not be conflated with a fundamentally different regularization target; its novelty lies in the extragradient-inspired lookahead inserted into a SAM-like procedure. Second, the strongest theoretical guarantees are established under strong convexity, smoothness, Lipschitz continuity, and Hessian Lipschitzness, while the nonconvex result is an extension under $L$-smoothness and bounded gradient variance. Third, the practical claim of robustness is specifically tied to reduced sensitivity to $\rho$, not to the elimination of tuning altogether, since the guidance still specifies default ranges and warns that larger $s$ can overshoot.

A common misconception would be to read the method as uniformly dominant in every reported number. The experimental record is broader than that: EISAM is reported to outperform the listed baselines on many headline metrics, especially test accuracy and AP, but the segmentation summary includes both the U-Net score values and a separate best validation figure, which requires careful interpretation. Another misconception would be to assume that the lookahead step merely reparameterizes SAM. The paper’s theory and sensitivity plots are presented precisely to argue otherwise: the dependence on $(1-s\mu)$ in the bounds and the flatter joint $(\rho,s)$ contour are the formal and empirical signatures of the added prediction step.

Taken together, the method is best understood as a sharpness-aware optimizer that preserves the operational simplicity of SAM while using a lightweight extragradient-inspired probe of the local loss geometry. In the paper’s own framing, this combination advances both theory and practice by tightening the generalization bound, reducing curvature at the attained minima, and simplifying tuning across diverse settings.

Source: https://www.emergentmind.com/topics/extragradient-inspired-sharpness-aware-minimization-eisam