Extragradient-Inspired Sharpness-Aware Minimization

• The paper introduces EISAM, a method that extends sharpness-aware minimization by adding an extragradient prediction step to improve generalization and tuning robustness.
• The algorithm performs a prediction, perturbation, and correction step, which together reduce sensitivity to the perturbation radius and bias optimization towards flatter minima.
• Empirical results and theoretical analysis demonstrate that EISAM tightens generalization bounds and outperforms baseline optimizers across diverse deep learning tasks.

Searching arXiv for the target paper and closely related SAM literature to ground the article in current preprints. Using the arXiv search tool for "(Fu et al., 7 Jul 2026) Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning" and related "Sharpness-Aware Minimization". Extragradient-Inspired Sharpness-Aware Minimization (EISAM) is an optimization method for deep learning that extends Sharpness-Aware Minimization (SAM) with an extragradient-inspired two-step update designed to improve generalization, reduce sensitivity to the perturbation radius, and bias training toward flatter minima. It is introduced in "Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning" (Fu et al., 7 Jul 2026). The method combines a prediction step that probes the local loss geometry with a sharpness-aware correction step implemented through a base optimizer such as SGD or Adam. The paper positions EISAM as a modification of SAM rather than a replacement for standard first-order training pipelines, and supports the proposal with theoretical bounds, benchmark experiments, and tuning guidance.

1. Optimization objective and conceptual motivation

EISAM is defined against the same sharpness-aware objective that motivates SAM. Given a training set $S$ and empirical risk

$F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$

SAM seeks solutions of

$$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$

with the inner maximization approximated by

$$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$

Within this formulation, the perturbation radius $\rho$ controls how aggressively the optimizer penalizes local sharpness.

EISAM modifies this scheme by inserting a prediction step before the sharpness-aware correction. The stated purpose of this extra lookahead is to “look ahead” in the negative-gradient direction, investigate the geometry of the loss landscape, reduce sensitivity to $\rho$, and bias optimization toward flatter minima. The abstract further states that this yields better generalization performance than SAM and simplifies tuning across diverse settings (Fu et al., 7 Jul 2026).

This construction places EISAM at the intersection of two established ideas: sharpness-aware training and extragradient-style lookahead. In the paper’s framing, the lookahead is lightweight rather than a separate higher-order procedure. A plausible implication is that EISAM should be understood less as a new objective than as a new update rule for approximately solving a sharpness-aware objective under improved hyperparameter robustness.

2. Algorithmic structure

At iteration $t$, with parameters $w_t\in\mathbb{R}^d$, minibatch $B_t$, prediction step size $s>0$, learning rate $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$0, and perturbation radius $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$1, EISAM performs three operations.

First, it computes a prediction step:

$F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$2

Second, it forms a sharpness perturbation at the predicted point:

$F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$3

which is often approximated for efficiency by using $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$4. The perturbed point is then $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$5.

Third, it applies a correction step with the base optimizer:

$F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$6

The paper states that the final parameter update is carried out by a base optimizer such as SGD or Adam.

The provided pseudocode makes the approximation explicit. One first stores $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$7, computes $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$8, sets $F_S(w)=\frac1n\sum_{i=1}^n f(w,z_i),$9, evaluates the sharpness-aware gradient $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$0, and finally updates with opt.update(w_t, g_t^{sam}, \eta). This sequence clarifies that EISAM retains the operational pattern of SAM while shifting the perturbation and correction to a predicted point.

A central limiting case is also stated explicitly: when $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$1, EISAM reduces to SAM, recovering SAM’s high $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$2-sensitivity. This is important both conceptually and diagnostically, because it isolates the extragradient-inspired prediction step as the distinctive mechanism.

3. Theoretical analysis

The theoretical development is given under standard convex-optimization assumptions: strong convexity, smoothness, Lipschitz continuity, and Hessian Lipschitzness. Specifically, $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$3 is assumed to be $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$4-strongly convex in $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$5, $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$6 is $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$7-Lipschitz, $$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$8, and

$$\min_{w}\;\max_{\|\epsilon\|\le\rho}\;F_S(w+\epsilon),$$9

The paper states that these assumptions guarantee bounded curvature and control higher-order remainders (Fu et al., 7 Jul 2026).

For uniform stability, the analysis considers two runs on datasets $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$0 and $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$1 differing in one sample, with trajectories $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$2 and $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$3 and gap $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$4. Lemmas 4.1–4.2 establish contraction for identical minibatches and a small drift for differing ones, leading to Theorem 4.1:

$$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$5

Compared to vanilla SAM, whose leading factor is stated to be $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$6 in place of $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$7, EISAM tightens the leading term by $$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$8.

For optimization error, Lemma 4.3 gives one-step progress under strong convexity:

$$\epsilon^*\approx\rho\,\nabla F_S(w)/\|\nabla F_S(w)\|.$$9

Theorem 4.2 then yields

$\rho$0

Combining the generalization and optimization terms, Theorem 4.3 bounds the expected excess risk:

$\rho$1

The paper interprets this as showing that a moderate $\rho$2 improves generalization with only a mild effect on the optimization rate, while $\rho$3 appears only in higher-order terms. This is the formal basis for the claim that EISAM has reduced sensitivity to the perturbation radius.

A nonconvex extension is also given. Under $\rho$4-smoothness and bounded gradient variance $\rho$5, Theorem 4.4 states for the averaged iterate $\rho$6 that

$\rho$7

This extends the argument beyond the strongly convex setting, but the theoretical guarantees remain conditional on the assumptions stated in the paper.

4. Empirical performance across tasks

The experimental program spans image classification, vision transformers, natural language processing, object detection, and segmentation. The reported baselines are SGD, Adam, SAM, ASAM, GSAM, FSAM, and EISAM. The setup includes CIFAR-10/100, ImageNet-1K, ViT with patch-sizes 8/16, BOOLQ with T5-small, COCO-2017 and LVIS v1.0 with Faster R-CNN w/ ResNet-50, and ISIC2018 with U-Net (Fu et al., 7 Jul 2026).

The CIFAR experiments report mean test accuracy over 3 seeds. On CIFAR-10, EISAM attains 96.84±0.09 on ResNet-18, 97.13±0.06 on ResNet-50, and 97.63±0.12 on WideRes-28-10, exceeding the corresponding reported SGD, SAM, ASAM, GSAM, and FSAM values. On CIFAR-100, EISAM reports 82.22±0.14 on ResNet-18 and 83.64±0.17 on ResNet-50, again above the listed baselines.

On ImageNet-1K, the reported top-1 accuracies are 70.89±0.15 for ResNet-18 and 78.01±0.07 for ResNet-50, which are higher than the corresponding figures given for SGD, SAM, ASAM, GSAM, and FSAM. For ViT-S-8 with CutMix on CIFAR-10, EISAM reaches 86.42±0.05, compared with SAM at 86.26±0.19 and Adam at 86.30±0.25.

The paper also reports results beyond image classification. On BOOLQ, EISAM gives validation/test accuracy 77.19/78.64, compared with Adam at 76.17/75.18, SAM at 77.14/76.02, and FSAM at 76.19/78.41. For object detection, the COCO AP progression is reported as SGD 22.23, SAM 27.53, and EISAM 27.65; on LVIS AP it is SGD 10.37, SAM 11.74, and EISAM 11.76. For segmentation on ISIC2018, the U-Net score is reported as Adam 1.2698, FSAM 1.2973, and EISAM 1.2735, with best val 89.11%.

Setting	Baseline values reported	EISAM
CIFAR-10, ResNet-50	SAM 96.86±0.05; GSAM 97.11±0.12; FSAM 97.08±0.24	97.13±0.06
CIFAR-100, ResNet-50	SAM 83.09±0.37; FSAM 83.23±0.46	83.64±0.17
ImageNet-1K, ResNet-50	SAM 77.81±0.04; ASAM 77.63±0.18	78.01±0.07
BOOLQ test	Adam 75.18; SAM 76.02; FSAM 78.41	78.64
COCO AP	SGD 22.23; SAM 27.53	27.65
LVIS AP	SGD 10.37; SAM 11.74	11.76

These results support the paper’s claim that EISAM consistently outperforms SGD, Adam, and SAM in test accuracy and training efficiency across various architectures. At the same time, the reported segmentation metric shows that task-specific interpretation remains necessary: the table lists EISAM as having a U-Net score of 1.2735, whereas FSAM is 1.2973, and the paper separately notes best validation at 89.11%. This suggests that aggregate conclusions should be read in the context of the metric being emphasized for a given task.

5. Hyperparameter behavior and implementation guidance

The paper includes explicit hyperparameter ranges for different regimes. For CIFAR, the training schedule is 200 epochs with cosine decay, learning rate in $\rho$8, weight decay in $\rho$9, $\rho$0, and $\rho$1. For ImageNet, the schedule is 90 epochs with $\rho$2, weight decay $\rho$3, $\rho$4, and $\rho$5. For BOOLQ, the configuration is 30 epochs with warmup+cosine, learning rate in $\rho$6, weight decay in $\rho$7, $\rho$8, and $\rho$9 (Fu et al., 7 Jul 2026).

The implementation guidance gives a broader tuning range: perturbation radius $t$0 from 0.001 to 0.2 with default 0.05, and prediction step size $t$1 from $t$2 to $t$3 with default $t$4. Smaller $t$5 is described as yielding more accurate geometry probing, while larger $t$6 can accelerate convergence but risks overshoot. Base optimizer hyperparameters such as $t$7, weight decay, and batch size are to be tuned as usual.

Sensitivity studies are central to the empirical characterization. Sweeping $t$8 on CIFAR-100 ResNet-50, the paper reports that training and test loss remain low and stable for $t$9, but degrade for $w_t\in\mathbb{R}^d$0. Sweeping $w_t\in\mathbb{R}^d$1, the loss curves are described as almost overlapping, indicating robustness to $w_t\in\mathbb{R}^d$2. A joint contour over $w_t\in\mathbb{R}^d$3 shows that EISAM’s loss surface is much flatter along $w_t\in\mathbb{R}^d$4 than SAM, and when $w_t\in\mathbb{R}^d$5 the method reduces to SAM and recovers its high $w_t\in\mathbb{R}^d$6-sensitivity. This is one of the clearest experimental differentiators claimed for the method.

The implementation procedure is also spelled out: store $w_t\in\mathbb{R}^d$7, perform $w_t\in\mathbb{R}^d$8, add $w_t\in\mathbb{R}^d$9, run a forward/backward pass to obtain $B_t$0, and then call base_opt.step(w,gₛₐₘ). The paper further recommends mixed-precision or gradient checkpointing for large models and notes that cosine scheduling of $B_t$1 in parallel with learning-rate decay often improves stability.

6. Computational profile, scope, and interpretation

EISAM is not presented as a zero-cost modification. Each iteration requires two gradient evaluations—prediction and correction—the same as SAM, and the approximation $B_t$2 is used to avoid a third. In the LVIS detection benchmark, the reported per-batch time is 0.2673 s for SGD, 0.5519 s for SAM, and 0.5615 s for EISAM; total time is 4515 s, 9286 s, and 9449 s, respectively; peak memory is 18040 MB, 18702 MB, and 18540 MB; and FLOPs per batch for all SAM variants are 17.8656 T. The paper therefore characterizes the additional overhead relative to SAM as marginal rather than negligible (Fu et al., 7 Jul 2026).

Several interpretive points follow directly from the evidence provided. First, EISAM should not be conflated with a fundamentally different regularization target; its novelty lies in the extragradient-inspired lookahead inserted into a SAM-like procedure. Second, the strongest theoretical guarantees are established under strong convexity, smoothness, Lipschitz continuity, and Hessian Lipschitzness, while the nonconvex result is an extension under $B_t$3-smoothness and bounded gradient variance. Third, the practical claim of robustness is specifically tied to reduced sensitivity to $B_t$4, not to the elimination of tuning altogether, since the guidance still specifies default ranges and warns that larger $B_t$5 can overshoot.

A common misconception would be to read the method as uniformly dominant in every reported number. The experimental record is broader than that: EISAM is reported to outperform the listed baselines on many headline metrics, especially test accuracy and AP, but the segmentation summary includes both the U-Net score values and a separate best validation figure, which requires careful interpretation. Another misconception would be to assume that the lookahead step merely reparameterizes SAM. The paper’s theory and sensitivity plots are presented precisely to argue otherwise: the dependence on $B_t$6 in the bounds and the flatter joint $B_t$7 contour are the formal and empirical signatures of the added prediction step.

Taken together, the method is best understood as a sharpness-aware optimizer that preserves the operational simplicity of SAM while using a lightweight extragradient-inspired probe of the local loss geometry. In the paper’s own framing, this combination advances both theory and practice by tightening the generalization bound, reducing curvature at the attained minima, and simplifying tuning across diverse settings.