VASSO: Variance Suppression in SAM

• Variance Suppression (VASSO) is an optimization method that stabilizes the inner linearization in sharpness-aware minimization using an exponential moving average of gradients.
• It reduces gradient variance to produce adversarial perturbations that are less ‘friendly’ and more representative of local sharpness, leading to improved generalization.
• VASSO offers a low-overhead upgrade to SAM, maintaining similar computational cost while effectively mitigating instability issues inherent in minibatch-based approximations.

Variance Suppression (VASSO) is a variance-suppression mechanism for sharpness-aware optimization that modifies the practical adversary construction used in Sharpness-Aware Minimization (SAM). In this line of work, the target is not the outer sharpness-aware objective itself, but the instability of the minibatch-based inner linearization that SAM uses to generate adversarial parameter perturbations. VASSO replaces the raw stochastic gradient in that inner step with an exponentially averaged direction, with the stated aim of producing adversaries that are more stable, less “friendly,” and more representative of local sharpness. The method appeared earlier under the spelling “VaSSO” and later under the spelling “VASSO” (Li et al., 2023, Li et al., 2 Sep 2025).

1. Nomenclature and scope

The method is presented as “VaSSO” in “Enhancing Sharpness-Aware Optimization Through Variance Suppression” (Li et al., 2023) and as “VASSO” in “VASSO: Variance Suppression for Sharpness-Aware Minimization” (Li et al., 2 Sep 2025). In both formulations, the underlying idea is the same: SAM is retained as the training paradigm, but the stochastic slope used to define the adversarial perturbation is stabilized by variance suppression. The method is therefore best understood as an optimization modification or wrapper for SAM-style methods, not as a separate regularizer added to the loss, and not as a new minimax objective in the strict sense (Li et al., 2 Sep 2025).

This scope matters because “variance suppression” here refers to the variance of the direction used in the inner linearized maximization. It does not refer to redefining sharpness-aware learning from scratch, and it is distinct from simply adding noise, changing the base optimizer, or introducing a separate explicit penalty. A plausible implication is that VASSO belongs more naturally to the family of practical adversary-generation schemes than to the family of objective-level reformulations of robust or sharpness-aware optimization (Li et al., 2 Sep 2025).

2. SAM, sharpness, and the “friendly adversary” problem

SAM is posed as the minimax problem

$$\min_{\mathbf{x}} \max_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}+\bm{\epsilon}),$$

where $\mathbf{x}$ denotes model parameters, $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$ is the expected training loss, and $\rho$ is a perturbation radius. In practice, the inner maximization is replaced by a first-order stochastic approximation: $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$ followed by the outer update

$$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$

The central criticism is that this adversary is computed from a single minibatch. It may therefore be adversarial for the batch $\mathcal B_t$ used to construct it, but “friendly” to other batches. Formally, for the construction batch one has approximately

$$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$

for small $\rho$, but for another batch $\mathcal B$ it can happen that

$\mathbf{x}$0

whenever

$\mathbf{x}$1

This is the paper’s “over-friendly adversary” or “friendly adversary” failure mode. It is also supported empirically: when the adversary is computed from a different random batch than the one used for the perturbed loss, SAM’s generalization gains nearly disappear (Li et al., 2 Sep 2025).

3. Algorithmic construction of VASSO

VASSO replaces the raw stochastic gradient used in the inner linearization with an exponential moving average: $\mathbf{x}$2 The adversarial perturbation is then defined by

$\mathbf{x}$3

while the outer update remains SAM-like: $\mathbf{x}$4

Operationally, each iteration samples a minibatch, computes $\mathbf{x}$5, updates $\mathbf{x}$6, forms $\mathbf{x}$7, computes the perturbed gradient, and updates parameters. This requires the same two gradient evaluations per iteration as vanilla SAM. The extra cost is one EMA vector update and storage of $\mathbf{x}$8. The memory overhead is described as effectively unchanged from SAM because one can store $\mathbf{x}$9 and its norm and reconstruct $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$0 when needed (Li et al., 2 Sep 2025).

The method formalizes adversary quality through

$$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$1

with anchor quantity $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$2. A stochastic linearization with slope $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$3 is defined to be $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$4-stable if

$$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$5

This definition makes VASSO’s target explicit: it seeks to stabilize the stochastic linearization used to approximate local sharpness (Li et al., 2 Sep 2025).

4. Theoretical properties

The analysis assumes that $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$6 is lower bounded, that the stochastic gradient is $$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$7-Lipschitz,

$$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$8

and that it is unbiased with bounded variance,

$$f(\mathbf{x})=\mathbb{E}_{\mathcal B}[f_{\mathcal B}(\mathbf{x})]$$9

Under these assumptions, generic SAM-type updates converge as long as $\rho$0, so VASSO inherits the same order of convergence as SAM. With $\rho$1, $\rho$2, and $\rho$3, the SAM convergence theorem gives

$\rho$4

and

$\rho$5

The distinctive result is the mean-squared-error control for the EMA direction: $\rho$6 By contrast, SAM only has

$\rho$7

Thus, for sufficiently large $\rho$8, VASSO shrinks the leading variance term from $\rho$9 to roughly $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$0. Using

$$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$1

the paper derives the stability comparison: SAM has $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$2-stability, whereas VASSO has

$$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$3

Asymptotically, VASSO is therefore about $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$4 times more stable than SAM. The tuning implication is explicit: $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$5 should be small enough to suppress variance, but not too small, because the residual term

$$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$6

is amplified when $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$7 is extremely small (Li et al., 2 Sep 2025).

The paper also gives an SNR-based intuition. Near convergence, when $$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$8 is small relative to gradient noise, the normalized SAM direction

$$\bm{\epsilon}_t \approx \argmax_{\|\bm{\epsilon}\|\le \rho} f(\mathbf{x}_t)+\langle \mathbf{g}_t(\mathbf{x}_t),\bm{\epsilon}\rangle = \rho \frac{\mathbf{g}_t(\mathbf{x}_t)}{\|\mathbf{g}_t(\mathbf{x}_t)\|},$$9

can point almost anywhere on the sphere. In training ResNet-18 on CIFAR-10, reported practical SNR values are in the range $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$0, under which SAM adversaries are nearly spread over the sphere. The EMA direction averages out noise and preserves persistent directional information, which the paper interprets as making perturbations more representative of “global” local sharpness (Li et al., 2 Sep 2025).

5. Empirical performance and efficient variants

The empirical evaluation covers image classification, domain generalization, label-noise robustness, and neural machine translation. In image classification, the paper evaluates VGG-11, ResNet-18, WRN-28-10, and PyramidNet-110 on CIFAR-10 and CIFAR-100, and ResNet-50 and ViT-S/32 on ImageNet, against SGD, SAM, ASAM, FisherSAM, and GSAM. On CIFAR, VASSO usually improves over SAM by around $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$1–$$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$2 accuracy points, with a smaller gain of about $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$3 on PyramidNet-110. On ImageNet, VASSO improves ResNet-50 over SAM from $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$4 to $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$5, and ViT-S/32 from $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$6 to $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$7. Combined with GSAM, denoted V+G, it reaches $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$8 on ResNet-50 and $$\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\,\mathbf{g}_t(\mathbf{x}_t+\bm{\epsilon}_t).$$9 on ViT-S/32 (Li et al., 2 Sep 2025).

In DomainBed domain generalization with a pretrained ResNet-50 and Adam base optimizer, VASSO attains the best average leave-one-out cross-validation accuracy among Adam, SAM, and GSAM: average $\mathcal B_t$0 versus $\mathcal B_t$1 for SAM and $\mathcal B_t$2 for GSAM. In neural machine translation on IWSLT-14 with a Transformer and AdamW, the best reported setting is $\mathcal B_t$3, giving BLEU $\mathcal B_t$4 versus $\mathcal B_t$5 for SAM and $\mathcal B_t$6 for AdamW. The label-noise results are substantially larger in magnitude. On CIFAR-10 with random label flips, the gap widens with noise level; at $\mathcal B_t$7 label noise, SAM gets $\mathcal B_t$8, while VASSO reaches $\mathcal B_t$9 with $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$0 and $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$1 with $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$2, an absolute gain of $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$3 points over SAM (Li et al., 2 Sep 2025).

The paper also reports flatness-related diagnostics. For ResNet-18 on CIFAR-10, the top Hessian eigenvalue $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$4 decreases from $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$5 for SGD to $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$6 for SAM and further to $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$7 for VASSO; the anisotropy proxy $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$8 decreases from $$f_{\mathcal B_t}(\mathbf{x}_t+\bm{\epsilon}_t)-f_{\mathcal B_t}(\mathbf{x}_t)\ge 0$$9 to $\rho$0 to $\rho$1. Loss-landscape visualizations are described as showing flatter valleys than SAM. This is presented as evidence that stabilizing the adversary strengthens the intended sharpness-aware bias rather than merely injecting another form of noise (Li et al., 2 Sep 2025).

An efficient variant, eVASSO, combines stabilized adversaries with an efficient SAM schedule. It introduces an i.i.d. Bernoulli variable $\rho$2 with

$\rho$3

At every iteration it computes $\rho$4 and updates $\rho$5, but only when $\rho$6 does it form the adversary and compute the second gradient at $\rho$7; otherwise it performs a plain SGD update. The expected number of gradient computations per iteration is $\rho$8, interpolating between SGD ($\rho$9) and VASSO ($\mathcal B$0). On CIFAR-10 with ResNet-18, eSAM at $\mathcal B$1 gets $\mathcal B$2, while eVASSO at $\mathcal B$3 gets $\mathcal B$4; at $\mathcal B$5, eSAM gets $\mathcal B$6, while eVASSO gets $\mathcal B$7. The paper’s headline tradeoff claim is that eVASSO with $\mathcal B$8 can surpass vanilla SAM and eSAM with $\mathcal B$9, implying about $\mathbf{x}$00 and $\mathbf{x}$01 savings in computational overhead relative to SAM and eSAM, respectively, for the same accuracy target (Li et al., 2 Sep 2025).

6. Interpretation, hyperparameters, and limitations

The main additional hyperparameter is $\mathbf{x}$02, the EMA coefficient. Reported tuning sets include $\mathbf{x}$03 and $\mathbf{x}$04. Smaller values generally improve stability more, especially in noisier settings such as label noise, but excessively small values may over-smooth or worsen finite-time behavior. In cleaner image-classification settings, values like $\mathbf{x}$05 or $\mathbf{x}$06 often work; under severe label noise, $\mathbf{x}$07 performs better. The perturbation radius $\mathbf{x}$08 is tuned similarly to SAM, and the paper generally uses the same $\mathbf{x}$09 for VASSO as for SAM (Li et al., 2 Sep 2025).

Several misconceptions are addressed implicitly by the formulation. VASSO does not change the outer SAM objective, does not replace the one-step closed-form perturbation by a multi-step adversary generator, and is not presented as a generic gradient-noise regularizer. It specifically modifies the slope in the inner stochastic linearization. The paper also argues that the observed behavior is distinct from the so-called $\mathbf{x}$10-sharpness phenomenon: larger variance in itself does not explain better generalization. A plausible implication is that VASSO’s contribution is best understood as improving the quality of the adversarial perturbation rather than simply adding another source of stochasticity (Li et al., 2 Sep 2025).

The limitations are also explicit. VASSO is not presented as universally dominant on every dataset and architecture; some gains are modest, and on certain held-out domains it is competitive rather than best. Full VASSO, like SAM, still needs two gradient computations per adversarial iteration. The theoretical and empirical arguments therefore support it as a low-friction upgrade to SAM, especially when minibatch noise is high, data are noisy, or distribution shift is important, rather than as a replacement for sharpness-aware optimization as a whole (Li et al., 2 Sep 2025).