---
title: 'Reweighted SAM: Delta-SAM Optimization'
url: https://www.emergentmind.com/topics/reweighted-sam
type: topic
---

# Reweighted SAM: Delta-SAM Optimization

Searching arXiv for the specified paper and closely related SAM literature to ground the article.
Reweighted SAM denotes a family of sharpness-aware training procedures in which a single shared adversarial weight perturbation is constructed for a mini-batch by reweighting per-instance gradients, so that the shared perturbation better approximates the stronger per-instance adversaries without incurring their prohibitive cost. In the formulation introduced as delta-SAM, or “Sharpness-Aware Minimization with Dynamic Reweighting,” the perturbation direction is obtained from a dynamically reweighted batch gradient whose weights are theoretically motivated by a second-order analysis of per-instance adversarial risk [2112.08772].

## 1. Position within sharpness-aware optimization

Sharpness-Aware Minimization (SAM) adversarially perturbs model weights and minimizes the loss at the worst-case point in a small $L_2$ neighborhood. For a batch of size $N$ with per-instance loss $l_i$, the per-batch SAM objective is
\[
\min_{\bm{w}} \max_{\bm{\epsilon}: \left\lVert\bm{\epsilon}\right\rVert_2\le \rho}\frac{1}{N}\sum_{i=1}^N l_i(\bm{w}+\bm{\epsilon}).
\]
In practice, the inner maximizer is approximated by a single first-order step using the batch-average loss $l(\bm{w}) = \frac{1}{N}\sum_{i=1}^N l_i(\bm{w})$, yielding
\[
\bm{\epsilon}^*\approx \arg\!\max_{\left\lVert\bm{\epsilon}\right\rVert_2\le \rho} \, l(\bm{w}) + \bm{\epsilon}^\intercal \nabla l(\bm{w})
= \rho \, \frac{\nabla l(\bm{w})}{\left\lVert\nabla l(\bm{w})\right\rVert_2}.
\]
The outer minimization then updates $\bm{w}$ to reduce the perturbed empirical risk [2112.08772].

Within this framework, standard per-batch SAM aggregates gradients over the mini-batch by averaging,
\[
\nabla l(\bm{w})=\frac{1}{N}\sum_{i=1}^N \nabla l_i(\bm{w}),
\]
and one SAM step roughly doubles compute, requiring two forward and two backward passes per batch. Reweighted SAM retains the shared-perturbation structure of SAM but replaces the uniform averaging implicit in the batch gradient with instance-dependent weights. This suggests a reinterpretation of SAM in which the central design question is not only how large the perturbation radius $\rho$ should be, but also how the shared perturbation direction should reflect heterogeneity across instances in the same mini-batch.

## 2. Per-batch and per-instance adversaries

The motivation for reweighting arises from the gap between a single perturbation shared by all examples in a batch and the stronger objective in which each example receives its own adversarial perturbation. The per-instance objective is
\[
\min_{\bm{w}} \frac{1}{N}\sum_{i=1}^N \max_{\bm{\epsilon}_i: \left\lVert\bm{\epsilon}_i\right\rVert_2\le \rho} \, l_i(\bm{w}+\bm{\epsilon}_i).
\]
In first-order practice, one would take
\[
\bm{\epsilon}_i \approx \rho\,\frac{\nabla l_i(\bm{w})}{\lVert\nabla l_i(\bm{w})\rVert_2}
\]
for each $i$ [2112.08772].

The associated sharpness increases are defined as
\[
\mathcal{R}_\text{batch}=\max_{\left\lVert\bm{\epsilon}\right\rVert_2\le \rho}\frac{1}{N}\sum_{i=1}^N \bigl(l_i(\bm{w} + \bm{\epsilon}) - l_i(\bm{w})\bigr),\quad
\mathcal{R}_\text{inst}=\frac{1}{N}\sum_{i=1}^N \max_{\left\lVert\bm{\epsilon}_i\right\rVert_2\le \rho} \bigl(l_i(\bm{w} + \bm{\epsilon}_i) - l_i(\bm{w})\bigr).
\]
Since the inner maximizers are independent in $\mathcal{R}_\text{inst}$, $\mathcal{R}_\text{inst} \ge \mathcal{R}_\text{batch}$. The reweighted construction is therefore motivated by the observation that sharing one perturbation across the batch yields a weaker adversary than summing per-instance worst cases [2112.08772].

The difficulty is computational. Per-instance perturbations are computationally expensive because the inner problem must be solved separately for each instance, requiring $O(N)$ backward passes and effectively distinct model copies, approximately $2N$ forward and $2N$ backward passes per batch. The data further reports that this is often unaffordable in practice. Reweighted SAM is designed precisely for this efficiency bottleneck: it attempts to retain a single batch-level perturbation while moving that perturbation toward the behavior of the per-instance adversary.

## 3. Theoretical derivation of dynamic reweighting

The delta-SAM derivation uses a second-order expansion
\[
l_i(\bm{w}+\bm{\epsilon})=l_i(\bm{w}) + \nabla l_i(\bm{w})^\intercal \bm{\epsilon} + \tfrac{1}{2} \bm{\epsilon}^\intercal \bm{H}_i(\bm{w}) \bm{\epsilon},
\]
together with the structured Hessian assumption
\[
\bm{H}_i(\bm{w}) = a_i \nabla l_i(\bm{w}) \nabla l_i(\bm{w})^\intercal,\quad a_i>0.
\]
Under the single-step inner maximization, the SAM-style perturbations are
\[
\bm{\epsilon} = \rho \, \frac{\nabla l(\bm{w})}{\left\lVert \nabla l(\bm{w})\right\rVert_2},\quad
\bm{\epsilon}_i = \rho \, \frac{\nabla l_i(\bm{w})}{\left\lVert \nabla l_i(\bm{w})\right\rVert_2}.
\]

Under these assumptions, the gradient of the per-instance adversarial risk, for the outer minimization and without differentiating through $\bm{\epsilon}_i$, is
\[
\frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst}
= \frac{1}{N}\sum_{i=1}^N \bigl(\nabla l_i(\bm{w} + \bm{\epsilon}_i) - \nabla l_i(\bm{w})\bigr)
= \frac{1}{N}\sum_{i=1}^N \bm{H}_i (\bm{w}) \bm{\epsilon}_i
= \frac{1}{N}\sum_{i=1}^N \rho a_i \left\lVert \nabla l_i(\bm{w})\right\rVert_2 \, \nabla l_i(\bm{w}).
\]
This expression identifies the target direction that a shared perturbation should approximate if it is to mimic the gradient induced by per-instance adversarial training [2112.08772].

For a reweighted shared perturbation $\bm{\epsilon}'$, let
\[
\mathcal{R}:=\frac{1}{N}\sum_{i} \bigl(l_i(\bm{w} + \bm{\epsilon}') - l_i(\bm{w})\bigr),
\quad
\frac{\partial}{\partial \bm{w} \mathcal{R}
= \left ( \frac{1}{N}\sum_{i=1}^N \bm{H}_i(\bm{w}) \right ) \bm{\epsilon}'.
\]
An ideal choice is
\[
\bm{\epsilon}'=\rho \cdot \left ( \frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst} \right ) \Big / \left\lVert \frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst}\right\rVert_2 ,
\]
which guarantees
\[
\left ( \frac{\partial}{\partial \bm{w}\mathcal{R} \right )^\intercal \frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst}
\propto \left ( \frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst} \right )^\intercal \!\left(\frac{1}{N}\sum_{i=1}^N \bm{H}_i(\bm{w})\right) \!\left ( \frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst} \right ) > 0,
\]
because $\frac{1}{N}\sum_i \bm{H}_i$ is positive definite. Since $\frac{\partial}{\partial \bm{w}\mathcal{R}_\text{inst}$ is not directly available, the paper motivates approximating it by a reweighted batch gradient proportional to $\sum_i a_i \lVert \nabla l_i\rVert \nabla l_i$ [2112.08772].

## 4. Reweighted perturbation and the delta-SAM algorithm

The dynamic weights are defined as
\[
g_i:=a_i \lVert \nabla l_i(\bm{w})\rVert_2,
\]
and the reweighted batch loss is
\[
l_\text{reweighted} = \frac{1}{N}\sum_{i=1}^N g_i \, l_i(\bm{w}),
\]
with $g_i$ treated as constants for the batch. Then
\[
\frac{\partial}{\partial \bm{w} l_\text{reweighted} \propto \frac{\partial}{\partial \bm{w} \mathcal{R}_\text{inst},
\]
which leads to the normalized reweighted batch gradient
\[
\nabla l_\mathcal{B}(\bm{w}) = \nabla \Big(\sum_{i=1}^N g_i l_i(\bm{w})\Big) = \sum_{i=1}^N g_i \, \nabla l_i(\bm{w}),
\]
and the perturbation
\[
\bm{\epsilon}^* = \rho \, \frac{\nabla l_\mathcal{B}(\bm{w})}{\left\lVert \nabla l_\mathcal{B}(\bm{w}) \right\rVert_2}.
\]
Writing $\mathbf{g}_i:=\nabla_{\bm{w}} l_i(\bm{w})$ and $\alpha_i:=g_i$, this becomes
\[
\bm{\epsilon}_{\text{reweighted}} = \rho \, \frac{\sum_{i=1}^B \alpha_i \, \mathbf{g}_i}{\big\lVert\sum_{i=1}^B \alpha_i \, \mathbf{g}_i\big\rVert_2}.
\]
This is the core “Reweighted SAM” perturbation in delta-SAM [2112.08772].

The algorithmic step for a mini-batch $\mathcal{B}=\{(\bm{x}_j,\bm{y}_j)\}_{j=1}^N$ proceeds as follows. First, one shared $\bm{r}\sim\mathcal{N}(0,\sigma \bm{I})$ is sampled and, in practice, rescaled to $\lVert \bm{r}\rVert_2=\rho$. In no_grad mode, the method computes per-instance losses $l_i(\bm{w})$, $l_i(\bm{w}+\bm{r})$, and $l_i(\bm{w}-\bm{r})$, and then computes dynamic weights $g_i$. Second, it forms $l_\text{reweighted}=\frac{1}{N}\sum_i g_i l_i(\bm{w})$ and backpropagates to obtain $\nabla l_\mathcal{B}(\bm{w})=\sum_i g_i \nabla l_i(\bm{w})$. Third, it computes the adversarial perturbation $\bm{\epsilon}^*=\rho\,\frac{\nabla l_\mathcal{B}(\bm{w})}{\lVert\nabla l_\mathcal{B}(\bm{w})\rVert_2}$. Fourth, with perturbed weights $\bm{w}+\bm{\epsilon}^*$, it computes the standard unweighted empirical risk $\frac{1}{N}\sum_i l_i(\bm{w}+\bm{\epsilon}^*)$, backpropagates, and updates $\bm{w}$ using the optimizer, such as SGD or Adam [2112.08772].

The implementation details are integral to the method’s intended behavior. The three forward-only passes used to estimate $g_i$ are performed in no_grad mode; a single shared random perturbation is used per batch to keep overhead small; the perturbation vector is normalized; and the training update is performed on the unweighted loss to avoid biasing training targets. This suggests that the reweighting is intended only to shape the adversarial direction, not to redefine the empirical objective.

## 5. Estimation of dynamic weights

The practical obstacle is efficient estimation of
\[
g_i=a_i \lVert \nabla l_i\rVert_2.
\]
Delta-SAM proposes a simple, unbiased estimator based on one shared random Gaussian weight perturbation $\bm{r}\sim\mathcal{N}(0,\sigma \bm{I})$ per batch and three forward passes, without backpropagation. Under the same Hessian assumption, the paper states
\[
E\!\left[\bigl(l_i(\bm{w} + \bm{r}) - l_i(\bm{w} - \bm{r})\bigr)^2\right]
= \sigma^2 \left\lVert \nabla l_i(\bm{w})\right\rVert_2^2,
\]
and
\[
E\!\left[l_i(\bm{w} + \bm{r}) + l_i(\bm{w} - \bm{r}) - 2l_i(\bm{w})\right]
= a_i\sigma^2 \left\lVert \nabla l_i(\bm{w})\right\rVert_2^2.
\]
Using a single shared $\bm{r}$ per batch, the dynamic weight is defined as
\[
g_i=\frac{\left|l_i(\bm{w} + \bm{r}) + l_i(\bm{w} - \bm{r}) - 2 l_i(\bm{w})\right|}{\max\!\left(\left|l_i(\bm{w} + \bm{r}) - l_i(\bm{w} - \bm{r})\right|, \eta\right)},
\]
where $\eta$ avoids division by zero [2112.08772].

The numerator is described as proxying curvature along $\bm{r}$ via the Hessian, while the denominator proxies gradient magnitude along $\bm{r}$, so that the ratio tracks $a_i \lVert\nabla l_i\rVert$. This is the mechanism by which delta-SAM obtains per-instance weights without computing per-instance Hessians or per-instance adversarial perturbations. A plausible implication is that the estimator’s quality depends not only on the validity of the structured Hessian assumption but also on the variance introduced by using only one random perturbation sample per batch.

The practical settings reported for this estimator include $\sigma=1$ with $\bm{r}$ rescaled to $\lVert\bm{r}\rVert_2=\rho$, and denominator floors $\eta$ searched in $\{1\text{e}{-4}, 2\text{e}{-4}, 5\text{e}{-4}, 1\text{e}{-3}\}$, with the paper often using $1\text{e}{-4}$. Perturbation radius $\rho$ is searched in $\{0.01, 0.02, 0.05\}$ on GLUE; for STS, $\rho \in \{0.01, 0.02, 0.05, 0.1\}$; and summarization uses $\rho=0.01$ [2112.08772].

## 6. Empirical behavior, computational profile, and limitations

The reported empirical results cover natural language understanding, self-supervised semantic textual similarity, and summarization. On GLUE development sets, BERTbase with SAM attains an average of 83.9, while delta-SAM attains 84.7; task-wise examples include MRPC $88.2 \rightarrow 89.5$, RTE $69.3 \rightarrow 72.2$, CoLA $63.1 \rightarrow 63.8$, and SST-2 $93.0 \rightarrow 93.7$. For RoBERTa_large, SAM attains 89.6 and delta-SAM 90.1, with reported gains including MRPC $91.4 \rightarrow 92.2$, CoLA $69.2 \rightarrow 71.1$, and RTE $88.5 \rightarrow 89.2$ [2112.08772].

In self-supervised STS, measured by Spearman’s $\rho$, Mirror-BERTbase reports baseline 74.85, +SAM 75.50, +delta-SAM 75.72, +SAM w/ random noise 76.44, and +delta-SAM w/ random noise 76.71. Mirror-RoBERTabase reports baseline 74.98, +SAM 75.18, +delta-SAM 75.27, +SAM w/ random noise 75.90, and +delta-SAM w/ random noise 76.31. On CNN/DailyMail summarization with BART-large, the baseline reports RG-1 44.16, RG-2 21.28, RG-L 40.90, while +delta-SAM reports RG-1 44.70, RG-2 21.54, RG-L 41.81, which the data identifies as the best RG-1 and RG-L among compared methods [2112.08772].

The paper also reports adversarial risk analysis using the per-instance metric $\mathcal{L}_\text{adv}$ with $\rho=0.05$. On MRPC and RTE, delta-SAM achieves lower $\mathcal{L}_\text{adv}$ than SAM, with examples MRPC $0.62 \rightarrow 0.59$ and RTE $0.78 \rightarrow 0.75$, together with higher accuracy, approaching per-instance perturbation performance. The reported ablations state that reweighting matters, that smaller datasets show larger gains, and that moderate values of $\rho$ matching baseline SAM often work best, whereas very large $\rho$ can destabilize training. The paper further notes that using random input noise in self-supervised STS improves results, and that delta-SAM with random noise outperforms baselines that also use random noise [2112.08772].

The computational profile is central to the method’s practical claim. Standard SAM uses two forward and two backward passes per batch, with time overhead approximately $2\times$ vanilla training. Per-instance SAM requires approximately $2N$ forward and $2N$ backward passes, plus memory for distinct perturbations; empirically it is reported as approximately $7\times$ delta-SAM for batch size 16. Delta-SAM uses the same number of backward passes as standard SAM, plus three forward-only passes per batch to estimate $g_i$, and empirically adds only about 18% extra computational cost over SAM on average. A concrete example is BERTbase on SST-2: 118 min for SAM versus 132 min for delta-SAM, approximately 12% slower [2112.08772].

The limitations stated in the source are correspondingly specific. Delta-SAM remains slower than vanilla training because of the extra forward passes, even though it is much cheaper than per-instance SAM. Its theoretical derivation depends on the second-order expansion and the Hessian approximation $\bm{H}_i=a_i \nabla l_i \nabla l_i^\intercal$; deviations from these assumptions may limit the accuracy of the reweighting. The source also identifies potential extensions, including better curvature estimation, multiple random perturbation samples to reduce estimator variance, integration with ASAM or GSAM or trust-region methods, and per-layer or blockwise reweighting on broader task families such as sequence tagging and question answering. This suggests that reweighted SAM is best understood not as a replacement for sharpness-aware optimization in general, but as a specific mechanism for approximating per-instance adversarial weight perturbations under a constrained computational budget [2112.08772].

Source: https://www.emergentmind.com/topics/reweighted-sam