---
title: SignSGD with Momentum
url: https://www.emergentmind.com/topics/signsgd-with-momentum
type: topic
---

# SignSGD with Momentum

SignSGD with Momentum is a sign-based stochastic optimization method that maintains an exponential moving average of stochastic gradients and updates parameters using only the coordinate-wise sign of that buffer. In the literature it is often called **Signum**. Relative to SGD with momentum, it replaces magnitude-sensitive updates by constant-magnitude coordinate steps, yielding 1-bit quantization of the update direction and therefore substantial memory and communication savings, especially in distributed settings. Its central trade-off is that the same quantization that enables compression also discards magnitude information, which can widen a generalization gap relative to well-tuned SGD or cause late-stage stagnation near minima [1802.04434] [2604.25550].

## 1. Algorithmic form and notation

In its canonical form, SignSGD updates by the sign of the stochastic gradient,
$$
x_{t+1} = x_t - \eta\, \operatorname{sign}(\tilde g_t),
$$
where the sign is applied coordinate-wise. SignSGD with Momentum replaces the raw stochastic gradient by a momentum buffer,
$$
m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad
x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),
$$
with $\beta \in (0,1)$ the momentum coefficient. In the terminology of the original signSGD paper, this momentum variant is **Signum** [1802.04434].

Subsequent analyses use slightly different but equivalent parameterizations. One line of work writes the buffer as an EMA,
$$
m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad
x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),
$$
while another uses a heavy-ball/Polyak-style recursion,
$$
v_t =
\begin{cases}
\nabla f(x_1;\xi_1), & t=1,\\[3pt]
(1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2,
\end{cases}
\qquad
x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).
$$
The EMA form differs slightly from the classical $m_t=\beta m_{t-1}+g_t$ notation, but one analysis notes that the difference is equivalent up to a rescaling of the step size and the effective momentum state [2507.12091] [2604.25550].

Algorithmically, the method has three defining properties. First, the sign operator is applied **after** momentum smoothing; it is not a moving average of signs. Second, the update is coordinate-wise and typically takes values in $\{-1,0,+1\}^d$ or $\{-1,+1\}^d$, depending on how zero coordinates are treated. Third, no additional per-coordinate normalization is present in the basic method beyond momentum smoothing and the sign operator [1802.04434] [2604.25550].

## 2. Foundational theory and the geometry of sign updates

The earliest non-convex analysis of signSGD and Signum is formulated under a lower-bounded objective, coordinate-wise smoothness, and unbiased stochastic gradients with coordinate-wise bounded variances. For signSGD, the original theory used a large-batch schedule, with step size $\delta_t = 1/\sqrt{LK}$ and batch size $n_t=K$, and established a non-convex stationarity guarantee in the $\ell_1$ norm of the gradient. For Signum, the same work gave an anytime analysis with
$$
\delta_t = \frac{\delta}{\sqrt{t+1}},\qquad n_t=t+1,
$$
together with a momentum warmup period $C(\beta)$; the paper reports that for $\beta=0.9$, $C\approx 54$ [1802.04434].

A distinctive feature of this theory is its emphasis on **relative $\ell_1/\ell_2$ geometry**. The paper defines the density of a vector $v\in \mathbb{R}^d$ by
$$
\phi(v) := \frac{\|v\|_1^2}{d\,\|v\|_2^2},
$$
with $\phi(\text{dense})\approx 1$ and $\phi(\text{sparse})\approx 1/d$. The resulting comparison between signSGD and SGD is not purely asymptotic; it depends on whether gradients are dense relative to curvature and stochastic noise. The original interpretation is that if gradients are at least as dense as curvature and stochasticity, signSGD can match or beat SGD while also providing compression, whereas if curvature and noise are much denser than gradients, SGD is preferable [1802.04434].

The same paper also established the distributed majority-vote formulation: each worker transmits the sign of its local stochastic gradient, the server aggregates coordinate-wise by majority vote,
$$
d_t = \operatorname{sign}\!\Big(\sum_{m=1}^M \operatorname{sign}(\tilde g_t^{(m)})\Big),
$$
and all workers update with $x_{t+1}=x_t-\eta d_t$. Under unimodal symmetric noise, majority vote was shown to improve the variance term by $\sqrt{M}$ and to compress worker-server communication to 1 bit per coordinate in both directions [1802.04434].

## 3. Momentum-based convergence under weaker assumptions

Later work substantially broadened the convergence theory for sign-based methods with momentum. Under standard $\ell_2$-smoothness and bounded variance, the “Sign-based Momentum Method” proved that SignSGD with momentum can achieve
$$
\mathbb{E}\big[\|\nabla f(x_\tau)\|_1\big] \le \mathcal{O}\!\big(d^{1/2}T^{-1/4}\big),
$$
with $\beta=\Theta(T^{-1/2})$ and $\eta=\Theta(d^{-1/2}T^{-3/4})$. Under $\ell_\infty$-smoothness and separable bounded noise, the same paper proved
$$
\mathbb{E}\big[\|\nabla f(x_\tau)\|_1\big] \le \mathcal{O}\!\big(T^{-1/4}\big).
$$
These results remove the need for large batch sizes and, in the $\ell_2$-smooth case, remove the need for unimodal symmetric stochastic noise. The paper states that this improves the prior momentum-based signSGD result by a factor of $\mathcal{O}(d^{1/2})$ [2507.12091].

The mechanism behind these bounds is an explicit control of the momentum error $\epsilon_t := v_t - \nabla f(x_t)$. A key descent inequality is
$$
f(x_{t+1}) \le f(x_t) - \eta\, \|\nabla f(x_t)\|_1 + 2\eta \sqrt{d}\,\|\nabla f(x_t)-v_t\| + \frac{\eta^2Ld}{2},
$$
paired with a recursion for $\mathbb{E}\|v_t-\nabla f(x_t)\|^2$ that makes the variance-reduction role of momentum quantitative [2507.12091].

A different line of analysis revisited SignSGD from a 1-bit quantization viewpoint and derived a **small-batch** rate under unimodal symmetric gradient noise. It introduced the signal-to-noise weighted stationarity measure
$$
\Phi_k := \sum_{i=1}^d |g_{k,i}|\,\min(1,S_{k,i}),
\qquad
S_{k,i} := \frac{|g_{k,i}|}{s_{k,i}},
$$
where $s_{k,i}=\sqrt{\operatorname{Var}(\tilde g_{k,i}\mid x_k)}$. With constant batch size $n$ and step size $\delta_k\equiv 1/\sqrt{\tilde L_1 K}$, the paper proved
$$
\mathbb{E}\Big[\frac{1}{K}\sum_{k=0}^{K-1}\Phi_k\Big]
\le
\frac{3\sqrt{\tilde L_1}}{\sqrt K}\Big(f_0-f^\star+\frac12\Big),
$$
and
$$
\mathbb{E}\Big[\frac{1}{K}\sum_{k=0}^{K-1}\|g_k\|_1\Big]
\le
\frac{3\sqrt{\tilde L_1}}{\sqrt K}\Big(f_0-f^\star+\frac12\Big) + \frac{\tilde \sigma_1}{\sqrt n}.
$$
The paper emphasizes that, unlike prior large-batch analyses, this rate holds for any fixed small batch $n$, and that the $\ell_1$-gradient bound contains a non-vanishing noise floor $\tilde \sigma_1/\sqrt n$ intrinsic to 1-bit quantization at low SNR [2604.25550].

A third theoretical direction concerns **unbounded smoothness**. The generalized SignSGD framework studies
$$
m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t,\qquad
v_t=\beta_2 v_{t-1}+(1-\beta_2)m_t^2,\qquad
x_{t+1}=x_t-\eta\,\frac{m_t}{\sqrt{v_t}},
$$
and observes that the endpoint $\beta_2=0$ recovers SignSGD with momentum because then $v_t=m_t^2$ and the update becomes $x_{t+1}=x_t-\eta\,\operatorname{sign}(m_t)$. Under coordinate-wise $(L_0,L_1)$-smoothness and bounded noise, the paper proves a $T^{-1/4}$-type stationarity bound and states that the $\beta_2=0$ endpoint matches clipped-SGD rates up to norm conventions, while removing detrimental dependence on the unbounded-smoothness term and large gradient norms in the final bound [2208.11195].

## 4. Distributed, communication-efficient, and robust variants

The original appeal of SignSGD with Momentum is inseparable from communication efficiency. In distributed majority-vote signSGD, each worker sends 1 bit per coordinate to the server and receives 1 bit per coordinate back. The 2018 analysis argued that this yields compressed gradients and an SGD-level convergence rate, and that, under unimodal symmetric noise, majority vote can achieve the same reduction in variance as full-precision distributed SGD [1802.04434].

More recent distributed analyses replace deterministic sign compression by an unbiased randomized sign operator. For a vector $\mathbf v$ with $\|\mathbf v\|_\infty\le R$, the operator $\operatorname{S}_R(\mathbf v)$ is defined coordinate-wise by
$$
[\operatorname{S}_R(\mathbf v)]_k =
\begin{cases}
+1,& \text{with probability } \dfrac{R+[\mathbf v]_k}{2R},\\[6pt]
-1,& \text{with probability } \dfrac{R-[\mathbf v]_k}{2R},
\end{cases}
\qquad
\mathbb{E}[\operatorname{S}_R(\mathbf v)] = \frac{\mathbf v}{R}.
$$
Using this device, the momentum-based majority-vote algorithms MVSM (v1) and MVSM (v2) obtain, respectively,
$$
\mathbb{E}\big[\|\nabla f(x_\tau)\|_1\big]
\le
\mathcal{O}\!\big(d^{1/2}T^{-1/2}+dn^{-1/2}\big),
$$
or alternatively
$$
\mathbb{E}\big[\|\nabla f(x_\tau)\|_1\big]
\le
\mathcal{O}\!\big(n^{1/2}T^{-1}+dn^{-1/2}\big),
$$
and, with unbiased sign at both workers and server,
$$
\mathbb{E}\big[\|\nabla f(x_\tau)\|\big]
\le
\mathcal{O}\!\left(\max\{d^{1/4}T^{-1/4},\, d^{1/10}T^{-1/5}\}\right).
$$
The paper stresses that both uplink and downlink remain 1-bit [2507.12091].

Momentum also matters for robustness when aggregation is non-linear. In a Byzantine-resilient setting, one paper distinguishes **server-side momentum** from **worker-side momentum** and shows that they are equivalent only for linear aggregation. For non-linear rules, including sign-based majority vote, worker-side momentum reduces the variance-norm ratio of the honest gradient estimates seen by the server. In the sign-based specialization, honest workers compute
$$
v_i^t=\beta v_i^{t-1}+g_i^t,\qquad s_i^t=\operatorname{sign}(v_i^t),
$$
the server returns
$$
\hat s^t=\operatorname{sign}\!\Big(\sum_{i=1}^n s_i^t\Big),
$$
and the model update is $x^{t+1}=x^t-\eta \hat s^t$. The paper’s conclusion is that worker-side momentum strengthens majority vote by increasing the honest margin in the coordinate-wise sign test [2003.00010].

## 5. Magnitude restoration, switching strategies, and related generalizations

A recurrent criticism of SignSGD with Momentum is that 1-bit quantization removes magnitude information. One recent refinement addresses this directly by injecting **annealed Gaussian noise before the sign operator**,
$$
x_{k+1}=x_k-\delta\,\operatorname{sign}(m_{k+1}+\xi_k),\qquad \xi_k\sim\mathcal N(0,\sigma_k^2 I),
$$
with schedule
$$
\sigma_k^2=\alpha(1+k)^{-\gamma},\qquad \gamma=0.55.
$$
For a single coordinate with signal $x>0$ and $\epsilon\sim \mathcal N(0,\sigma^2)$,
$$
\mathbb P[\operatorname{sign}(x+\epsilon)=\operatorname{sign}(x)] = \Phi(|x|/\sigma),
$$
and
$$
\mathbb E[\operatorname{sign}(x+\epsilon)] = 2\Phi(x/\sigma)-1 \approx \sqrt{\frac{2}{\pi}}\frac{x}{\sigma}
\quad\text{when } x/\sigma \approx 0.
$$
The stated interpretation is that pre-sign dithering turns hard thresholding into a probabilistic encoder whose expectation is proportional to the magnitude for small coordinates, thereby partially restoring information lost by the sign quantizer. The same study also considers post-sign injection,
$$
x_{k+1}=x_k-\delta\big(\operatorname{sign}(m_{k+1})+\xi_k\big),
$$
but reports instability or marginal gains compared to pre-sign dithering [2604.25550].

The same paper adapts the SWATS idea to sign-based updates by introducing a **projection-based switch** from SignSGD-M to SGD. The calibrated SGD learning rate is chosen so that the SGD step matches the projection of the sign step onto the current gradient direction:
$$
\lambda_k \|\tilde g_k\|_2^2 = \delta \langle \operatorname{sign}(m_{k+1}), \tilde g_k\rangle,
$$
hence
$$
\lambda_k
=
\delta\,
\frac{\big|\langle \operatorname{sign}(m_{k+1}),\tilde g_k\rangle\big|}
{\|\tilde g_k\|_2^2+\epsilon}.
$$
An EMA of $\lambda_k$ is maintained until a fixed switching epoch, after which ordinary SGD is run with the EMA value as step size. The stated purpose is to avoid magnitude mismatch at hand-off and to continue training along a smooth trajectory [2604.25550].

Several related optimizers can be read as extensions of the SignSGD-with-momentum template. Generalized SignSGD recovers Signum as the $\beta_2=0$ endpoint of an Adam-like update with $v_t$ formed from $m_t^2$ rather than $g_t^2$ [2208.11195]. The S3 optimizer retains sign-like bounded updates through a normalized momentum ratio and notes that, in a stationary-gradient regime, its update reduces to $\operatorname{Sign}(g)$; the same paper states that Lion resembles SignSGD with momentum [2507.06464]. A parameter-free “ALIAS Adam version” takes the sign of a first-moment buffer $m^{t+1}$ but multiplies it by a per-coordinate scale $S^{t+1}$ derived from an automatically updated $d^t$, a first moment, and a second moment [2506.03725]. In zero-order optimization, JAGUAR SignSGD uses coordinate momentum and updates with $\operatorname{sign}(m^t)$ while requiring two function evaluations per iteration and $2d+1$ scalar state [2506.04430].

## 6. Empirical behavior, applications, and limitations

Empirically, SignSGD with Momentum has repeatedly been competitive with adaptive optimizers, though not uniformly so. The original Signum paper reports that on ResNet-50 v2 for ImageNet, Signum matches Adam in both accuracy and convergence speed, while both methods trail a well-tuned SGD with weight decay by about $2\%$ top-1. On CIFAR-10 with ResNet-20, Signum and Adam achieve very similar final test accuracy, and momentum is explicitly identified as an important source of robustness relative to pure signSGD [1802.04434].

More recent studies isolate optimizer effects on small and medium vision models. In single-worker ResNet-18 experiments, one paper reports on CIFAR-10:
- SGD (well-tuned): **91.38%**
- Pure SignSGD-M (clean): **90.82%**
- Hybrid (SignSGD-M $\rightarrow$ SGD via projection at epoch $\sim 25$): **92.18%**

For CIFAR-100 on the same architecture, the reported values are:
- SGD (well-tuned): **~77%**
- Adam: **~67%**
- Pure SignSGD-M (clean): **~65%**
- SignSGD-M + pre-sign dithering ($\alpha=0.5$): **~73%**
- SignSGD-M + pre-sign dithering ($\alpha=0.1$): **~76%**
- SignSGD-M + post-sign dithering ($\alpha=0.1$): **~66%**
- Hybrid (SignSGD-M $\rightarrow$ SGD, switch at epoch $\sim 60$): **~72%**

The paper’s interpretation is that SignSGD-M progresses faster early, but clean sign-based updates can stagnate late; pre-sign dithering narrows the generalization gap substantially on the harder CIFAR-100 task, while the calibrated switch can outperform pure SGD on CIFAR-10 [2604.25550].

The improved-analysis paper reports additional empirical support. In the centralized setting, ResNet-18 on CIFAR-10 shows faster decrease in training loss, better accuracy, and faster decay of $\|\nabla f\|_1$ and $\|\nabla f\|_2$ for the momentum-based sign method relative to signSGD, SGD with momentum, and AdamW. In the distributed setting, ResNet-50 on CIFAR-100 across 8 nodes shows that MVSM (both v1 and v2) achieves the lowest loss and highest accuracy while using 1-bit communication in both directions [2507.12091].

The limitations are equally consistent across papers. First, 1-bit quantization discards magnitude information and can create a generalization gap relative to well-tuned SGD [2604.25550]. Second, the original geometric analysis predicts failure when gradients are sparse but curvature or noise are dense [1802.04434]. Third, some theory remains assumption-heavy: unbiased gradients, bounded variance, explicit dependence on the training horizon $T$, or unimodal symmetric noise in some analyses [2507.12091] [2604.25550]. Fourth, even when momentum improves stability, it does not introduce spectral preconditioning under generic covariance. A recent analysis of SignSGD as a proxy for Adam states that SignSGD, with or without momentum, performs no spectral preconditioning under generic covariance and does not reproduce the square-root acceleration of SignSVD or Muon [2605.09552].

In practical terms, the method is most attractive when the advantages of directional robustness and extreme communication compression dominate the cost of losing magnitude information. The literature therefore treats SignSGD with Momentum less as a single fixed optimizer than as a family centered on the update $x_{t+1}=x_t-\eta\,\operatorname{sign}(m_t)$: one branch studies sharper convergence guarantees, another restores magnitude information probabilistically, another calibrates a late switch back to SGD, and others fold the same sign-of-momentum principle into adaptive, parameter-free, or zero-order schemes [1802.04434] [2604.25550] [2507.12091].

Source: https://www.emergentmind.com/topics/signsgd-with-momentum