SignSGD with Momentum

• SignSGD with Momentum is a sign-based stochastic optimization algorithm that uses exponential moving averages of gradients to perform 1-bit, coordinate-wise updates.
• It achieves significant memory and communication savings in distributed settings by transmitting only the sign of the momentum buffer.
• Enhanced convergence analyses show that momentum smoothing, dithering, and projection switching mitigate the loss of gradient magnitude, balancing efficiency and stability.

SignSGD with Momentum is a sign-based stochastic optimization method that maintains an exponential moving average of stochastic gradients and updates parameters using only the coordinate-wise sign of that buffer. In the literature it is often called Signum. Relative to SGD with momentum, it replaces magnitude-sensitive updates by constant-magnitude coordinate steps, yielding 1-bit quantization of the update direction and therefore substantial memory and communication savings, especially in distributed settings. Its central trade-off is that the same quantization that enables compression also discards magnitude information, which can widen a generalization gap relative to well-tuned SGD or cause late-stage stagnation near minima (Bernstein et al., 2018, Chen et al., 28 Apr 2026).

1. Algorithmic form and notation

In its canonical form, SignSGD updates by the sign of the stochastic gradient,

$$x_{t+1} = x_t - \eta\, \operatorname{sign}(\tilde g_t),$$

where the sign is applied coordinate-wise. SignSGD with Momentum replaces the raw stochastic gradient by a momentum buffer,

$$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$

with $\beta \in (0,1)$ the momentum coefficient. In the terminology of the original signSGD paper, this momentum variant is Signum (Bernstein et al., 2018).

Subsequent analyses use slightly different but equivalent parameterizations. One line of work writes the buffer as an EMA,

$$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$

while another uses a heavy-ball/Polyak-style recursion,

$v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$

The EMA form differs slightly from the classical $m_t=\beta m_{t-1}+g_t$ notation, but one analysis notes that the difference is equivalent up to a rescaling of the step size and the effective momentum state (Jiang et al., 16 Jul 2025, Chen et al., 28 Apr 2026).

Algorithmically, the method has three defining properties. First, the sign operator is applied after momentum smoothing; it is not a moving average of signs. Second, the update is coordinate-wise and typically takes values in $\{-1,0,+1\}^d$ or $\{-1,+1\}^d$, depending on how zero coordinates are treated. Third, no additional per-coordinate normalization is present in the basic method beyond momentum smoothing and the sign operator (Bernstein et al., 2018, Chen et al., 28 Apr 2026).

2. Foundational theory and the geometry of sign updates

The earliest non-convex analysis of signSGD and Signum is formulated under a lower-bounded objective, coordinate-wise smoothness, and unbiased stochastic gradients with coordinate-wise bounded variances. For signSGD, the original theory used a large-batch schedule, with step size $\delta_t = 1/\sqrt{LK}$ and batch size $n_t=K$, and established a non-convex stationarity guarantee in the $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$0 norm of the gradient. For Signum, the same work gave an anytime analysis with

$$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$1

together with a momentum warmup period $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$2; the paper reports that for $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$3, $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$4 (Bernstein et al., 2018).

A distinctive feature of this theory is its emphasis on relative $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$5 geometry. The paper defines the density of a vector $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$6 by

$$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$7

with $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$8 and $$m_{t+1} = \beta m_t + (1-\beta)\,\tilde g_t,\qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(m_{t+1}),$$9. The resulting comparison between signSGD and SGD is not purely asymptotic; it depends on whether gradients are dense relative to curvature and stochastic noise. The original interpretation is that if gradients are at least as dense as curvature and stochasticity, signSGD can match or beat SGD while also providing compression, whereas if curvature and noise are much denser than gradients, SGD is preferable (Bernstein et al., 2018).

The same paper also established the distributed majority-vote formulation: each worker transmits the sign of its local stochastic gradient, the server aggregates coordinate-wise by majority vote,

$\beta \in (0,1)$0

and all workers update with $\beta \in (0,1)$1. Under unimodal symmetric noise, majority vote was shown to improve the variance term by $\beta \in (0,1)$2 and to compress worker-server communication to 1 bit per coordinate in both directions (Bernstein et al., 2018).

3. Momentum-based convergence under weaker assumptions

Later work substantially broadened the convergence theory for sign-based methods with momentum. Under standard $\beta \in (0,1)$3-smoothness and bounded variance, the “Sign-based Momentum Method” proved that SignSGD with momentum can achieve

$\beta \in (0,1)$4

with $\beta \in (0,1)$5 and $\beta \in (0,1)$6. Under $\beta \in (0,1)$7-smoothness and separable bounded noise, the same paper proved

$\beta \in (0,1)$8

These results remove the need for large batch sizes and, in the $\beta \in (0,1)$9-smooth case, remove the need for unimodal symmetric stochastic noise. The paper states that this improves the prior momentum-based signSGD result by a factor of $$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$0 (Jiang et al., 16 Jul 2025).

The mechanism behind these bounds is an explicit control of the momentum error $$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$1. A key descent inequality is

$$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$2

paired with a recursion for $$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$3 that makes the variance-reduction role of momentum quantitative (Jiang et al., 16 Jul 2025).

A different line of analysis revisited SignSGD from a 1-bit quantization viewpoint and derived a small-batch rate under unimodal symmetric gradient noise. It introduced the signal-to-noise weighted stationarity measure

$$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$4

where $$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$5. With constant batch size $$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$6 and step size $$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$7, the paper proved

$$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$8

and

$$m_{k+1} = \beta m_k + (1-\beta)\tilde g_k,\qquad x_{k+1} = x_k - \delta\, \operatorname{sign}(m_{k+1}),$$9

The paper emphasizes that, unlike prior large-batch analyses, this rate holds for any fixed small batch $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$0, and that the $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$1-gradient bound contains a non-vanishing noise floor $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$2 intrinsic to 1-bit quantization at low SNR (Chen et al., 28 Apr 2026).

A third theoretical direction concerns unbounded smoothness. The generalized SignSGD framework studies

$v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$3

and observes that the endpoint $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$4 recovers SignSGD with momentum because then $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$5 and the update becomes $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$6. Under coordinate-wise $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$7-smoothness and bounded noise, the paper proves a $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$8-type stationarity bound and states that the $v_t = \begin{cases} \nabla f(x_1;\xi_1), & t=1,\[3pt] (1-\beta)\,v_{t-1} + \beta\, \nabla f(x_t;\xi_t), & t\ge 2, \end{cases} \qquad x_{t+1} = x_t - \eta\, \operatorname{sign}(v_t).$9 endpoint matches clipped-SGD rates up to norm conventions, while removing detrimental dependence on the unbounded-smoothness term and large gradient norms in the final bound (Crawshaw et al., 2022).

4. Distributed, communication-efficient, and robust variants

The original appeal of SignSGD with Momentum is inseparable from communication efficiency. In distributed majority-vote signSGD, each worker sends 1 bit per coordinate to the server and receives 1 bit per coordinate back. The 2018 analysis argued that this yields compressed gradients and an SGD-level convergence rate, and that, under unimodal symmetric noise, majority vote can achieve the same reduction in variance as full-precision distributed SGD (Bernstein et al., 2018).

More recent distributed analyses replace deterministic sign compression by an unbiased randomized sign operator. For a vector $m_t=\beta m_{t-1}+g_t$0 with $m_t=\beta m_{t-1}+g_t$1, the operator $m_t=\beta m_{t-1}+g_t$2 is defined coordinate-wise by

$m_t=\beta m_{t-1}+g_t$3

Using this device, the momentum-based majority-vote algorithms MVSM (v1) and MVSM (v2) obtain, respectively,

$m_t=\beta m_{t-1}+g_t$4

or alternatively

$m_t=\beta m_{t-1}+g_t$5

and, with unbiased sign at both workers and server,

$m_t=\beta m_{t-1}+g_t$6

The paper stresses that both uplink and downlink remain 1-bit (Jiang et al., 16 Jul 2025).

Momentum also matters for robustness when aggregation is non-linear. In a Byzantine-resilient setting, one paper distinguishes server-side momentum from worker-side momentum and shows that they are equivalent only for linear aggregation. For non-linear rules, including sign-based majority vote, worker-side momentum reduces the variance-norm ratio of the honest gradient estimates seen by the server. In the sign-based specialization, honest workers compute

$m_t=\beta m_{t-1}+g_t$7

the server returns

$m_t=\beta m_{t-1}+g_t$8

and the model update is $m_t=\beta m_{t-1}+g_t$9. The paper’s conclusion is that worker-side momentum strengthens majority vote by increasing the honest margin in the coordinate-wise sign test (El-Mhamdi et al., 2020).

5. Magnitude restoration, switching strategies, and related generalizations

A recurrent criticism of SignSGD with Momentum is that 1-bit quantization removes magnitude information. One recent refinement addresses this directly by injecting annealed Gaussian noise before the sign operator,

$\{-1,0,+1\}^d$0

with schedule

$\{-1,0,+1\}^d$1

For a single coordinate with signal $\{-1,0,+1\}^d$2 and $\{-1,0,+1\}^d$3,

$\{-1,0,+1\}^d$4

and

$\{-1,0,+1\}^d$5

The stated interpretation is that pre-sign dithering turns hard thresholding into a probabilistic encoder whose expectation is proportional to the magnitude for small coordinates, thereby partially restoring information lost by the sign quantizer. The same study also considers post-sign injection,

$\{-1,0,+1\}^d$6

but reports instability or marginal gains compared to pre-sign dithering (Chen et al., 28 Apr 2026).

The same paper adapts the SWATS idea to sign-based updates by introducing a projection-based switch from SignSGD-M to SGD. The calibrated SGD learning rate is chosen so that the SGD step matches the projection of the sign step onto the current gradient direction:

$\{-1,0,+1\}^d$7

hence

$\{-1,0,+1\}^d$8

An EMA of $\{-1,0,+1\}^d$9 is maintained until a fixed switching epoch, after which ordinary SGD is run with the EMA value as step size. The stated purpose is to avoid magnitude mismatch at hand-off and to continue training along a smooth trajectory (Chen et al., 28 Apr 2026).

Several related optimizers can be read as extensions of the SignSGD-with-momentum template. Generalized SignSGD recovers Signum as the $\{-1,+1\}^d$0 endpoint of an Adam-like update with $\{-1,+1\}^d$1 formed from $\{-1,+1\}^d$2 rather than $\{-1,+1\}^d$3 (Crawshaw et al., 2022). The S3 optimizer retains sign-like bounded updates through a normalized momentum ratio and notes that, in a stationary-gradient regime, its update reduces to $\{-1,+1\}^d$4; the same paper states that Lion resembles SignSGD with momentum (Peng et al., 9 Jul 2025). A parameter-free “ALIAS Adam version” takes the sign of a first-moment buffer $\{-1,+1\}^d$5 but multiplies it by a per-coordinate scale $\{-1,+1\}^d$6 derived from an automatically updated $\{-1,+1\}^d$7, a first moment, and a second moment (Medyakov et al., 4 Jun 2025). In zero-order optimization, JAGUAR SignSGD uses coordinate momentum and updates with $\{-1,+1\}^d$8 while requiring two function evaluations per iteration and $\{-1,+1\}^d$9 scalar state (Petrov et al., 4 Jun 2025).

6. Empirical behavior, applications, and limitations

Empirically, SignSGD with Momentum has repeatedly been competitive with adaptive optimizers, though not uniformly so. The original Signum paper reports that on ResNet-50 v2 for ImageNet, Signum matches Adam in both accuracy and convergence speed, while both methods trail a well-tuned SGD with weight decay by about $\delta_t = 1/\sqrt{LK}$0 top-1. On CIFAR-10 with ResNet-20, Signum and Adam achieve very similar final test accuracy, and momentum is explicitly identified as an important source of robustness relative to pure signSGD (Bernstein et al., 2018).

More recent studies isolate optimizer effects on small and medium vision models. In single-worker ResNet-18 experiments, one paper reports on CIFAR-10:

• SGD (well-tuned): 91.38%
• Pure SignSGD-M (clean): 90.82%
• Hybrid (SignSGD-M $\delta_t = 1/\sqrt{LK}$1 SGD via projection at epoch $\delta_t = 1/\sqrt{LK}$2): 92.18%

For CIFAR-100 on the same architecture, the reported values are:

• SGD (well-tuned): ~77%
• Adam: ~67%
• Pure SignSGD-M (clean): ~65%
• SignSGD-M + pre-sign dithering ($\delta_t = 1/\sqrt{LK}$3): ~73%
• SignSGD-M + pre-sign dithering ($\delta_t = 1/\sqrt{LK}$4): ~76%
• SignSGD-M + post-sign dithering ($\delta_t = 1/\sqrt{LK}$5): ~66%
• Hybrid (SignSGD-M $\delta_t = 1/\sqrt{LK}$6 SGD, switch at epoch $\delta_t = 1/\sqrt{LK}$7): ~72%

The paper’s interpretation is that SignSGD-M progresses faster early, but clean sign-based updates can stagnate late; pre-sign dithering narrows the generalization gap substantially on the harder CIFAR-100 task, while the calibrated switch can outperform pure SGD on CIFAR-10 (Chen et al., 28 Apr 2026).

The improved-analysis paper reports additional empirical support. In the centralized setting, ResNet-18 on CIFAR-10 shows faster decrease in training loss, better accuracy, and faster decay of $\delta_t = 1/\sqrt{LK}$8 and $\delta_t = 1/\sqrt{LK}$9 for the momentum-based sign method relative to signSGD, SGD with momentum, and AdamW. In the distributed setting, ResNet-50 on CIFAR-100 across 8 nodes shows that MVSM (both v1 and v2) achieves the lowest loss and highest accuracy while using 1-bit communication in both directions (Jiang et al., 16 Jul 2025).

The limitations are equally consistent across papers. First, 1-bit quantization discards magnitude information and can create a generalization gap relative to well-tuned SGD (Chen et al., 28 Apr 2026). Second, the original geometric analysis predicts failure when gradients are sparse but curvature or noise are dense (Bernstein et al., 2018). Third, some theory remains assumption-heavy: unbiased gradients, bounded variance, explicit dependence on the training horizon $n_t=K$0, or unimodal symmetric noise in some analyses (Jiang et al., 16 Jul 2025, Chen et al., 28 Apr 2026). Fourth, even when momentum improves stability, it does not introduce spectral preconditioning under generic covariance. A recent analysis of SignSGD as a proxy for Adam states that SignSGD, with or without momentum, performs no spectral preconditioning under generic covariance and does not reproduce the square-root acceleration of SignSVD or Muon (Paquette et al., 10 May 2026).

In practical terms, the method is most attractive when the advantages of directional robustness and extreme communication compression dominate the cost of losing magnitude information. The literature therefore treats SignSGD with Momentum less as a single fixed optimizer than as a family centered on the update $n_t=K$1: one branch studies sharper convergence guarantees, another restores magnitude information probabilistically, another calibrates a late switch back to SGD, and others fold the same sign-of-momentum principle into adaptive, parameter-free, or zero-order schemes (Bernstein et al., 2018, Chen et al., 28 Apr 2026, Jiang et al., 16 Jul 2025).