signSGD: Efficient Distributed Optimization

• signSGD is a sign-based first-order optimization method that replaces gradient values with their sign, ensuring fixed per-coordinate update magnitudes.
• It leverages 1-bit communication with majority vote aggregation to significantly reduce communication costs in large-scale distributed training.
• Variants like Signum and StoSignSGD address issues of bias and numerical stability, improving convergence in non-convex, low-precision, and heterogeneous data regimes.

signSGD is a sign-based first-order optimization method that replaces each gradient coordinate by its sign and therefore updates with a fixed per-coordinate magnitude rather than a magnitude proportional to the gradient itself. In its canonical stochastic form, the method uses

$$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$

and was introduced as a communication-frugal optimizer for large-scale non-convex learning, especially in distributed deep learning where transmitting full-precision gradients is costly (Bernstein et al., 2018). The method rapidly became a reference point for several adjacent literatures: compressed optimization, distributed majority-vote training, error-feedback correction, sign-based adaptive methods, federated and hierarchical learning, and more recent analyses of non-smooth, small-batch, and large-language-model training regimes (Bernstein et al., 2018).

1. Core formulation

The basic signSGD update applies the sign operator coordinate-wise to a stochastic gradient estimate. For a differentiable objective $f(x)$, iterate $x_k\in\mathbb{R}^d$, stochastic gradient $\tilde g_k$, and stepsize $\delta_k$, the update is

$$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$

where $\operatorname{sign}(v)_i\in\{-1,+1\}$ is taken coordinate-wise; the zero case can be handled by either convention without affecting the analysis (Bernstein et al., 2018). The defining feature is that the step magnitude on each coordinate is $\delta_k$, independent of $|\tilde g_{k,i}|$.

A momentum counterpart, usually called Signum, replaces the instantaneous sign by the sign of an exponential moving average,

$$m_{k+1}=\beta m_k+(1-\beta)\tilde g_k,\qquad x_{k+1}=x_k-\delta_k\,\operatorname{sign}(m_{k+1}),$$

with $f(x)$0 (Bernstein et al., 2018). In the original analysis, Signum is the practically important variant; in the empirical results on deep vision models, it is the momentum form rather than raw signSGD that is compared against Adam.

The method also has a direct relationship to adaptive optimization. One line of work states that signSGD is closely connected to Adam and RMSProp: if Adam’s exponential moving averages are taken with very short timescales, the resulting step is essentially $f(x)$1; a later distributed-systems paper states more sharply that signSGD is a special case of Adam when $f(x)$2 (Bernstein et al., 2018, Bernstein et al., 2018). This connection partly explains why signSGD has often been studied both as a compressor and as a minimal model for sign-like adaptive updates.

At a geometric level, non-stochastic sign gradient descent is classical steepest descent in the $f(x)$3 norm (Bernstein et al., 2018). In stochastic settings, however, the analysis is not a trivial corollary of deterministic steepest descent, because the sign operator is biased and because the quality of each sign bit depends on per-coordinate signal-to-noise ratios.

2. Foundational assumptions and original convergence theory

The original non-convex analysis is built on three assumptions. First, the objective is lower bounded: $f(x)$4. Second, $f(x)$5 is coordinate-wise smooth: there exist nonnegative constants $f(x)$6 such that

$f(x)$7

where $f(x)$8. Third, the stochastic gradient oracle is unbiased with coordinate-wise bounded variance,

$f(x)$9

For a minibatch of size $x_k\in\mathbb{R}^d$0, the variance scales as $x_k\in\mathbb{R}^d$1 (Bernstein et al., 2018).

Under these assumptions, the original paper proves a non-convex convergence rate for single-worker signSGD in a large-batch-per-iteration regime. With $x_k\in\mathbb{R}^d$2, $x_k\in\mathbb{R}^d$3 iterations, stepsizes $x_k\in\mathbb{R}^d$4, and minibatch size $x_k\in\mathbb{R}^d$5, the total number of stochastic gradient calls is $x_k\in\mathbb{R}^d$6, and the averaged stationarity measure in $x_k\in\mathbb{R}^d$7 geometry satisfies an $x_k\in\mathbb{R}^d$8 rate (Bernstein et al., 2018). The proof uses the one-step improvement inequality

$x_k\in\mathbb{R}^d$9

together with a coordinate-wise sign-error bound,

$\tilde g_k$0

The resulting stationarity certificate is therefore naturally expressed in $\tilde g_k$1, not $\tilde g_k$2, terms.

For Signum, the theorem is cast in an anytime form with decaying stepsize and growing batch size,

$\tilde g_k$3

plus a short warmup $\tilde g_k$4 to let the momentum bias settle. For $\tilde g_k$5, the paper reports $\tilde g_k$6 (Bernstein et al., 2018). The asymptotic bound again scales as $\tilde g_k$7, but the constants reflect an explicit bias-variance trade-off: larger $\tilde g_k$8 reduces the variance term via longer averaging and simultaneously increases the curvature-induced bias term.

Subsequent theory relaxed some aspects of this original regime. A later analysis under random reshuffling proved the first convergence result for sign-based methods in nonconvex finite-sum optimization without requiring per-iteration batch sizes to be of the same order as the total number of iterations; it obtained $\tilde g_k$9 for SignRR and $\delta_k$0 for variance-reduced and momentum variants (Qin et al., 2023). More recently, a small-batch theory introduced the signal-to-noise-weighted stationarity measure

$\delta_k$1

and established an $\delta_k$2 rate for constant batch size under unimodal symmetric gradient noise, thereby removing the original large-batch assumption (Chen et al., 28 Apr 2026).

3. Majority vote, 1-bit communication, and distributed training

The distributed version of signSGD uses a parameter server. Each of $\delta_k$3 workers computes a local stochastic gradient $\delta_k$4, sends the sign vector

$\delta_k$5

and the server aggregates by coordinate-wise majority vote,

$\delta_k$6

The server then broadcasts $\delta_k$7, and every worker updates

$\delta_k$8

This yields 1-bit compression in both directions: workers send 1 bit per coordinate to the server, and the server sends 1 bit per coordinate back to each worker (Bernstein et al., 2018).

The original distributed theory proves two levels of guarantee. Under the basic assumptions above, majority vote is at least as good as single-worker signSGD. Under an additional unimodal symmetric per-coordinate noise assumption, it improves the stochastic term exactly as full-precision distributed SGD does at the level of the analysis: $\delta_k$9 is replaced by $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$0 in the bound (Bernstein et al., 2018). The key quantity is the per-coordinate signal-to-noise ratio $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$1. For a single worker, the sign error probability is bounded by $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$2 without symmetry assumptions, and by a sharper Gauss-inequality-based bound under unimodal symmetric noise: $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$3 Majority vote then reduces the error approximately like $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$4 (Bernstein et al., 2018).

This compression is exceptionally aggressive. One comparison reports per-iteration communication of $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$5 bits for full-precision distributed SGD, $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$6 bits for QSGD or TernGrad, and $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$7 bits for signSGD with majority vote (Bernstein et al., 2018). A later systems paper summarized the same contrast as a $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$8 communication reduction relative to full-precision distributed SGD and argued that majority vote is robust when up to 50% of workers behave adversarially in the blind multiplicative model (Bernstein et al., 2018).

Distributed sign aggregation also admits coding-theoretic interpretations. Majority vote is the maximum-likelihood decoder of a repetition code of the true sign, which explains why repeated independent sign bits reduce error with additional workers (Bernstein et al., 2018). Later work on Byzantine robustness made this viewpoint explicit through “Election Coding,” which preserved 1-bit communication while adding computational redundancy to tolerate malicious workers (Sohn et al., 2019).

4. Geometry, preconditioning, and characteristic failure modes

The most distinctive aspect of signSGD theory is its $$x_{k+1}=x_k-\delta_k\,\operatorname{sign}(\tilde g_k),$$9 geometric viewpoint. The original paper defines a density measure

$\operatorname{sign}(v)_i\in\{-1,+1\}$0

which is close to $\operatorname{sign}(v)_i\in\{-1,+1\}$1 for dense vectors and close to $\operatorname{sign}(v)_i\in\{-1,+1\}$2 for highly sparse vectors (Bernstein et al., 2018). This allows the comparison

$\operatorname{sign}(v)_i\in\{-1,+1\}$3

From these quantities, the paper defines the dimensionless ratios

$\operatorname{sign}(v)_i\in\{-1,+1\}$4

where $\operatorname{sign}(v)_i\in\{-1,+1\}$5 is a lower bound on gradient density across iterates (Bernstein et al., 2018).

The interpretation is direct. If gradients are as dense as or denser than curvature and noise, signSGD can match or beat SGD while greatly reducing communication. If curvature and stochasticity are much denser than gradients, SGD is preferable, because signSGD may push many coordinates in highly curved directions even when their gradients are small (Bernstein et al., 2018). Empirical measurements on ResNet-20/CIFAR-10 found that both gradients and noise were dense and appeared coupled over training, which is consistent with the regime where signSGD is competitive.

The method’s failure modes are equally characteristic. Under distributed data heterogeneity, majority vote does not track the mean gradient but the coordinate-wise median of local gradients. A later analysis formalized this by proving that, for odd $\operatorname{sign}(v)_i\in\{-1,+1\}$6,

$\operatorname{sign}(v)_i\in\{-1,+1\}$7

and showed that with non-iid workers the expected median can disagree with the global mean gradient. In that case signSGD can converge to a point with nonzero true gradient (Chen et al., 2019). This explains why the homogeneous-worker assumption is not a merely technical convenience in classical majority-vote analysis.

A second failure mode is the bias of deterministic sign compression itself. Convex counterexamples were used to show that plain signSGD can fail to converge to the optimum and can generalize poorly relative to SGD, precisely because the sign compressor is biased and discards magnitude information (Karimireddy et al., 2019). More recent work pushed this point further in non-smooth optimization: deterministic sign updates can diverge on non-smooth objectives such as those induced by ReLUs, max-pools, and mixture-of-experts gating, which motivated structurally stochastic, unbiased sign operators in later variants (Yu et al., 16 Apr 2026).

A complementary theoretical perspective comes from an exact high-dimensional analysis of signSGD on linear least squares. That work derived limiting SDE and ODE descriptions showing four explicit effects of the sign operator: an effective learning-rate factor, noise compression through the label-noise distribution, diagonal preconditioning via $\operatorname{sign}(v)_i\in\{-1,+1\}$8, and gradient-noise reshaping through a sign covariance $\operatorname{sign}(v)_i\in\{-1,+1\}$9 (Xiao et al., 2024). This suggests that the often-invoked “preconditioning” effect of sign-based methods is not only heuristic but can be quantified in analytically solvable models.

5. Corrections, extensions, and new operating regimes

A large part of the later literature can be understood as an attempt either to repair signSGD’s bias or to extend it to settings where the original analysis is too restrictive. The most influential repair is error feedback. For a general compressor $\delta_k$0, EF-SGD stores a residual $\delta_k$1 and updates via

$\delta_k$2

Applied to the scaled sign compressor $\delta_k$3, error feedback removes the leading-order optimization penalty of biased compression: the non-convex convergence rate matches that of SGD, and empirical results show substantial improvements in both convergence and generalization relative to plain signSGD and Signum (Karimireddy et al., 2019).

Another strand addresses distributed non-iid data. “Stochastic Sign Descent with Momentum” was proposed specifically because signSGD fails in partitioned-data distributed training; the new method replaces deterministic signs by a stochastic sign operator and proves convergence under standard bounded-variance assumptions with the optimal asymptotic rate in that regime (Safaryan et al., 2019). A different solution for heterogeneous workers is to correct majority vote itself: signSGD with federated voting learns per-worker reliability weights online and uses weighted majority voting, which restores a convergence guarantee when workers use heterogeneous minibatch sizes and ordinary majority vote can fail (Park et al., 2024).

Variance reduction has also been adapted to sign-based optimization. “Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction” improves the classical expectation-case rate from $\delta_k$4 to $\delta_k$5 with SSVR, and further to $\delta_k$6 in finite-sum problems (Jiang et al., 2024). Random-reshuffling variants, SignRVR and SignRVM, provide similar improvements under the practically dominant epoch-wise data access model rather than with-replacement sampling (Qin et al., 2023).

Recent work has also re-examined signSGD through quantization and stochasticity. One 2026 paper interprets the sign operator as a 1-bit quantizer and introduces pre-sign Gaussian dithering with annealed variance

$\delta_k$7

together with a projection-calibrated switch from SignSGD with momentum to SGD (Chen et al., 28 Apr 2026). Another 2026 paper proposes StoSignSGD, which replaces the deterministic sign by a structurally stochastic operator

$\delta_k$8

and exploits the identity $\delta_k$9 coordinate-wise. In online convex optimization this yields a dimension-free regret bound matching a lower bound up to constants, and in non-convex non-smooth optimization it improves earlier complexity bounds by dimensional factors (Yu et al., 16 Apr 2026). These results directly target the regime in which deterministic sign compression is least reliable.

6. Empirical record, applications, and broader legacy

The original empirical picture was mixed but influential. On ImageNet with ResNet-50 v2, Signum matched Adam in training speed and final test accuracy, but was approximately $|\tilde g_{k,i}|$0 worse than a well-tuned SGD baseline with weight decay; it nevertheless outperformed SGD without weight decay (Bernstein et al., 2018). On CIFAR-10 with ResNet-20, extensive sweeps showed that Signum closely tracked Adam across settings, and both were competitive with SGD (Bernstein et al., 2018). These results established sign-based optimization as a serious practical baseline rather than a purely compression-oriented curiosity.

In distributed systems, the communication story was strong enough to motivate dedicated implementations. A PyTorch parameter-server system for majority-vote Signum reported about a $|\tilde g_{k,i}|$1 reduction in wall-clock training time for ResNet-50 on ImageNet when using 15 AWS p3.2xlarge machines, while preserving the 1-bit communication pattern in both uplink and downlink (Bernstein et al., 2018). That paper also framed majority vote as fault tolerant relative to full-precision SGD, because no single worker can dominate the aggregated update through an arbitrarily large vector.

The method has also migrated into domains far removed from the original ImageNet setting. In meta-learning, “Sign-MAML” uses signSGD in the inner loop of model-agnostic meta-learning. The resulting first-order update is derived exactly from the sign operator’s almost-everywhere zero derivative, rather than by dropping Hessian terms heuristically; on FS-CIFAR100, it reported $|\tilde g_{k,i}|$2 for 5-way 1-shot versus $|\tilde g_{k,i}|$3 for FO-MAML and $|\tilde g_{k,i}|$4 for MAML, with time per iteration essentially the same as FO-MAML and about half of MAML (Fan et al., 2021). In zero-order optimization, JAGUAR SignSGD introduced coordinate momentum for sign-based finite-difference learning and provided what it describes as the first rigorous stochastic non-convex convergence guarantee for zero-order SignSGD, while matching or exceeding first-order baselines on several large-language-model fine-tuning benchmarks with substantially reduced memory (Petrov et al., 4 Jun 2025).

More recent large-model evidence is especially notable in low-precision and non-smooth regimes. StoSignSGD was reported to remain stable in FP8 pretraining where AdamW failed catastrophically, with $|\tilde g_{k,i}|$5 to $|\tilde g_{k,i}|$6 speedups relative to established baselines, and to improve fine-tuning accuracy on 7B LLMs for mathematical reasoning (Yu et al., 16 Apr 2026). A plausible implication is that sign-based optimization now occupies two distinct roles: a communication primitive for distributed learning and a numerical-stability primitive for low-precision or non-smooth large-model training.

Across these developments, signSGD has remained a conceptually compact optimizer with unusually wide reach. Its core update rule is minimal, but the later literature has shown that its real significance lies in the structures it exposes: coordinate-wise descent in $|\tilde g_{k,i}|$7 geometry, 1-bit communication, majority-vote decoding, magnitude-free robustness, and the precise ways in which bias, heterogeneity, and non-smoothness alter optimization dynamics (Bernstein et al., 2018, Xiao et al., 2024).