Sign Gradient Descent: An Optimization Overview

• Sign Gradient Descent is a family of first-order optimization methods that replaces full gradient values with their coordinate-wise sign, enabling ℓ∞ steepest descent.
• Variants such as signSGD, Signum, and scaled SignGD adapt the basic algorithm for stochastic settings, momentum acceleration, and deterministic convergence, respectively.
• The method is valued for its communication efficiency and effectiveness on heavy-tailed data, with applications spanning deep learning and language model training.

Sign Gradient Descent (SignGD) denotes a family of first-order optimization methods that replace each gradient coordinate by its sign, so the update direction is determined coordinate-wise by $\pm 1$ or $0$ rather than by the full gradient magnitude. In its basic deterministic form, the update is $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$; in stochastic settings signSGD uses the sign of a stochastic gradient, and momentum variants such as Signum use the sign of an exponential moving average. The method has been studied as a communication-efficient optimizer, as steepest descent in $\ell_\infty$ geometry, and as a simplified proxy for Adam-like adaptive methods, particularly in settings where coordinate-wise normalization matters more than Euclidean scaling (Balles et al., 2020, Bernstein et al., 2018, Kunstner et al., 25 May 2025).

1. Core formulations and variants

At the algorithmic level, SignGD differs from classical gradient descent only in the replacement of the gradient by its coordinate-wise sign. For a differentiable objective $f:\mathbb{R}^d\to\mathbb{R}$, standard gradient descent uses $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$, whereas plain SignGD uses $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$. The stochastic analogue signSGD replaces $\nabla f(x_k)$ by a stochastic gradient $\hat g_k$, and momentum versions such as Signum apply the sign operator to a momentum buffer rather than to the raw gradient (Bernstein et al., 2018).

A distinct but closely related variant is scaled SignGD, which multiplies the sign vector by the gradient’s $\ell_1$-norm: $0$0 This update restores a global magnitude term while preserving one-bit directional information per coordinate, and it plays a central role in convergence analyses carried out in $0$1-geometry (Li et al., 2021).

Variant	Update	Characteristic use
SignGD	$0$2	Deterministic sign descent
signSGD	$0$3	Stochastic optimization
Signum	$0$4	Momentum sign method
Scaled SignGD	$0$5	$0$6-steepest descent
Bigram sign descent	$0$7	Proxy for Adam in heavy-tailed LLMs

These formulations share the same basic structural feature: every active coordinate receives a uniform step magnitude. That property is central both to the practical attraction of sign-based methods and to their theoretical deviations from Euclidean gradient descent.

2. Geometric interpretation

A major theoretical interpretation of SignGD is that it is steepest descent with respect to the $0$8 norm. Under $0$9-smoothness,

$$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$0

the steepest descent direction becomes $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$1, yielding the norm-scaled sign update

$$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$2

In this view, classical gradient descent is the $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$3-steepest descent method, while SignGD is the $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$4-steepest descent method (Balles et al., 2020).

This geometric perspective also unifies earlier analyses based on separable smoothness. If

$$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$5

then $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$6 is $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$7-smooth with constant $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$8. The paper “The Geometry of Sign Gradient Descent” argues that $$x_{t+1}=x_t-\eta\,\operatorname{sign}(\nabla f(x_t))$$9-smoothness is the weaker and more natural abstraction for sign-based methods because those methods never exploit finer coordinate-wise smoothness constants directly (Balles et al., 2020).

The same work identifies Hessian structure that favors sign methods. Two conditions are singled out: the Hessian should be to some degree concentrated on its diagonal, and its maximal eigenvalue should be much larger than the average eigenvalue. In the paper’s formulation, sign-based methods are preferable over gradient descent if the Hessian is axis-aligned enough and the spectrum has a few large outliers with a bulk near zero. This suggests that sign-based updates are geometrically well matched to objectives with strong diagonal or block-diagonal structure and sharply anisotropic spectra, a pattern the paper notes is common in deep networks (Balles et al., 2020).

The connection to adaptive methods follows the same line. Adam’s update can be decomposed into a coordinate-wise damping factor multiplied by $\ell_\infty$0, and experiments reported in the geometry paper show that shuffling or averaging those damping factors across coordinates yields performance extremely similar to original Adam in CNN training. The paper interprets this as evidence that the sign component is primary and the elementwise scaling is secondary (Balles et al., 2020).

3. Deterministic convergence theory and scaled sign descent

Plain constant-step SignGD is not uniformly convergent. “On Faster Convergence of Scaled Sign Gradient Descent” gives a simple strongly convex counterexample: for $\ell_\infty$1, starting from $\ell_\infty$2, plain SignGD oscillates between $\ell_\infty$3 and $\ell_\infty$4 and never converges. This behavior motivates scaled SignGD, whose update direction is still sign-based but whose step length is multiplied by $\ell_\infty$5 (Li et al., 2021).

Under $\ell_\infty$6-smoothness with respect to $\ell_\infty$7, scaled SignGD satisfies a one-step descent bound of the form

$\ell_\infty$8

for $\ell_\infty$9, where $f:\mathbb{R}^d\to\mathbb{R}$0. If the objective is $f:\mathbb{R}^d\to\mathbb{R}$1-strongly convex with respect to $f:\mathbb{R}^d\to\mathbb{R}$2, or more generally satisfies the Polyak–Łojasiewicz inequality

$f:\mathbb{R}^d\to\mathbb{R}$3

then the method converges linearly in function value: $f:\mathbb{R}^d\to\mathbb{R}$4 If only smoothness is available, the same analysis yields the standard sublinear stationarity bound

$f:\mathbb{R}^d\to\mathbb{R}$5

Thus scaled SignGD restores deterministic descent guarantees comparable to classical first-order methods, but in the $f:\mathbb{R}^d\to\mathbb{R}$6 geometry natural to sign updates (Li et al., 2021).

This literature also sharpens the distinction between plain sign methods and scaled sign methods. Plain SignGD uses fixed per-coordinate steps and can persistently overshoot; scaled SignGD preserves the sign direction while reintroducing a global gradient magnitude, which is enough to recover linear convergence under strong convexity or PL. The resulting method remains “purely sign” in the sense emphasized by the paper, but no longer inherits the simplest oscillatory failure mode of unscaled SignGD (Li et al., 2021).

4. Stochastic, parallel, and distributed sign methods

The original signSGD analysis in non-convex optimization emphasized coordinate-wise geometry and communication efficiency. Under coordinate-wise smoothness and bounded coordinate-wise variance, signSGD achieves a non-convex convergence rate comparable to SGD, but expressed through an $f:\mathbb{R}^d\to\mathbb{R}$7-based gradient quantity; the same paper extends the method to distributed learning by majority vote and proves that, under a unimodal symmetric noise assumption, majority vote achieves the same reduction in variance as full precision distributed SGD while using 1-bit compression in both directions (Bernstein et al., 2018).

A subsequent line of work reframed this analysis through success probabilities. “Stochastic Sign Descent Methods: New Algorithms and Better Theory” assumes that each coordinate sign is correct with probability $f:\mathbb{R}^d\to\mathbb{R}$8, defines a $f:\mathbb{R}^d\to\mathbb{R}$9-weighted norm

$x_{k+1}=x_k-\alpha_k\nabla f(x_k)$0

and proves single-node and shared-data parallel convergence in this metric. In the shared-data parallel setting, majority vote improves the effective success probabilities and yields exponential noise reduction with the number of nodes. The same paper also identifies a fundamental issue with deterministic sign aggregation in the partitioned-data setting: sign aggregation is invariant under positive rescaling of local objectives, even though the global objective changes. To resolve this, it introduces Stochastic Sign Descent with Momentum (SSDM), which uses a stochastic sign operator and converges under standard bounded variance assumptions with the optimal asymptotic $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$1 complexity for non-convex stationarity (Safaryan et al., 2019).

Variance reduction became the next major development. “SignSVRG: fixing SignSGD via variance reduction” combines an SVRG-style estimator with a sign-plus-uniform-noise construction and obtains an $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$2 rate for expected gradient norm in the smooth non-convex finite-sum setting and an $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$3 rate in the smooth convex case, thereby recovering deterministic-like convergence guarantees while preserving sign-based updates (Chzhen et al., 2023). “Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction” then improves the general stochastic signSGD rate from $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$4 to $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$5, and for finite sums attains $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$6; it also develops heterogeneous-majority-vote distributed variants with improved rates over earlier sign methods (Jiang et al., 2024).

Taken together, this theory establishes a stable pattern. Sign-based methods derive their advantages from coordinate-wise normalization and communication compression, but naïve sign compression is biased. Modern analyses either alter the update rule—through scaling, stochastic sign operators, or momentum—or alter the estimator—through variance reduction or majority vote—to recover the convergence properties lost by plain sign descent.

5. Heavy-tailed data, Zipf’s law, and language-model training

A major recent development is the use of SignGD to explain optimizer behavior in language modeling under heavy-tailed token statistics. “Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law” studies a linear next-token prediction model with token frequencies

$x_{k+1}=x_k-\alpha_k\nabla f(x_k)$7

with the key natural-language case $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$8, i.e. Zipf’s law. In that model, the Hessian eigenvalues are exactly the token frequencies, $x_{k+1}=x_k-\alpha_k\nabla f(x_k)$9, and the initial coordinates relative to the optimum are the conditional frequencies $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$0. Sign descent is defined coordinate-wise by

$$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$1

and is used as a deterministic proxy for Adam and other sign-based adaptive optimizers (Kunstner et al., 25 May 2025).

The scaling laws are striking. For gradient descent, the Zipf case $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$2 is “worst-case”: to reach small relative error $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$3, the number of iterations scales as $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$4, nearly linear in vocabulary size. For sign descent, by contrast, the relevant regime is $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$5, which includes Zipf, and the iteration complexity scales as

$$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$6

The paper’s interpretation is that heavy tails create many low-frequency directions with tiny eigenvalues; gradient descent makes almost no progress in those directions, whereas sign descent gives them uniform per-coordinate steps and thereby avoids being throttled by the smallest eigenvalues. For Zipf-distributed data, this yields square-root rather than almost-linear dependence on $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$7 (Kunstner et al., 25 May 2025).

A complementary softmax analysis reaches a similar conclusion from a different angle. “Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance” studies KL minimization for a softmax unigram model with $$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$8. For normalized steepest descent with weight decay, it defines the complexity

$$x_{k+1}=x_k-\alpha_k\operatorname{sign}(\nabla f(x_k))$$9

and shows that in $\nabla f(x_k)$0 geometry, corresponding to SignGD, the complexity scales as $\nabla f(x_k)$1, whereas in $\nabla f(x_k)$2 geometry it scales as $\nabla f(x_k)$3. The source of the gap is not a better smoothness constant but the much smaller $\nabla f(x_k)$4-norm of the optimal logits under Zipf imbalance. This suggests that heavy-tailed class imbalance favors $\nabla f(x_k)$5-steepest descent even in a convex softmax head, and provides an additional theoretical route linking sign-based updates to the empirical success of Adam-like methods in language modeling (Yadav et al., 30 Nov 2025).

6. Applications, robustness, and current directions

Beyond standard deep learning optimization, SignGD has been used as an implicit regularizer and as a domain-specific descent rule. In kernel methods, “Fast Robust Kernel Regression through Sign Gradient Descent with Early Stopping” connects $\nabla f(x_k)$6-regularized kernel regression to kernel sign gradient descent (KSGD), and reports that robust kernel regression through sign gradient descent is one to two orders of magnitude faster than existing robust kernel methods on five real data sets, without compromised accuracy (Allerbo, 2023). In protein folding, “Sign Gradient Descent Algorithms for Kinetostatic Protein Folding” replaces normalized torque updates in the kinetostatic compliance method by a sign-torque update with shrinking step size and reports that, for the same initial step size $\nabla f(x_k)$7, SignGD reaches a lower free energy level in about 600 iterations, whereas the conventional method requires reducing $\nabla f(x_k)$8 to 0.001 and then about 1500 iterations to reach a comparable level (Mohammadi et al., 2023).

The method is also implicated in robustness trade-offs. “Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods” uses signGD as a memory-free proxy for Adam and RMSProp and shows on a synthetic frequency-domain model that while GD and signGD can both achieve near-zero standard risk, their adversarial risks differ because robustness to $\nabla f(x_k)$9-bounded perturbations is inversely proportional to the model’s weight norm. In the paper’s three-dimensional construction, signGD grows weight on an irrelevant frequency and thereby incurs larger adversarial risk; in deep networks, SGD-trained models are reported to have smaller Lipschitz constants and better robustness than models trained with adaptive methods (Ma et al., 2023). This does not negate the optimization advantages of sign methods, but it marks a recurring tension between optimization efficiency and perturbation robustness.

Recent work has concentrated on making sign methods more practical and more principled. “Sign-SGD is the Golden Gate between Multi-Node to Single-Node Learning” develops parameter-free single-node, stochastic, and multi-node variants that estimate effective step sizes online rather than relying on unknown problem constants (Medyakov et al., 4 Jun 2025). “Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent” derives $\hat g_k$0-based gradient noise scales for signSGD and Signum and reports that adaptive batch size strategies using non-Euclidean GNS reduce training steps by up to 66% for Signum on a 160 million parameter Llama model while matching the validation loss of constant-batch baselines (Naganuma et al., 3 Feb 2026). “StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training LLMs” addresses the non-smooth divergence pathology of classical SignSGD by injecting structural stochasticity into the sign operator; in low-precision FP8 pretraining it reports a 1.44$\hat g_k$1 to 2.14$\hat g_k$2 speedup relative to established baselines, and in 7B LLM fine-tuning it outperforms both AdamW and SignSGD on mathematical reasoning tasks (Yu et al., 16 Apr 2026).

These developments clarify the present status of SignGD. It is no longer a single algorithm but a design family centered on coordinate-wise sign information, $\hat g_k$3-geometry, and scale-free updates. Its strengths are most evident when communication, normalization, or heavy-tailed conditioning dominate; its weaknesses are most visible near minima, under naïve fixed-step discretizations, or when robustness to irrelevant directions matters more than raw optimization speed.