When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds

Abstract: Sign-based optimization algorithms, such as SignSGD and Muon, have garnered significant attention for their remarkable performance in training large foundation models. Despite this empirical success, we still lack a theoretical understanding of when and why these sign-based methods outperform vanilla SGD. The core obstacle is that under standard smoothness and finite variance conditions, SGD is known to be minimax optimal for finding stationary points measured by $\ell_2$-norms, thereby fundamentally precluding any complexity gains for sign-based methods in standard settings. To overcome this barrier, we analyze sign-based optimizers leveraging $\ell_1$-norm stationarity, $\ell_\infty$-smoothness, and a separable noise model, which can better capture the coordinate-wise nature of signed updates. Under this distinct problem geometry, we derive matched upper and lower bounds for SignSGD and explicitly characterize the problem class in which SignSGD provably dominates SGD. Specifically, we compare the \emph{upper bound of SignSGD} with the \emph{lower bound of SGD}, illustrating that SignSGD effectively reduces the complexity by a factor of $d$ under \emph{sparse noise}, where $d$ is the problem dimension. Furthermore, we elevate this framework to the matrix domain, providing an equivalent optimal lower bound for the Muon optimizer, proving that extending the sign operator to matrices preserves this optimal scaling with dimensionality. Finally, we bridge our theoretical bounds to practice, demonstrating that the theoretical superiority of SignSGD accurately predicts its faster convergence during the pretraining of a 124M parameter GPT-2 model.

• The paper establishes matching upper and lower bounds for SignSGD under ℓ∞-smoothness and ℓ₁-stationarity, demonstrating a sharp complexity advantage over SGD.
• It proves a dimensional complexity gap by showing that SGD’s bound scales with d, whereas SignSGD achieves a factor-d reduction in high-dimensional sparse noise.
• Empirical results on nanoGPT and toy models confirm that sign-based methods deliver consistent convergence where SGD stalls due to coordinate noise disparities.

Dimension-Free Characterization of SignSGD: When and Why It Outperforms SGD

Problem Geometry and Motivation

The empirical success of sign-based optimizers, particularly SignSGD, in large-scale foundation model pretraining has posed critical questions about their theoretical foundations and complexity advantages. Standard non-convex stochastic optimization theory asserts SGD as minimax optimal under $\ell_2$-norm stationarity, smoothness, and bounded variance, with iteration complexity $\Omega(\epsilon^{-4})$ [arjevani2023lower]. This precludes provable improvements under the canonical Euclidean setting. The paper proposes a rigorous geometric shift, analyzing coordinate-wise sign-based optimization in the context of $\ell_\infty$-smoothness, separable noise, and $\ell_1$-stationarity. This geometric transition both better aligns with sign-based updates and enables explicit complexity separations between SignSGD and SGD.

Theoretical Results: Tight Bounds for SignSGD

Upper and Lower Bounds for SignSGD

Under $\ell_\infty$-smoothness and separable noise, the paper proves both matching upper and lower bounds for SignSGD. The convergence rate is:

$$\mathbb{E}\left[\min_t ||\nabla f(x_t)||_1 \right] = \Theta\left( \sqrt{\frac{L_\infty \Delta}{N}} + \left(\frac{||\sigma||_1^2L_\infty \Delta}{N}\right)^{1/4} \right)$$

where $N$ is the total stochastic gradient queries, $L_\infty$ is the coordinate-wise Lipschitz constant, $\Delta$ is the initial suboptimality, and $||\sigma||_1$ is the sum of coordinate noise magnitudes.

Dimension-Dependent Lower Bound for SGD

The stochastic lower bound for SGD (under the same geometry) is:

$\Omega(\epsilon^{-4})$0

where $\Omega(\epsilon^{-4})$1 is the problem dimension and $\Omega(\epsilon^{-4})$2 is the $\Omega(\epsilon^{-4})$3 noise norm. The deterministic and stochastic terms in SGD's bound incur an explicit factor $\Omega(\epsilon^{-4})$4, setting up a strictly worse complexity scaling compared to SignSGD.

Provable Complexity Separation

The comparison reveals that SignSGD achieves a dimensional complexity reduction factor of $\Omega(\epsilon^{-4})$5 relative to SGD when noise is sparse (i.e., most coordinates have negligible variance). The paper formalizes this via the noise density function $\Omega(\epsilon^{-4})$6, showing maximal separation for $\Omega(\epsilon^{-4})$7 (highly skewed/sparse noise).

Empirical Evidence: SignSGD Outperforms SGD in High-Dimensional, Sparse Noise Settings

Toy Problem Illustrations

Experiments on deterministic and stochastic toy models confirm the theoretical separations: for imbalanced quadratic objectives dominated by one dimension, SignSGD converges steadily while SGD stagnates due to bottlenecked curvature. For problems with sparsely concentrated noise, SignSGD maintains robust convergence rates, outperforming SGD which is paralyzed by large coordinate variance disparities.

Figure 1: Deterministic case with anisotropic curvature—SignSGD retains steady convergence rates regardless of skewed coordinate curvature, while SGD is bottlenecked.

LLM Pretraining

SignSGD and SGD are benchmarked on nanoGPT (124M parameters) pretraining on the C4 dataset. SGD achieves rapid initial loss decay but plateaus prematurely, while SignSGD converges steadily and faster throughout training.

Figure 2: The training and validation loss for nanoGPT trained on C4—SignSGD demonstrates robust and consistent convergence superiority.

Tracking the noise density $\Omega(\epsilon^{-4})$8 empirically during training uncovers essential distinctions: in nanoGPT, gradient noise becomes increasingly sparse ($\Omega(\epsilon^{-4})$9 during pretraining), validating the theoretical regime where SignSGD becomes advantageous. In contrast, for standard CNN training, noise homogenizes into a dense regime ($\ell_\infty$0), aligning with the case where SGD and SignSGD perform comparably.

Figure 3: Noise density on nanoGPT—gradient noise becomes increasingly sparse, favoring sign-based descent.

Matrix Domain Extension: Muon and Optimal Scaling

The analysis generalizes to matrix-valued objectives, focusing on Muon—a matrix sign optimizer. The paper proves spectral-norm smoothness/nuclear-norm stationarity bounds for Muon:

$\ell_\infty$1

and constructs optimal lower bounds via reduction from SignSGD, establishing unimprovable, dimension-free scaling under the matrix geometry. This theoretically validates Muon's strong empirical performance in large-scale LLM training.

Implications and Future Directions

The results demonstrate that restructuring problem geometry to align with the nature of sign-based descent fundamentally changes complexity theory. Under $\ell_\infty$2-smoothness, sparse/separable noise models, and $\ell_\infty$3-stationarity, SignSGD provably escapes the minimax complexity barrier faced by SGD and achieves strict dimensional gains. This explains observed LLM training dynamics and motivates systematic use of sign-based methods in high-dimensional, noise-heterogeneous regimes. The matrix extension supports continued development and deployment of Muon-type optimizers for foundation models.

Future directions include:

• Further exploration of noise modeling, especially under heavy-tailed distributions commonly encountered in real-world deep learning.
• Extension of dimension-free complexity bounds to adaptive optimizers (Adam, AdaGrad-type) under non-Euclidean geometries.
• Analysis of sign-based variance reduction and momentum, and their impact on accelerated methods.
• Investigation of theoretical limits for other stationarity measures and norm geometries relevant to large-scale neural architectures.

Conclusion

This paper supplies the first tight, dimension-free complexity separation between SignSGD and SGD under $\ell_\infty$4-smoothness, separable stochastic noise, and $\ell_\infty$5-stationarity. Empirical evidence from LLM training confirms the theoretical predictions, identifying sign-based descent as intrinsically superior in regimes of sparse, heterogeneous gradient noise. These insights generalize to the matrix domain, establishing Muon-type optimization as both theoretically optimal and practically efficient for large-scale parameter updates. This work motivates further fundamental advances in non-Euclidean, coordinate-adaptive optimization for deep learning.