Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy

Abstract: SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.

• The paper demonstrates that tuning SignSGD in small-batch regimes yields an O(1/√K) convergence rate under unimodal symmetric noise.
• It introduces Gaussian pre-sign dithering which recovers sub-quantum gradient information, outperforming both Adam and non-dithered methods on CIFAR tasks.
• A SWATS-inspired hybrid switch from SignSGD-M to SGD mitigates stagnation and calibrates learning rates, enhancing final accuracy on deep networks.

Enhancing SignSGD: Small-Batch Analysis, Dithering, and Hybrid Switching

Overview

The paper "Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy" (2604.25550) systematically addresses key limitations of SignSGD optimizers in the context of deep learning. The analysis focuses on three principal contributions: a finite-sample convergence result for SignSGD under small-batch, unimodal symmetric noise; an empirical and theoretical investigation of Gaussian pre-sign dithering; and a SWATS-inspired projection-based switching strategy from SignSGD-M to SGD. These contributions are contextualized with controlled single-worker experiments using ResNet architectures on CIFAR tasks to isolate pure optimizer-side effects from communication considerations.

Small-Batch Convergence Analysis for SignSGD

The study extends the theoretical understanding of SignSGD by providing a convergence rate for constant mini-batch regimes, moving beyond the large-batch results established in prior work. The analysis employs a signal-to-noise-ratio-weighted (SNR-weighted) stationarity metric $\Phi_k$ defined as

$\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$

where $S_{k,i}$ denotes the instantaneous signal-to-noise ratio of the $i$-th coordinate. This measure interpolates between $\ell_1$-gradient and quadratic behavior, more faithfully reflecting the informativeness retained by the sign bit in the presence of gradient noise.

The main result demonstrates that, under unimodal symmetric gradient noise and coordinate-wise smoothness, the SNR-weighted stationarity decays at the predicted $\mathcal{O}(1/\sqrt{K})$ rate across $K$ iterations with fixed batch size. However, the magnitude metric $\|g_k\|_1$ plateaus to a noise-determined floor $\mathcal{O}(1/\sqrt{n})$ absent in the large-batch regime, capturing the inherent accuracy ceiling of aggressively quantized sign-based methods at low batch sizes.

This analysis tightens the understanding of the quantization-statistics tradeoff and provides a refined diagnostic for optimizer efficacy under limited sample regimes.

Pre-Quantization Dithering: Annealed Gaussian Noise

A central contribution is the insight that Gaussian dithering before the sign operator can probabilistically encode sub-quantum-magnitude information lost in standard SignSGD-M updates. The method introduces an annealed pre-sign noise term

$m_{k+1} \gets \beta m_k + (1-\beta)\tilde{g}_k,$

$\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$0

where the annealing schedule follows $\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$1, gradually reducing exploration as optimization progresses. This process mimics classical dithering in quantization, decorrelating quantization error and restoring expectation-proportional updates when $\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$2.

On CIFAR-100, annealed pre-sign dithering consistently outperforms both Adam and non-dithered SignSGD-M, narrowing the generalization gap relative to full-precision SGD as shown below.

Figure 1: Test accuracy on CIFAR-100 (ResNet-18) for SignSGD-M with pre- and post-sign Gaussian dithering, against SGD and Adam.

Larger dither variances beyond the optimal regime degrade accuracy, while post-sign noise (perturbing the already-normalized update) both fails to improve and destabilizes training at larger strengths. This empirically confirms that only input-side dithering can recover lost gradient information in the sign transformation.

Hybrid Switching: Momentum-Aware Projection to SGD

To address the late-stage stagnation and magnitude mismatch that occur when naively switching from SignSGD-based methods to SGD, the paper adapts the SWATS projection heuristic. At the switching point, the SGD step size $\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$3 is chosen such that

$\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$4

yielding

$\Phi_k = \sum_{i=1}^d |g_{k,i}| \min(1, S_{k,i})$5

and subsequently applying SGD with an exponential moving average (EMA) of this projected rate.

Figure 2: The hybrid method transitions at epoch 25 from SignSGD-M to SGD with projection-calibrated learning rate on CIFAR-10.

On CIFAR-10, this hybrid strategy leverages the rapid initial progress of SignSGD-M, then surpasses both SGD and SignSGD-M in final accuracy due to the transition, as shown:

Figure 3: On CIFAR-100, hybrid switching mitigates late-stage stagnation of SignSGD-M, approaching but not matching tuned SGD.

On CIFAR-100, the hybrid approach raises final accuracy substantially above vanilla SignSGD-M (by approximately 3 percentage points), but does not fully reach the performance of the SGD baseline. These results support the conclusion that hybridization is particularly beneficial when there is a pronounced stagnation regime, but that the late-stage generalization dynamics of SGD are only partially approximated by such a switch.

Practical and Theoretical Implications

The findings clarify that SignSGD-based quantization incurs irreducible statistical error at small batch sizes, which cannot be addressed purely through scale or time averaging. Annealed pre-sign dithering offers a simple, communication-invariant modification that significantly restores accuracy, with minimal algorithmic complexity and clear connections to quantization error mitigation in classical signal processing.

The hybrid projection switch is a pragmatic step towards robust optimizer scheduling in settings requiring aggressive compression early and high-precision updates late in training. Its learnable step calibration circumvents common learning-rate tuning pitfalls associated with abrupt transitions between fundamentally distinct step geometries.

These extensions collectively improve the operational viability of sign-based optimizers for modern deep networks, especially as compute and communication budgets become increasingly stringent.

Future Directions

A natural progression is to validate these optimizer augmentations in true distributed, multi-worker contexts where communication bandwidth is the primary bottleneck and majority-vote aggregation is used. Further, combining input-side dithering with error-feedback accumulation or adaptive-triggered switching (e.g., using the stability of EMA calibration) are promising directions to robustify practical adoption. The effect of these modifications on implicit regularization and sharpness of the converged minima also warrants investigation, given persistent gaps on more difficult tasks.

Conclusion

This work situates SignSGD within a rigorous quantization-theoretic framework and demonstrates that augmenting with pre-sign dithering and momentum-aware projection-based switching recaptures substantial generalization lost to 1-bit updates. The proposed methods achieve strong numerical results, with pre-sign dithering surpassing Adam on CIFAR-100 and hybrid switching surpassing both parent methods on CIFAR-10. These insights guide both algorithm and system-level optimizer choices for deep learning under constrained memory and communication regimes, and open avenues for informed, theoretically-grounded optimizer hybridization.