Sparse Communication for Distributed Gradient Descent

Abstract: We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates. Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices. This method can be combined with quantization to further improve the compression. We explore different configurations and apply them to neural machine translation and MNIST image classification tasks. Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task. Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.

• The paper introduces Gradient Dropping that removes the smallest 99% of gradient updates to reduce communication costs during distributed SGD.
• It employs residual accumulation to ensure that omitted gradient values do not hinder model convergence or performance.
• Experimental results show speed-ups of up to 49% for MNIST and 22% for neural machine translation tasks, confirming the method's efficiency.

Sparse Communication for Distributed Gradient Descent

The paper explores enhancements to distributed stochastic gradient descent (SGD) focusing on sparse communication to improve computational efficiency. Authors Alham Fikri Aji and Kenneth Heafield propose exchanging sparse updates instead of dense ones within the context of gradient updates, exploiting the inherent positive skewness of these updates. By mapping the smallest 99% of gradient updates to zero, the study achieves a significant reduction in communication bandwidth requirements.

Methodology

The authors introduce a technique termed Gradient Dropping, which involves removing the R% smallest gradients based on their absolute values. This method simplifies previous efforts by using a single threshold for both positive and negative gradients. To address concerns about convergence when smaller gradients are ignored, the method incorporates residuals to accumulate these untransferred updates. The Gradient Dropping process is detailed algorithmically, showcasing its integration into distributed SGD with parameter sharding.

Experimental Framework

Experiments were conducted on MNIST image classification and neural machine translation (NMT) tasks. Various dropping ratios (90%, 99%, and 99.9%) were tested to assess their impact on training performance, convergence rate, and model accuracy. Notably, the study identifies that the MNIST dataset is more tolerant to gradient reduction compared to the complex demands of NMT tasks.

Results

The results indicate substantial training speed-ups: up to 49% for MNIST and 22% for NMT tasks, without SACRIFICING significant accuracy or BLEU score losses. The study finds that even a 99% drop rate, which reduces data exchange by 50-fold, still permits effective model learning. Further, quantization techniques such as 1-bit and 2-bit quantizations were explored, revealing that NMT tasks—due to their gradient characteristics—did not benefit from 1-bit quantization as previously assumed.

Implications and Future Work

From the theoretical standpoint, the approach challenges the orthodoxy of gradient communication by demonstrating that large proportions of gradient information can be discarded without degrading model performance. In practical terms, this allows for substantial reductions in communication overhead in distributed systems, making this approach particularly suitable for multi-node environments where communication costs are more pronounced.

Looking ahead, the authors advocate for testing the proposed sparse communication methods in environments with more costly communication to reaffirm and extend their findings. Additionally, they recognize the potential application of gradient dropping in broader neural network contexts beyond the MNIST classification.

Conclusion

This research heralds a significant advancement in distributed machine learning, particularly SGD, through sparse communication methods tailored to exploit gradient skewness. The findings provide a foundation for further research in scalable and efficient model training, with promising implications for complex systems and large-scale datasets.