Hierarchy of nonlinearities for decentralized optimization

Determine whether some gradient nonlinearities are better suited to decentralized optimization than others and establish a hierarchy among different nonlinearities across different optimization regimes.

Background

The paper establishes that gradient clipping enables decentralized SGD to attain order-optimal convergence rates and linear speed-up under heavy-tailed noise, whereas prior work shows that gradient normalization can fail to converge even with noiseless gradients. These contrasting behaviors motivate a broader investigation of how different nonlinear transformations interact with decentralized communication and optimization dynamics.

The unresolved problem is to compare nonlinearities systematically across regimes and determine whether their suitability can be ordered or characterized theoretically.

References

Our results also highlighted an important distinction between clipping and normalization in decentralized settings, raising broader questions about the effects of nonlinearities in decentralized optimization. Are some nonlinearities better suited to decentralized settings, and if so, can we establish a hierarchy across different regimes?

— Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping  (2610.10527 - Armacki et al., 7 Oct 2026) in Section Conclusion