Extend Shadowheart SGD to statistical heterogeneity

Extend the Shadowheart SGD framework and analysis to the setting with statistical heterogeneity across workers (non-identically distributed data), while preserving the model of arbitrary computation and communication heterogeneity and compressed, asynchronous centralized training.

Background

The paper focuses on device heterogeneity—arbitrary and heterogeneous computation and communication times—under centralized asynchronous training with compression. This excludes differences in data distributions across workers, which is central in federated learning.

The authors explicitly defer handling statistical heterogeneity, identifying it as a next step to broaden the applicability of their results to more realistic federated learning scenarios.

References

Due to our in-depth focus on device heterogeneity and the challenges that need to be overcome, we do not consider statistical heterogeneity, and leave an extension to this setup to future work.

— Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity  (2402.04785 - Tyurin et al., 2024) in Section 1 (Introduction)

A direct empirical comparison against the above methods adapted to the GDT setting is left as future work.

— GeoMesh: Workload-Balanced and Sign-Compressed Geo-Distributed LLM Training  (2609.18388 - Shin et al., 16 Sep 2026) in Appendix, Section “Related Work,” subsection “Communication compression”