Determine the cause of variability in distributed AllReduce performance

Determine whether rank desynchronization, together with the last-arriving-rank throughput measurement methodology, causes the large transient variability observed in AllReduce communication and throughput for pure data-parallel training across Alps, LUMI, DGX A100, JUPITER, and Leonardo.

Background

The paper observes substantial runtime fluctuations in AllReduce operations beyond a single node, particularly for pure data parallelism using many small, non-blocking AllReduce operations. Because the throughput metric is determined by the slowest rank at each step, transient delays in one rank can produce large apparent slowdowns.

The authors conjecture that rank desynchronization and the measurement procedure jointly contribute to this behavior, but do not establish the causal mechanism. Resolving this issue is important for distinguishing genuine network contention from measurement sensitivity and collective-library variability.

References

We conjecture this stems partly from rank desynchronization, compounded by our throughput methodology, which uses the last-arriving rank at each step and so is particularly sensitive to any single rank's transient slowdown.

— Characterizing the Scalability and Performance of Large-Scale AI Training Under Multi-Tenancy  (2609.00817 - Raffi et al., 1 Sep 2026) in Section 5, subsection “Alps” (Section 5.3)