Extend hierarchical routing to backward reduce-scatter

Extend the FlashMask-3 hierarchical routing scheme from forward all-gather communication to backward reduce-scatter communication.

Background

FlashMask-3 uses topology-aware hierarchical routing for forward all-gather communication, exploiting intra-node NVLink and inter-node RoCE connectivity to reduce cross-node traffic. The backward pass instead uses reduce-scatter communication, and the paper explicitly identifies extending hierarchical routing to this operation as unfinished future work. Solving this problem would provide a more symmetric topology-aware communication strategy across both directions of distributed attention.

References

Hierarchical routing is currently applied only to the all-gather operation; extending it to the backward reduce-scatter is left as future work.

— FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention  (2610.17901 - Wang et al., 13 Oct 2026) in Appendix, Section "Limitations and Broader Impact" (sentence beginning "Hierarchical routing is currently applied...")