Optimizing Stochastic Gradient Push under Broadcast Communications

Abstract: We consider the problem of minimizing the convergence time for decentralized federated learning (DFL) in wireless networks under broadcast communications, with focus on mixing matrix design. The mixing matrix is a critical hyperparameter for DFL that simultaneously controls the convergence rate across iterations and the communication demand per iteration, both strongly influencing the convergence time. Although the problem has been studied previously, existing solutions are mostly designed for decentralized parallel stochastic gradient descent (D-PSGD), which requires the mixing matrix to be symmetric and doubly stochastic. These constraints confine the activated communication graph to undirected (i.e., bidirected) graphs, which limits design flexibility. In contrast, we consider mixing matrix design for stochastic gradient push (SGP), which allows asymmetric mixing matrices and hence directed communication graphs. By analyzing how the convergence rate of SGP depends on the mixing matrices, we extract an objective function that explicitly depends on graph-theoretic parameters of the activated communication graph, based on which we develop an efficient design algorithm with performance guarantees. Our evaluations based on real data show that the proposed solution can notably reduce the convergence time compared to the state of the art without compromising the quality of the trained model.

• The paper derives new SGP convergence bounds parameterized by graph metrics, enabling optimal mixing matrix design for faster federated learning.
• It introduces a four-stage algorithm that balances communication costs and convergence speed under realistic broadcast and interference constraints.
• Empirical results show up to 45% reduction in transmission slots without sacrificing model accuracy, validating the efficacy of the proposed approach.

Optimizing Stochastic Gradient Push for Decentralized Federated Learning under Broadcast Constraints

Motivation and Theoretical Context

Decentralized federated learning (DFL) eliminates reliance on central parameter servers, directly enabling edge devices to exchange model updates. While this paradigm facilitates robustness and load balancing, wireless edge networks introduce significant communication bottlenecks, especially in scenarios where broadcast communications dominate and interference arises.

A core challenge in DFL is the design of the mixing matrix, which determines both the convergence rate (expressed in iterations) and the communication cost per iteration. The design task is nontrivial: a denser mixing matrix incurs greater per-iteration transmission overhead, but enables faster convergence, resulting in a complex cost-convergence tradeoff.

Previous work primarily optimized mixing matrices under decentralized parallel stochastic gradient descent (D-PSGD), requiring symmetry and doubly stochasticity. These constraints enforce symmetric (bidirected) communication graphs and restrict design flexibility. However, stochastic gradient push (SGP) algorithms, leveraging the Push-Sum mechanism, allow asymmetric (column-stochastic) mixing matrices and directed graphs, thus enlarging the solution space. The paper investigates mixing matrix optimization for SGP, aiming to minimize wall-clock convergence time—defined as the total number of collision-free transmission slots—under broadcast and interference constraints.

Analytical Contributions

The paper’s central theoretical advance is the derivation of a new SGP convergence theorem where the required number of iterations to achieve $\epsilon$-convergence is explicitly parameterized by graph-theoretic metrics of the activated directed communication graph: the maximum in/out-degree, the minimum mixing weight, and the graph diameter. Specifically, for periodic mixing matrix schemes, the iteration bound is shown to scale as $O(\Delta^2 B^2 / \delta^{4\Delta B})$ with $\Delta$ the diameter, $B$ the period, and $\delta$ the minimum mixing weight. This enables a closed-form objective for mixing matrix design:

$\min_{G_a} \tau(G_a) \Delta^2 (1+D_a^+)^{4\Delta}$

where $\tau(G_a)$ is the per-iteration slot cost, $D_a^+$/$D_a^-$ are maximal out/in-degrees, and $\Delta$ is the diameter of the directed subgraph $O(\Delta^2 B^2 / \delta^{4\Delta B})$0. The per-iteration communication cost, $O(\Delta^2 B^2 / \delta^{4\Delta B})$1, is further bounded via Brooks’ theorem on graph coloring and tied to maximum conflict graph degree, leading to computable bounds suitable for algorithmic optimization.

Uniform mixing weights are shown to be conditionally optimal for minimizing convergence time, yielding $O(\Delta^2 B^2 / \delta^{4\Delta B})$2 for node $O(\Delta^2 B^2 / \delta^{4\Delta B})$3 at iteration $O(\Delta^2 B^2 / \delta^{4\Delta B})$4.

Graph Design Algorithm

The paper proposes a four-stage algorithm for communication graph construction:

1. Minimum-degree spanning tree: Build a tree to minimize node degree, ensuring connectivity with minimal communication overhead.
2. Diameter reduction via extra edges: Add $O(\Delta^2 B^2 / \delta^{4\Delta B})$5 edges linking distant nodes, reducing diameter and accelerating convergence.
3. Edge orientation for strong connectivity: Use preorder numbers via DFS to assign link directions within bridge-connected components, ensuring strong connectivity while maintaining low degree.
4. Cost-preserving augmentation: Iteratively add non-conflicting directed links that do not increase per-iteration communication cost, with the aim of further reducing diameter or improving degree distributions.

The procedure is grounded in combinatorial bounds, guaranteeing strong connectivity and quantifiable performance in terms of the derived objective.

Performance Evaluation and Empirical Results

Empirical evaluation is performed on CIFAR-10 image classification using a ResNet variant in two topologies: a random geometric graph and the Roofnet mesh. The proposed SGP-based design is compared with D-PSGD baselines, MATCHA, and state-of-the-art broadcast-aware sampling (BASS) approaches.

Key numerical outcomes:

• The proposed SGP-based design achieves an 11–21% reduction in convergence time (transmission slots) compared to BASS optimized (2604.15549).
• Comparison with baseline D-PSGD shows 38–45% reduction in total slots, demonstrating practical efficacy of combining SGP flexibility and topology-driven optimization.
• Cost-preserving link augmentation contributes significant gains in logical and physical convergence rates, shown in ablation studies.

Importantly, the empirical results confirm no compromise in final model quality (accuracy), indicating that optimization of asymmetric communication patterns via SGP can achieve faster convergence without degradation.

Practical and Theoretical Implications

The research establishes that relaxing mixing matrix symmetry constraints—moving from D-PSGD to SGP—enables activation of more flexible, strongly-connected directed communication subgraphs. In wireless networks with broadcast and interference, this flexibility demonstrably yields faster convergence under practical slot-based scheduling.

Theoretical implications include the explicit characterization of SGP convergence in terms of diameter, maximal degree, and minimum mixing weight, facilitating tractable optimization of topology and weights. The methodology also exposes the value in cost-preserving link augmentation to exploit latent parallelism.

Future developments may consider generalized base topologies beyond bidirected graphs, dynamic interference models, or other push-based decentralized algorithms (e.g., push-pull (You et al., 8 Jun 2025), push-diging [35-3-1570]) for further gains. The analytical techniques can also be extended or adapted to scenarios with dynamic bandwidth or varying slot definitions.

Conclusion

This work addresses mixing matrix optimization for decentralized federated learning under broadcast communications using stochastic gradient push, deriving explicit convergence bounds and proposing a graph-theoretic design algorithm with proven performance guarantees. Empirical evidence attests to reduced convergence times over state-of-the-art methods under realistic wireless settings, with no compromise in model accuracy. The theoretical framework and algorithmic innovations presented offer robust foundations for further advances in communication-efficient decentralized learning, particularly in the context of asymmetric, interference-aware wireless networks.