Unifying Local Communications and Local Updates for LLM Pretraining

Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives. This can become a bottleneck when bandwidth or worker speed is heterogeneous. We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to the recently popular "outer optimizer" to allow a practical gossip-based training framework that is compatible with adaptive optimizers, allows for local optimizer steps, and can utilize sparse randomized peer communication. Empirically, on a number of standard LLM training tasks, we demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies and, unlike existing decentralized methods in the LLM setting, it allows to obtain performance competitive with DiLoCo when utilizing multiple local steps. In the heterogeneous bandwidth setting we demonstrate the advantage of GASLoC showing that it can significantly outperform DiLoCo.

• The paper introduces GASLoC, a novel decentralized algorithm that unifies local computation and sparse peer-to-peer communication for LLM pretraining.
• It demonstrates that outer momentum reduces communication complexity from O(χ) to O(√χ), ensuring efficient convergence even under heterogeneous bandwidth constraints.
• Empirical evaluations on LLaMA-3 style models show that GASLoC achieves comparable validation loss to traditional methods while enhancing fault tolerance and scalability.

Unifying Local Communications and Local Updates for LLM Pretraining

Motivation and Context

Training LLMs in distributed setups faces communication bottlenecks, especially over heterogeneous networks or in the presence of stragglers. Conventional approaches rely on synchronous All-Reduce collectives, which maintain identical model states and strict global synchronization, but scale poorly with bandwidth limitations or node variability. While local update and communication-reducing schemes (e.g., Local SGD, DiLoCo) offer improvements, they ultimately still depend on globally synchronous communication steps. Decentralized optimization and gossip-based communication have long been studied in convex optimization, but their integration with modern LLM pretraining—characterized by homogeneous data and adaptive optimizers—remains underdeveloped and not well validated empirically.

GASLoC: Algorithmic Framework and Theoretical Insights

The paper proposes GASLoC (Generalized Accelerated Sparse Communication Local Computation), a decentralized training framework that unifies local computation (multiple local optimizer steps) and local communication (sparse peer-to-peer, time-varying randomized exchanges) with an outer optimization/acceleration loop. The formulation generalizes DiLoCo: it recovers DiLoCo in the complete-graph (All-Reduce) case and interpolates continuously to highly sparse, decentralized communication patterns.

The central mechanism involves each worker performing an adaptive number of local steps ($H_i$), followed by a communication step over a randomly sampled local neighborhood, combined via a communication-accelerated outer optimizer. The model states are mixed either with 1-peer, 2-peer, or complete-graph averaging, determined by a randomly sampled Laplacian. Theoretical analysis extends unified convergence proofs from earlier decentralized-SGD frameworks, proving that with appropriate hyperparameter tuning, GASLoC preserves optimization efficiency and scales consensus error favorably: introducing outer momentum reduces the communication complexity from scaling with $\chi$ (the spectral gap of the communication graph) down to $\sqrt{\chi}$.

Figure 1: Each training round, a different sparse set of peer-to-peer exchanges propagates information without global synchronization, as in GASLoC-1-Peer.

The outer optimizer acts not only as a classical parameter server or communication-accelerator but also as a bridge between standard federated and modern LLM distributed optimization theory.

Robustness, Adaptation, and Communication-Efficient Scheduling

GASLoC enables per-worker adaptation of local step counts ($H_i$), a critical property in the heterogeneous bandwidth regime. Because global collectives are eliminated, nodes with slower links can reduce their $H_i$ without blocking faster nodes, transparently improving effective hardware utilization, idle time, and straggler resilience.

Figure 2: GASLoC allows bandwidth-limited nodes to proceed at reduced local step counts, reducing idle time for faster workers compared to synchronous collectives (DiLoCo).

This is contrasted against DiLoCo and DDP: under straggler conditions, GASLoC’s wall-clock performance is substantially improved because slow nodes' effect is localized.

Empirical Evaluation: Homogeneous and Heterogeneous Regimes

Experiments benchmark GASLoC against DDP, DiLoCo, DAdam, and Local DAdam, pretraining LLaMA-3-style transformer models on FineWeb in both single-step ($H=1$) and multi-step ($H=30$) settings, at model scales up to 551M parameters and up to 32 workers.

Single-Step ($H=1$) Regime

GASLoC-2-Peer consistently outperforms DAdam even on the complete-graph topology, while using significantly sparser communication. For example, with 16 replicas on a 551M-parameter model, DDP achieves a validation loss of 2.64, while GASLoC-2-Peer matches this result despite each worker communicating with only two peers per round.

Multi-Step ($H>1$) Regime

GASLoC achieves validation loss comparable to DiLoCo for $H=30$ across all tested topologies. Sparse communication (1/2-Peer) retains robustness and performance while enhancing efficiency and fault tolerance. GASLoC outperforms Local DAdam adaptations, which degrade notably as $\chi$0 increases.

Heterogeneous Bandwidth

Simulations introduce a single straggler node with bandwidth constraints (10–20% that of other nodes). GASLoC drastically improves time-to-validation-loss relative to DiLoCo, which is bottlenecked by global synchronization. The per-worker adaptation in GASLoC aligns synchronization latency with actual per-node compute and communication, delivering improved scaling in realistic infrastructure.

Figure 3: GASLoC maintains efficient progress when a worker is severely bandwidth-constrained ($\chi$1 capacity), in contrast to collective synchronization methods.

Figure 4: Compute utilization estimates for a $\chi$2B-parameter model show that GASLoC achieves higher utilization under bandwidth-constrained regimes compared to All-Reduce based methods.

Communication Topology and Spectral Properties

The paper establishes that randomized peer communication (both 1-peer and 2-peer schemes) achieves an expected spectral gap scaling as $\chi$3, independent of $\chi$4, sharply improving mixing properties relative to deterministic sparse topologies (e.g., cycles or exponential rings). 2-Peer mixing, with lower variance, empirically outperforms 1-Peer, further justifying the randomized approach.

Figure 5: Illustration of GASLoC's time-varying 1-Peer communication topology.

Figure 6: Time-varying 2-Peer graph, with improved average spectral gap properties.

Figure 7: Communication pattern in GASLoC-2-Peer, supporting stable performance in multi-step settings.

Hyperparameter and Optimizer Sensitivity

GASLoC's behavior is stable across optimizer hyperparameter sweeps. Momentum-based outer optimizers (especially Nesterov) consistently outperform plain SGD, both in decentralized sparse and complete-graph regimes. Sparse communication does not increase the difficulty of hyperparameter tuning relative to global collectives, and practical performance is robust to a wide range of choices.

Practical and Theoretical Implications

Practically, GASLoC addresses the challenge of scaling LLM pretraining on heterogeneous clusters, such as across data centers with non-uniform interconnects, cloud providers, or non-dedicated hardware. It directly enables higher hardware utilization in bandwidth-constrained environments, mitigates the effect of stragglers, and provides fault tolerance through localized dependencies rather than full collectives.

Theoretically, GASLoC situates the accelerated consensus mechanism of the outer optimizer as central to reducing communication complexity, linking decentralized gossip theory with modern adaptive/federated and DiLoCo-style optimization. The strong empirical demonstration on LLMs underlines the value of momentum modulation in both consensus and optimization error dynamics.

Conclusion

GASLoC demonstrates that decentralized, adaptive, and communication-accelerated methods can replace global collectives in LLM pretraining without sacrificing convergence or validation loss, even under sparse and randomized peer-to-peer communication. This unification connects classical decentralized optimization theory to practical, large-scale LLM training and shows robust scaling under hardware heterogeneity. Future work may involve further exploration of asynchronous protocols, optimized edge sampling, adaptation to non-homogeneous data settings, and scaling to even larger models and hardware platforms. The approach opens the possibility of efficient, robust distributed training across diverse infrastructure and network regimes, with significant implications for democratizing LLM development at scale.