Decoupled DiLoCo for Resilient Distributed Pre-training

Abstract: Modern large-scale LLM pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across accelerators. Due to this coupling, transient slowdowns, hardware failures, and synchronization overhead stall the entire computation, wasting significant compute time at scale. While recent distributed methods like DiLoCo reduced communication bandwidth, they remained fundamentally synchronous and vulnerable to these system stalls. To address this, we introduce Decoupled DiLoCo, an evolution of the DiLoCo framework designed to break the lock-step synchronization barrier and go beyond SPMD to maximize training goodput. Decoupled DiLoCo partitions compute across multiple independent learners'' that execute local inner optimization steps. These learners asynchronously communicate parameter fragments to a central synchronizer, which circumvents failed or straggling learners by aggregating updates using a minimum quorum, an adaptive grace window, and dynamic token-weighted merging. Inspired bychaos engineering'', we achieve significantly improved training efficiency in failure-prone environments with millions of simulated chips with strictly zero global downtime, while maintaining competitive model performance across text and vision tasks, for both dense and mixture-of-expert architectures.

• The paper presents a decoupled, asynchronous training framework that prioritizes availability and fault tolerance by decoupling learners from global synchronization.
• It employs a minimum quorum, adaptive grace window, and token-weighted averaging to efficiently reconcile model fragments across independent learners.
• Empirical tests show 88% goodput under aggressive hardware failures with no degradation in model quality, validating its resilience and scalability.

Decoupled DiLoCo: A Highly Resilient, Asynchronous Framework for Large-Scale Distributed Pre-training

Motivation and Theoretical Foundations

LLM pre-training at scale is systemically hampered by the SPMD (Single Program Multiple Data) training paradigm, which enforces strict global synchronization in every compute step. This creates acute sensitivity to hardware failures, transient slowdowns, and stragglers—as even localized issues can stall the full training run and waste significant compute resources. Approaches like elastic data-parallel training increase reconfigurability, but fundamentally retain whole-system synchronization, leading to persistent downtime and degraded cluster-level goodput under large-scale, failure-prone deployments.

Decoupled DiLoCo introduces a complete departure from this paradigm by prioritizing availability and partition tolerance (in analogy to the CAP theorem) over strict consistency. By enabling independent progress among decoupled learners, the system confines the blast radius of hardware faults, maximally exploiting the available compute even under continuous fault conditions, and reduces the system-wide synchronization footprint.

Algorithmic Design: Asynchronous Learner/Syncer Decoupling

The core algorithmic contribution involves decomposing the training cluster into $M$ independent, fully decoupled learners, each operating on disjoint compute and data shards. Each learner executes local inner optimization (e.g., AdamW) without coordination with other learners. Model parameters are partitioned into $P$ fragments, which are periodically synchronized with a central syncer node using an outer optimization protocol. Crucially, the syncer aggregates and reconciles parameter updates asynchronously via minimum-quorum and grace-window logic, and does not require all learners to participate at each aggregation.

Figure 1: Example of two asynchronous learners and three model fragments in Decoupled DiLoCo; stalling of one learner does not block continued training of the other, and missed updates are replayed on recovery.

This protocol is parameterized by three core choices:

• Minimum Quorum ($K$): Only $K\leq M$ learners are needed to proceed with aggregation, allowing laggards or failed nodes to be excluded transiently.
• Adaptive Grace Window ($\xi_\text{grace}$): Rather than synchronizing immediately at quorum, the syncer may wait within a time window slack to maximize sample efficiency by incorporating more updates, without stalling progress.
• Token-Weighted Averaging: Each learner’s contribution is weighted by the number of tokens processed since its last synchronization, mitigating skew from fast vs. slow learners.

  Figure 2: Cluster-level fault isolation: with decoupling, failures in one slice only affect that learner, other slices continue unimpeded.

This design can readily incorporate hardware heterogeneity, supports dynamic resizing for scavenging opportunistic compute, and carries true to the “availability and partition tolerance” axioms.

System Architecture

Decoupled DiLoCo implements the above protocol in a bifurcated architecture: learner workers, each with partitioned accelerator resources, and a syncer, sharded across CPU-only resources for resilience. Learners communicate only with the syncer and not each other. Distributed step progress is tracked using vector clocks, supporting robust deterministic replay, distributed checkpointing (via Chandy-Lamport snapshots), and recovery of absent/failing learners with bounded model state staleness.

Figure 4: System overview—independent learners perform local training, send parameter fragments over the datacenter network to CPU-based syncers, which aggregate and rebroadcast updates asynchronously.

A critical feature is the ability to upsize and downsize learner counts without global blocking or model copy transmission overhead, in contrast to data-parallel upsize which always blocks for synchronization.

Empirical Validation: Fault Tolerance, Goodput, and Zero Performance Degradation

Robustness and practical utility are demonstrated through large-scale simulated fault conditions, ‘chaos engineering’ experiments, and hardware-heterogeneous deployments. Under extremely aggressive simulated rates (e.g., 1.2 million chips, $MTBI_\text{chip}=1$ year), Decoupled DiLoCo sustains 88% goodput compared to 58% for elastic data parallel. Simultaneously, there is no statistically significant degradation in ML model quality on both text and vision tasks—even under continuous and aggressive hardware interruptions.

Figure 3: Under heavy hardware failures, Decoupled DiLoCo achieves high goodput and matches elastic data-parallel in downstream ML performance, with drastically reduced bandwidth requirements.

Heterogeneity experiments show full compute utilization without blocking, regardless of learner step timing variability; with adaptive grace windows and quorum of $K=1$, the protocol is maximally available and sidesteps the slowest-chip bottleneck.

Figure 5: XLA operation traces for $K=8$, $K=1$, and $K=1$ with adaptive grace; Decoupled DiLoCo exploits all available compute despite heterogeneity in hardware speed.

Scavenging experiments validate seamless addition and removal of learning resources with no penalty, enabling time-efficient opportunistic acceleration of pre-training.

Scaling and Algorithmic Ablations

Experiments with model scales from 2B, 5B, to 9B—both dense and MoE—demonstrate that Decoupled DiLoCo consistently matches centralized data-parallel baselines in downstream evaluation, with fully predictable scaling behavior. Ablations show that balanced tensor fragmentation minimizes bandwidth peaks with no adverse effect on model quality, and a bespoke Radial-Directional Averaging (RDA) merge operation for fragments yields optimal scaling at large learner counts.

Figure 6: Decoupled DiLoCo matches data-parallel model quality across model scales up to 9B parameters and 141B tokens, on diverse text and vision tasks.

Practical Implications and Theoretical Impact

By decoupling learners and relaxing strict parameter consistency, Decoupled DiLoCo essentially enables features that remain beyond reach for classic SPMD regimes:

• Near-perfect system uptime and maximal exploitation of hardware budgets.
• Robustness to hardware faults, network instability, and asynchronous compute environments.
• Native support for dynamic, geo-distributed, and heterogeneous cluster resources, enabling cost-effective and failure-resilient pre-training at extreme scale.
• Significantly reduced bandwidth pressure, supporting opportunistic and scavenged deployments.

The architecture directly anticipates and is tailored to imminent AI infrastructure trends: highly distributed, unreliable, and geo-diverse compute (including low-bandwidth or space-based clusters), as well as increasing hardware heterogeneity.

Conclusion

Decoupled DiLoCo provides a principled, empirically validated, and well-engineered answer to large-scale distributed pre-training under real-world operating conditions. It fundamentally challenges the need for global parameter synchronization as the default, instead placing availability, partition tolerance, and compute efficiency first. There is no observable loss in downstream model performance, even at the highest scales and under adversarial hardware failures. The system architecture and algorithmic innovations set a new milestone in resilient, flexible, and bandwidth-efficient foundation model pre-training—pointing the way to robust AI cluster operation in the presence of ubiquitous failures and heterogeneous resource pools.

Implications for future AI development include the obsolescence of cluster-blocking synchronization and a shift toward asynchronous, fault-tolerant, and scavenger-ready distributed training protocols, unlocking new efficiencies and scale in model pre-training.

---

Reference:

"Decoupled DiLoCo for Resilient Distributed Pre-training" (2604.21428)