DiLoCo-based Distributed Merging

• DiLoCo-based distributed merging is a framework that synchronizes periodic, locally computed parameter-space deltas using an outer optimizer like SGD with Nesterov momentum.
• It minimizes communication frequency by letting workers perform multiple local updates before merging, thereby matching or exceeding the performance of standard data-parallel approaches.
• Advanced variants include compression techniques such as Top-k sparsification and low-bit quantization, as well as asynchronous and streaming strategies to tackle network and staleness challenges.

DiLoCo-based distributed merging denotes a family of low-communication training methods derived from Distributed Low-Communication training (DiLoCo), in which multiple workers execute many local optimization steps before periodically reconciling their models through a merge in parameter space rather than through per-step gradient all-reduce. In its canonical form, each worker starts from a shared model, performs $H$ local inner-optimizer steps, forms a parameter delta relative to the last synchronized model, averages those deltas across workers, and applies an outer optimizer—typically SGD with Nesterov momentum—to obtain the next global model. The resulting design is neither standard data-parallel training nor one-shot checkpoint averaging; it is an online, repeated model-merging procedure embedded inside large-scale pretraining (Douillard et al., 2023, Charles et al., 12 Mar 2025).

1. Formal structure of the merge

DiLoCo operates on two nested timescales: an inner loop of local training and an outer loop of synchronization. With $K$ workers and synchronization interval $H$, worker $i$ performs $H$ local updates on its shard $\mathcal D_i$ before any cross-worker communication. A representative formulation writes the local update as

$$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$

After those $H$ steps, worker $i$ forms a parameter-space displacement relative to the last synchronized reference,

$\Delta_i^{(t)} = \theta^{(t-H)} - \theta_i^{t},$

and the outer merge averages these displacements,

$K$0

The global model is then updated by an outer optimizer rather than by direct parameter replacement. In the MuLoCo specification, the outer optimizer is SGD with Nesterov momentum: $K$1 This is the core merge rule of the family: average worker deltas, then transform that average through an outer momentum step (Thérien et al., 29 May 2025).

DiLoCo is closely related to FedAvg and local SGD, but it is not identical to either. The original formulation states that when $K$2 is SGD, the method is equivalent to classical Federated Averaging, and when $K$3 it reduces to one-shot model souping. In ordinary use, however, the outer optimizer is Nesterov momentum and the merge is optimizer-mediated rather than plain arithmetic averaging. This distinguishes DiLoCo-based merging from both step-synchronous gradient averaging and post-hoc checkpoint interpolation (Douillard et al., 2023).

2. What is merged, and what is not

The defining synchronized object in DiLoCo-based methods is a parameter delta or pseudo-gradient, not the raw minibatch gradient and not the full local optimizer state. In the original formulation the communicated quantity is

$K$4

while OpenDiLoCo implements the same idea by explicitly computing

$K$5

and placing that tensor into each parameter’s param.grad before all-reduce and outer optimization. The merge is therefore best understood as aggregation of model displacements in parameter space, followed by an outer optimizer step, rather than interpolation of whole checkpoints in the narrow FedAvg sense (Jaghouar et al., 2024).

This separation of roles is operationally important. Inner optimizer states remain local. The original DiLoCo study explicitly reports that synchronizing the inner Adam state did not lead to significant improvements while increasing communication cost by $K$6, so each replica keeps its own first and second moments. The outer optimizer state, by contrast, is associated with the global merge trajectory. A DiLoCo-based merge therefore has three logically distinct state types: local model parameters, local inner-optimizer state, and global outer-optimizer state. Only the parameter-space delta is the mandatory synchronization payload (Douillard et al., 2023).

A similar separation persists across later variants. MuLoCo changes the inner optimizer from AdamW to Muon on hidden layers, but preserves the DiLoCo-style synchronization protocol. SparseLoCo replaces global outer momentum with local outer error-feedback accumulators, yet still merges compressed approximations to pseudo-gradients. Streaming and asynchronous variants fragment, delay, or reweight the synchronized object, but they continue to operate on model-difference signals rather than on per-step gradients (Thérien et al., 29 May 2025, Sarfi et al., 21 Aug 2025).

3. Communication rationale and scaling behavior

The principal systems motivation for DiLoCo-based merging is that wide-area or cross-cluster training is constrained more by communication frequency and synchronization barriers than by raw floating-point throughput. Standard data-parallel training requires cross-worker communication every step; DiLoCo reduces that frequency by a factor of $K$7. In the original 8-worker C4 experiments with default $K$8, DiLoCo communicated $K$9 less often than per-step synchronization while matching or slightly outperforming the fully synchronous 8$H$0-batch baseline on a 150M-parameter transformer (Douillard et al., 2023).

Subsequent scaling-law analysis treated DiLoCo itself as a scalable merge-based optimizer. In that study, synchronization cadence was usually fixed at $H$1, though $H$2 was also examined. The reported behavior was that $H$3 performed worst, loss generally increased as $H$4 increased, and the degradation from infrequent synchronization became less pronounced as model size grew. The study also fit joint laws such as

$H$5

and emphasized that the optimal outer learning rate depends mainly on replica count $H$6 and cadence $H$7, not strongly on model size $H$8. This suggests that DiLoCo-based merging is not merely a bandwidth workaround; it has a scale-dependent optimization regime of its own (Charles et al., 12 Mar 2025).

OpenDiLoCo translated this regime into an operational framework for globally distributed training. It trained across two continents and three countries, reported $H$9 compute utilization, and observed that workers trained independently for about $i$0 minutes before communicating, with outer all-reduce taking about $i$1 seconds on average and accounting for only $i$2 of training time. It also showed that pseudo-gradients can be all-reduced in FP16 without noticeable performance hit. These results established DiLoCo-based merging as a concrete systems strategy for poor-bandwidth interconnects rather than a purely algorithmic curiosity (Jaghouar et al., 2024).

4. Compression of the merge payload

Once communication frequency is reduced, the next bottleneck is the size of the synchronized delta itself. Standard DiLoCo still all-reduces a dense model-sized update at every outer round. Later work therefore focused on compressing the worker-local delta before aggregation. A generic compressed pipeline is

$i$3

The compression operators studied include Top-$i$4 sparsification, quantization to $i$5, $i$6, or $i$7 bits, DCT Top-$i$8, and random-$i$9. Error feedback is implemented through a residual accumulator

$H$0

and all compressed runs improve when error feedback is enabled (Thérien et al., 29 May 2025).

MuLoCo is the clearest example of optimizer-dependent compressibility. It preserves the DiLoCo merge protocol but replaces AdamW in the local loop with Muon on hidden layers and AdamW on embeddings and output layers. The study reports that MuLoCo with error feedback can compress communicated deltas to $H$1 bits with next to no performance degradation, and that MuLoCo with $H$2-bit quantization and error feedback outperforms standard AdamW-DiLoCo while communicating $H$3 less and having identical memory complexity. The memory accounting is also explicit: DiLoCo with AdamW inner optimization uses $H$4 parameter memory for optimizer accumulators, DiLoCo+EF uses $H$5, MuLoCo uses $H$6, and MuLoCo+EF uses $H$7 (Thérien et al., 29 May 2025).

SparseLoCo pushes the same line of development further by combining outer error feedback, Top-$H$8 sparsification, and low-bit quantization. Its core outer rule is

$H$9

In a 512M-parameter LLaMA-style setup on DCLM with $\mathcal D_i$0, SparseLoCo at $\mathcal D_i$1 density and $\mathcal D_i$2-bit quantization reported loss $\mathcal D_i$3, pseudo-gradient size $\mathcal D_i$4 MB, and $\mathcal D_i$5 synchronizations, compared with dense DiLoCo loss $\mathcal D_i$6, pseudo-gradient size $\mathcal D_i$7 GB, and the same $\mathcal D_i$8 synchronizations. The paper further states that sparse aggregation can actually improve model performance, whereas random-$\mathcal D_i$9 is markedly worse than Top-$$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$0 (Sarfi et al., 21 Aug 2025).

5. Streaming, delay, and overlap-aware merging

A separate line of work addresses not the number of communicated bits alone, but the blocking nature of the merge barrier. Streaming DiLoCo partitions the model into fragments $$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$1 and synchronizes fragments at staggered offsets $$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$2, so that fragment $$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$3 is synchronized when $$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$4. For fragment $$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$5, worker $$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$6 computes

$$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$7

then applies the delayed synchronized fragment through

$$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$8

This changes the merge from a monolithic full-model event into a fragment-wise, staggered, and partially delayed reconciliation. Empirically, the paper reported that streaming plus overlap plus FP4 communication can reduce required bandwidth by two orders of magnitude while preserving quality close to DiLoCo and data parallelism (Douillard et al., 30 Jan 2025).

Eager Updates reinterprets the delayed outer merge more explicitly. Instead of applying a wholly stale averaged delta, it constructs a worker-specific surrogate

$$\mathcal{L} \gets f_\theta(x,y;\theta_i^{(t)}), \qquad g_i^{(n)} = \nabla_\theta \mathcal{L}, \qquad \theta_i^{(t)} \gets \mathrm{InnerOpt}(\theta_i^{(t)}, \nabla \mathcal L).$$9

The merge is thus exact-current for the local worker’s own contribution and one-outer-step stale for all remote contributions. On a 500M model, one-outer-step eager overlap reached evaluation loss $H$0 versus $H$1 for no overlap, while naive one-step-delayed merging without the eager correction degraded to $H$2 unless the outer learning rate was lowered (Kale et al., 18 Feb 2025).

CoCoDC retains the fragment-wise non-blocking merge of Streaming DiLoCo but adds explicit delay compensation and adaptive transmission. It models the desired current-time global fragment by Taylor expansion from the stale synchronized fragment and uses a corrected local rate

$H$3

followed by

$H$4

It also schedules fragment transmissions by an impact score

$H$5

and reported up to $H$6 fewer training steps than Streaming DiLoCo to reach comparable perplexity in a cross-region simulation (Zhu et al., 24 Apr 2025).

6. Asynchrony, decentralization, and unresolved limitations

Recent work generalizes DiLoCo-based distributed merging beyond synchronous or near-synchronous periodic averaging. HeLoCo studies asynchronous low-communication training under heterogeneous devices and non-IID data, where workers return stale pseudo-gradients

$H$7

with staleness $H$8. It uses outer momentum as a reference direction and applies block-wise correction before merging: aligned blocks are preserved, anti-aligned blocks are attenuated, and weakly aligned blocks are rotated toward momentum. The method reported up to $H$9 improvement over asynchronous Nesterov at fixed token budget, up to $i$0 over asynchronous momentum look-ahead, and up to $i$1 over the synchronous baseline under severe system heterogeneity (Asif et al., 29 May 2026).

Other variants relax exact synchronization more structurally. Factored Gossip DiLoCo factorizes outer synchronization into a non-blocking Mix1 on prior model states and a blocking Mix2 on current outer pseudo-gradients, turning exact global averaging into a tunable approximate-mixing process. Decoupled DiLoCo replaces lock-step rounds with a central synchronizer that merges fragment pulls from a minimum quorum of learners, uses an adaptive grace window, and weights learner contributions by processed tokens. DiLoCoX extends DiLoCo-style pseudo-gradient merging to 100B-scale decentralized training through pipeline parallelism, one-step-delayed overlap, distributed outer optimizer state, and low-rank plus quantized compression; it reported pretraining a 107B model over a $i$2 Gbps network with a claimed $i$3 speedup over vanilla AllReduce while maintaining negligible degradation in convergence (Koneputugodage et al., 22 Jun 2026, Douillard et al., 23 Apr 2026, Qi et al., 26 Jun 2025).

The design space also now includes modular and architecture-aware variants. DiPaCo adapts DiLoCo-style merging to path-composed modular models, where only workers sharing a module participate in merging that module, and the synchronized quantity becomes a module-wise outer gradient rather than a monolithic model delta. This indicates that DiLoCo-based merging is compatible not only with dense replicated models but also with partially overlapping parameterizations (Douillard et al., 2024).

The main empirical caution is that stable pretraining loss does not guarantee preservation of downstream-capable representations. In a nanochat-based study, DiLoCo achieved stable convergence and approximately $i$4 communication reduction in pretraining, yet after mid-training and SFT it underperformed standard DDP on MMLU, GSM8K, HumanEval, and ChatCORE. The paper reported, for example, mid-training ChatCORE $i$5 for DDP versus $i$6 for DiLoCo and $i$7 for a hybrid that switched from DiLoCo pretraining back to DDP; after SFT the corresponding values were $i$8, $i$9, and $\Delta_i^{(t)} = \theta^{(t-H)} - \theta_i^{t},$0. It further stated that using DiLoCo-pretrained weights and running later stages with DDP fails to recover performance, interpreting this as irreversible representation drift from asynchronous or delayed updates (Acker et al., 14 Nov 2025).

Taken together, these results define DiLoCo-based distributed merging as a broad research program rather than a single algorithm. The shared core is periodic reconciliation of locally trained model trajectories through parameter-space deltas and an outer optimizer. The main axes of variation are the synchronized object, the timing and topology of the merge, the degree of compression, the treatment of staleness, and the evaluation criterion used to judge merge quality. A plausible implication is that future progress will depend as much on representational and downstream diagnostics as on communication volume and pretraining loss alone.