Sparse Expert Synchronization (SPES)

• SPES is a decentralized framework that partitions expert parameters across nodes while maintaining a shared parameter block for efficient MoE pretraining.
• It utilizes sparse synchronization and an expert-merging warm-up strategy to lower GPU memory usage and communication costs by up to 65% compared to full-model methods.
• Empirical results demonstrate that SPES achieves competitive convergence and downstream performance while significantly reducing resource overhead in large-scale MoE training.

SParse Expert Synchronization (SPES) is a memory-efficient decentralized framework for pretraining mixture-of-experts (MoE) LLMs in settings where each node cannot train the full model locally. It was introduced in “Pretraining A LLM using Distributed GPUs: A Memory-Efficient Decentralized Paradigm” (Zhang et al., 12 Feb 2026). The central idea is to assign each node ownership of only a subset of experts while keeping the shared dense parameters trainable on every node. Nodes perform local optimization on their owned experts and the shared block, then periodically synchronize without transmitting the full parameter set. The framework also includes an expert-merging warm-up strategy intended to accelerate early convergence by exchanging knowledge among experts during the initial training phase.

1. Formal structure of the framework

SPES is defined over an MoE model with shared parameters $\psi$ and expert parameters $\Phi = \{\phi_1,\ldots,\phi_M\}$, where $M$ is the total number of experts and $N$ is the number of nodes. The experts are partitioned into disjoint owner subsets $P_1,\ldots,P_N$, and node $i$ owns experts $j \in P_i$. A block-mask matrix $U_i$ is used to zero out updates to all parameters except the shared block $\psi$ and the experts owned by node $i$ (Zhang et al., 12 Feb 2026).

At communication round $\Phi = \{\phi_1,\ldots,\phi_M\}$0, node $\Phi = \{\phi_1,\ldots,\phi_M\}$1 begins from the globally broadcast model

$\Phi = \{\phi_1,\ldots,\phi_M\}$2

then performs $\Phi = \{\phi_1,\ldots,\phi_M\}$3 local update steps over its local data $\Phi = \{\phi_1,\ldots,\phi_M\}$4:

$\Phi = \{\phi_1,\ldots,\phi_M\}$5

$\Phi = \{\phi_1,\ldots,\phi_M\}$6

with $\Phi = \{\phi_1,\ldots,\phi_M\}$7. By construction, gradients for all non-owned experts are masked out.

This parameterization makes expert ownership explicit and turns expert training into a sparse update problem. The design differs from fully replicated decentralized training in that local computation is concentrated on node-owned experts rather than the entire expert set. A plausible implication is that MoE architectures are particularly well suited to this regime because expert parameters admit a natural partition across nodes.

2. Sparse synchronization and expert merging

After $\Phi = \{\phi_1,\ldots,\phi_M\}$8 local steps, SPES performs a sparse synchronization step. The shared block is aggregated by averaging:

$\Phi = \{\phi_1,\ldots,\phi_M\}$9

Experts are not averaged. Instead, each expert is updated solely from its owner node:

$M$0

where $M$1 denotes the owner of expert $M$2 (Zhang et al., 12 Feb 2026).

The framework optionally applies an expert-merging warm-up for $M$3. Similarity between experts is computed using the cosine similarity of input weights:

$M$4

For each expert $M$5, the top-$M$6 similar peers $M$7 are selected, and the expert is updated by

$M$8

where $M$9 is a mixing coefficient that linearly decays to zero by step $N$0.

The algorithmic workflow includes server broadcast, node-local sparse updates, optional warm-up merging at synchronization time, transmission of updated shared parameters and owned experts, shared-block aggregation, owner-based expert replacement, and decay of $N$1. The key hyperparameters are the synchronization interval $N$2, the merge coefficient $N$3, the merge horizon $N$4, and the merge fan-in $N$5. Although the method is described as decentralized, the reported implementation includes a parameter server for broadcast and aggregation; this indicates that the defining sparsity property lies in expert ownership and synchronization semantics rather than in the elimination of all coordinating infrastructure.

3. Memory, communication, and convergence properties

SPES is motivated by the observation that decentralized methods based on federated optimization still require each node to train the entire model, which remains constrained by GPU memory limitations. The framework’s memory analysis distinguishes a full-model baseline from the sparse expert regime. A fully decentralized FedAvg node requires approximately

$N$6

floats, reflecting optimizer states and gradients for all parameters. In SPES, node $N$7 stores

$N$8

where $N$9 is the parameter count of locally owned experts and the $P_1,\ldots,P_N$0 term remains because the node stores read-only copies of all experts (Zhang et al., 12 Feb 2026).

Under the assumption $P_1,\ldots,P_N$1, the per-node memory scales as $P_1,\ldots,P_N$2. Communication exhibits a similar asymmetry. Each round, node $P_1,\ldots,P_N$3 uploads at most $P_1,\ldots,P_N$4, yielding uplink of approximately $P_1,\ldots,P_N$5, while the download is approximately $P_1,\ldots,P_N$6. Full-model schemes exchange $P_1,\ldots,P_N$7 in both directions.

The convergence statement is given as a sketch under standard $P_1,\ldots,P_N$8-smoothness and bounded variance assumptions. The appendix’s Theorem 1 yields

$P_1,\ldots,P_N$9

Here $i$0 and $i$1 denote variances on shared and expert gradients, $i$2 captures data heterogeneity across owners, and the merge term vanishes if $i$3 is small. The stated interpretation is that shared-block variance shrinks by $i$4 as the number of nodes increases, while expert blocks remain owner-only and are aided by merging warm-up.

4. Experimental configuration and model regimes

The reported experiments cover from-scratch pretraining and upcycling regimes. The 2B-parameter runs use $i$5 nodes, each with one NVIDIA L40S (48 GB) GPU, connected by 17 Gbps Ethernet to a parameter server with a 64-core Xeon and 720 GB RAM. The 7B-parameter runs use $i$6 nodes, each with $i$7 NVIDIA A800 GPUs connected by NVLink, with 13 Gbps Ethernet to a server with a 96-core Xeon and 1.4 TB RAM. The 9B upcycling setup uses the same 4-node A800 configuration and initializes from a 1.7B dense checkpoint (Zhang et al., 12 Feb 2026).

The model architectures are summarized in the paper’s Table 3. The 2B MoE uses 16 layers, 24 heads, hidden size 1,536, $i$8 experts, and $i$9 active experts. The 7B MoE uses 16 layers, 16 heads, hidden size 2,048, $j \in P_i$0 experts, and $j \in P_i$1 active experts. The 9B upcycled MoE uses 28 layers, 16 heads, hidden size 2,048, $j \in P_i$2 experts, and $j \in P_i$3 active experts.

The training data and hyperparameters are likewise specified.

Regime	Data mixture	Core optimization settings
2B, 7B from scratch	Ultra-FineWeb (64.2%), SlimPajama (27.2%), StarCoder (6.6%), arXiv (0.7%), OpenWebMath (0.4%), Pes2o (0.5%), Algebraic Stack (0.4%)	Qwen v1 tokenizer; AdamW $j \in P_i$4
9B upcycling	Nemotron-CC v2 (63.3%), Nemotron-Math (16.4%), code (11.9%), SFT-style (8.4%)	same tokenizer and optimizer family

The learning-rate schedules are $j \in P_i$5 for 2B, $j \in P_i$6 for 7B, and $j \in P_i$7 for 9B, each with linear warm-up over 2,000 steps. The per-node token batches are $j \in P_i$8M$j \in P_i$9 for 2B, $U_i$0M$U_i$1 for 7B, and $U_i$2M$U_i$3 for 9B. Synchronization uses $U_i$4 for 2B and 7B, while 1B ablations use $U_i$5 late. The merge configuration is $U_i$6, $U_i$7, $U_i$8 steps, with merging every 500 steps.

5. Empirical results

The reported memory and communication results directly quantify the sparse synchronization design. For the 2B model on $U_i$9 GB GPUs, FedAvg and DiLiCo require approximately 55 GB per GPU, whereas SPES uses approximately 35 GB, corresponding to a 36% reduction. For the 7B model on 4$\psi$0A800, SPES keeps per-GPU memory under 48 GB without intra-node sharding. Per communication round, SPES requires approximately 9.8 GB uplink versus 28.6 GB for a full-model scheme, a 65% reduction (Zhang et al., 12 Feb 2026).

Training throughput is compared against a centralized baseline. Centralized 7B FSDP on 8$\psi$1A800 with InfiniBand HDR 200 Gbps yields 3.79k tok/s/GPU, while SPES with 13 Gbps Ethernet achieves 3.67k tok/s/GPU. Figure 1 is reported to show that, after an initial lag, SPES converges to similar or slightly better validation scores than centralized training and DiLiCo when averaged over ARC-E, ARC-C, PIQA, SciQ, OBQA, BoolQ, SIQA, and WinoGrande.

Downstream evaluations are given in the main text and appendix. SPES-2B trained on 500B tokens matches or outperforms MobiLlama-0.8B, TinyLlama-1.1B, and OpenLM-1.1B. SPES-7B trained on 500B tokens obtains 89.9 on SciQ, 74.7 on PIQA, 44.8 on SIQA, 62.7 on BoolQ, 72.1 on ARC-E, and 43.8 on ARC-C, and is reported as being on par with MoE++, which was trained on 1T tokens. SPES-9B, trained for 400B tokens from a strong dense initialization, reaches 95.3 on SciQ, 78.9 on PIQA, 47.5 on SIQA, 77.3 on BoolQ, 81.5 on ARC-E, and 57.3 on ARC-C.

Ablation results attribute part of the observed performance to the merge strategy and synchronization schedule. Expert merging improves the average score from 50.5 to 51.3, a gain of 0.8 points. Varying $\psi$2 shows the best performance at $\psi$3, with larger $\psi$4 degrading results. Varying the number of nodes from 2 to 8 leaves performance relatively stable, with average score changing from 50.6 to 49.5. Among merge hyperparameters, $\psi$5, $\psi$6, and $\psi$7k are reported as optimal.

6. Limitations, operating regimes, and prospective extensions

The paper identifies three principal advantages of SPES: major reduction in per-node GPU memory, substantially lower communication suitable for Internet-connected heterogeneous clusters, and competitive convergence and final quality relative to centralized baselines despite sparse expert updates (Zhang et al., 12 Feb 2026). These properties make the method especially relevant in memory-constrained environments, under low-bandwidth cross-node links in the 10–20 Gbps range, and for MoE architectures with large expert counts, where expert partitioning is natural.

The reported limitations are correspondingly specific. SPES shows slightly slower initial convergence due to sparse token utilization. It requires careful tuning of the synchronization interval $\psi$8 and the merging schedule to balance divergence against communication cost. Heterogeneous data across nodes, represented in the theory by $\psi$9, can introduce bias; merging is stated to mitigate but not eliminate this effect. These observations address a likely misconception that sparsifying expert updates is cost-free: the method reduces memory and communication pressure, but it does not remove optimization trade-offs.

Several extensions are proposed. These include combining SPES with intra-node FSDP or ZeRO to move beyond 10B MoE on commodity cards, adaptive $i$0 or event-triggered synchronization based on divergence thresholds, more advanced expert mixtures such as clustering or knowledge distillation in place of simple cosine merging, application to multimodal MoE or encoder–decoder architectures, and theoretical treatment of non-IID client distributions and asynchronous updates. This suggests that SPES is best understood not as a single fixed training recipe, but as a sparse synchronization paradigm for distributed MoE pretraining under severe memory and bandwidth constraints.