--- title: Drop-Upcycling in MoE Models url: https://www.emergentmind.com/topics/drop-upcycling type: topic --- # Drop-Upcycling in MoE Models Drop-upcycling is a framework for model parameter transfer that strategically combines pre-trained dense model initialization with partial statistical re-initialization of expert subnetworks. Originally developed in the context of mixture-of-experts (MoE) transformers, drop-upcycling addresses the convergence limitations observed in prior MoE construction methods—specifically naïve upcycling—by promoting expert specialization without sacrificing early-stage performance. Drop-upcycling and related upcycling techniques can also be applied to non-expert submodule transfers, such as the post-training adaptation of key-value (KV) compression modules in attention architectures. ## 1. Background and Motivation Traditional MoE architectures augment transformer networks by replacing each dense feed-forward (FFN) sublayer with a router and multiple expert FFNs. Only a sparse subset of experts (e.g., top-$k$ per token) is activated, resulting in increased model capacity relative to the number of active parameters and compute required per token. The upcycling approach introduced by Komatsuzaki et al. (2023) begins from a pre-trained dense model and replicates its FFN weights across all MoE experts, ensuring rapid early-stage transfer and nontrivial performance “jump start.” However, because all experts start identically, there is little incentive for progression beyond the initial local optimum and expert specialization occurs slowly. The result is a marked slowdown in the learning-curve slope compared to scratch-trained MoE models—upcycled models “coast” rather than specialize and eventually plateau at suboptimal performance. This phenomenon motivates hybrid schemes that combine knowledge transfer with statistical diversity [2502.19261]. ## 2. Drop-Upcycling Algorithm Drop-upcycling produces an MoE by initializing experts as partial mixtures of pre-trained and newly sampled weights. The procedure is as follows: **A. Expert Replication** All non-FFN parameters are copied into the MoE directly. Each expert $i$ of the $n$ experts in a layer receives a copy of the original FFN matrices: $$ W^{(i)}_{\text{gate}} = W_{\text{gate}}, \quad W^{(i)}_{\text{up}} = W_{\text{up}}, \quad W^{(i)}_{\text{down}} = W_{\text{down}} $$ The router weights $W_{\rm router}$ are randomly initialized. **B. Diversity via Partial Re-initialization** For each expert, a fraction $r$ of the weight columns (for $W_{\rm gate}$, $W_{\rm up}$) or rows (for $W_{\rm down}$) is randomly selected. For each weight type: 1. A subset $\mathcal S$ of size $\lfloor r d_f \rfloor$ is drawn from the intermediate dimension. 2. The empirical mean $\mu_{\text{type}}$ and standard deviation $\sigma_{\text{type}}$ of those coordinates are computed. 3. New weights $R_{\text{type}}$ are sampled as $R_{\text{type}} \sim \mathcal N(\mu_{\text{type}}, \sigma_{\text{type}}^2)$. 4. The updated matrix is $$ \widetilde W_{\text{type}} = I_{\mathcal S} \odot R_{\text{type}} + (1 - I_{\mathcal S}) \odot W_{\text{type}}, $$ where $I_{\mathcal S}$ is the binary mask for $\mathcal S$. Each expert thereby retains a $(1-r)$ fraction of the original weights while receiving independent random initialization in the remaining $r$ fraction. For $r=0.5$, every expert is “half old, half new.” **C. MoE Training** Standard MoE training is applied to the initialized model, using the cross-entropy loss for next-token prediction: $$ \mathcal L_{\text{NLL}} = -\sum_t \log p(x_t\,|\,x_{