Pruning and Distilling Mixture-of-Experts into Dense Language Models

Abstract: Mixture-of-Experts (MoE) is now the dominant architecture for frontier LLMs, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ~4B-token distillation at 1.6x faster training wall-clock speed.

• The paper introduces a three-stage MoE-to-dense conversion pipeline that selects diverse experts with D-optimal activation-weighted scoring, concatenates them into a dense FFN, and recovers quality through forward-KL distillation.
• The method achieves 58.10% average accuracy on Qwen3 at matched parameter counts, outperforming dense-to-dense pruning by 6.3 percentage points while training 1.6× faster after approximately 4 billion distillation tokens.
• The strongest recipe keeps exactly the router’s top-k experts without merging, uses DO-ACP selection and standard teacher routing, and shows the greatest gains when the original MoE has a large, redundant expert pool.

Overview

This paper presents the first systematic framework for converting a trained Mixture-of-Experts (MoE) LLM into a fully dense architecture. Prior MoE compression methods reduce the number of experts but retain the MoE structure, preserving the fundamental deployment limitation that all expert parameters must reside in memory regardless of per-token activation. The proposed pipeline proceeds in three stages: (1) experts are scored by importance, the top-$K$ are selected and grouped into $k$ groups (where $k$ is the router's top-$k$ count), with within-group merging via score-weighted averaging when $K > k$; (2) the resulting group representatives are block-concatenated into a standard dense FFN whose intermediate dimension equals $k \times d_{\text{expert}}$, with static down-projection scaling approximating average routing behavior; and (3) forward-KL knowledge distillation from the MoE teacher recovers quality lost during conversion. Attention layers, embeddings, and layer norms are copied unchanged.

The design space is evaluated exhaustively: 7 scoring methods, 5 grouping strategies, 2 down-projection scalings, and $K \in \{8, 16, 32, 64, 128\}$ on Qwen3-30B-A3B, yielding 350 configurations, with cross-model validation on DeepSeek-V2-Lite and GPT-OSS-20B.

Diversity-aware expert selection

The central methodological contribution is a D-optimal expert selection criterion that jointly captures importance and mutual diversity. Given base importance $I_e$ (instantiated as conditional probability CP or its activation-weighted variant ACP) and the expert output Gram matrix $$\mathbf{G}_{ij} = \mathbb{E}_t[\langle f_i(t), f_j(t)\rangle]$$, the method forms the importance-weighted kernel $\mathcal{K}_{ij} = \sqrt{I_i I_j}\,\mathbf{G}_{ij}$ and selects the subset maximizing $k$0 via greedy Schur-complement updates in $k$1 time. The paper supplies three theoretical results: a counterexample showing independent top-$k$2 scoring can incur constant reconstruction error under expert redundancy while log-determinant selection achieves zero error; proof that the normalized objective is monotone submodular, giving the standard $k$3 greedy guarantee; and an incoherence bound showing diversity corrections matter most when mutual coherence among experts is high — consistent with the empirical observation that D-optimal gains shrink as the expert pool shrinks. A finite-sample stability theorem for the empirical DO-CP/DO-ACP estimators is provided in the appendix.

Empirically, D-optimal selection raises the effective rank of the selected expert kernel from 6.07 to 7.37 (CP) and 6.31 to 6.93 (ACP) at $k$4 on Qwen3, and DO-ACP shares at most 0.08 overlap with frequency-based selections.

Scoring dominates the design space

Across the 35 distilled scoring$k$5grouping combinations (0.3B tokens each), three tiers emerge with a 5.7 pp spread between best and worst scoring families — roughly five times the grouping spread (~1.2 pp). Frequency-based methods (SF, PP, PS), which prior work uses universally, form the bottom tier (~37%): they select nearly identical generalist-expert sets. Removing the frequency factor via CP yields +3 pp by promoting rarely-selected but confidently-routed specialists; adding output magnitude (ACP) adds another 0.5 pp; applying the D-optimal criterion adds ~2 pp more. DO-ACP achieves the best accuracy across all 35 combinations and all three models.

Grouping matters little in the regime that wins: the best $k$6 is overwhelmingly 8 or 16 (32 of 35 configurations), and at $k$7 grouping is irrelevant entirely. Notably, although larger $k$8 yields lower pre-distill perplexity (e.g., ACP$k$9OC at $k$0: PPL 2,002 vs. 6,334 at $k$1), distillation reverses this ranking — pure pruning ($k$2, no weight averaging) outperforms merging by +2.0 pp post-distill for ACP. On all three architectures, the single best configuration uses pure pruning with DO-ACP: $k$3 on Qwen3 (43.41%), $k$4 on DeepSeek-V2-Lite (42.39%), $k$5 on GPT-OSS-20B (33.71%). However, the interaction is nuanced: on DeepSeek, merging beats pure pruning for SF (+1.1 pp), CP (+2.5 pp), and ACP (+0.6 pp), suggesting frequency-based selection of redundant generalists benefits from averaging while diverse specialists stand alone.

Comparison against dense-to-dense pruning

The strongest baseline comparison is controlled at matched total parameter count (~30B teacher, ~3.4B student): dense-to-dense (D2D) pruning of Qwen3-32B following the Minitron methodology, with an architecture search over five candidates selecting width-only pruning preserving all 64 layers. After identical distillation budgets:

Configuration	Avg accuracy (%)
DO-ACP, $k$6	58.10
SF, $k$7	53.46
D2D pruning (Qwen3-32B → 3.4B)	51.84
Random FFN + teacher attn	45.44

After ~4B-token distillation, MoE-to-dense outperforms D2D by +6.3 pp and runs 1.6× faster in wall-clock time (73 s/step vs. 116 s/step on 2×B200 GPUs), because the MoE teacher activates only 3B parameters per token versus the dense teacher's full 32B. At the 0.3B-token budget the gap is even larger (+10.1 pp), and D2D sits only +0.6 pp above the random-FFN baseline, indicating dense pruning provides little structural advantage at this compression ratio. The gap is largest on MMLU (46.1% vs. 26.6%), suggesting diversity-aware expert selection particularly preserves knowledge-intensive capability. Qualitative analysis of chain-of-thought generations corroborates this: DO-ACP has both the lowest catastrophic failure rate (54.5%) and the lowest knowledge-error rate (4.2%, vs. 12.5% for D2D), with the largest subject-level gains on humanities (+24 pp over SF).

Distillation choices

Among distillation variants, forward KL on logits is clearly optimal: reverse KL loses −6.24 pp, and adding intermediate hidden-state MSE loses −1.91 pp, consistent with Minitron findings when all teacher layers are preserved. Expanded teacher routing (activating $k$8 experts during distillation, following Kim et al.) yields a modest +0.70 pp at $k$9 but degrades monotonically beyond $k$0 and costs ~2× teacher FLOPs per MoE layer, so standard routing is preferred when data is not the bottleneck.

Cross-model validation and scope conditions

Findings generalize across DeepSeek-V2-Lite (base model, shared experts requiring modified scaling since routing probabilities are not renormalized) and GPT-OSS-20B (post-trained reasoning model evaluated in completion mode, which understates native-format capability). Two trends qualify the results. First, the scoring-method gap compresses with smaller expert pools: 7.1 pp on Qwen3 (128 experts), 4.3 pp on DeepSeek (64), 1.6 pp on GPT-OSS (32), matching the incoherence-theorem prediction that diversity corrections matter less when few interchangeable candidates exist. Second, the advantage over random FFN initialization follows the same trend (+13.3, +12.1, +3.7 pp respectively), indicating effectiveness depends on expert-pool redundancy. Teacher-variant ablations confirm scoring rankings and the pure-pruning advantage hold for both base and post-trained teachers.

Limitations

The paper concedes several boundaries explicitly. When merging is used ($k$1), merge weights are tied to selection scores; decoupling them may improve such configurations. Extended training reaches only ~4B tokens, so the quality ceiling at tens of billions of tokens remains unestablished. The benefit over random initialization shrinks substantially with smaller expert pools (GPT-OSS: +3.3 pp vs. Qwen3: +10.7 pp), leaving open how the recipe transfers to architectures with very few experts or different routing normalization schemes. The function-space merge optimality result applies to output averaging; parameter-space averaging of nonlinear SwiGLU experts remains a proxy exact only in special cases.

Conclusion

This work establishes MoE-to-dense conversion as a viable and superior alternative to dense-to-dense pruning for producing small dense models from MoE lineages. The identified recipe is simple: score experts with DO-ACP (D-optimal selection over activation-weighted conditional probabilities), retain exactly the top-$k$2 experts per layer without weight averaging, concatenate into a dense FFN, and distill with forward KL using standard routing. Under matched compute and parameters, this produces students that beat dense-pruned counterparts by +6.3 pp after ~4B-token distillation at 1.6× faster training speed, with the caveat that gains scale with expert-pool size.