REAP the Experts: Why Pruning Prevails for One-Shot MoE compression (2510.13999v1)

Abstract: Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we demonstrate that expert pruning is a superior strategy for generative tasks. We prove that merging introduces an irreducible error by causing a "functional subspace collapse", due to the loss of the router's independent, input-dependent control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.

• The paper demonstrates that expert pruning with the REAP criterion surpasses merging by preserving router control in one-shot MoE compression.
• The paper provides theoretical proof of 'functional subspace collapse' in expert merging, introducing an irreducible error compared to selective pruning.
• Empirical evaluations across various SMoE models reveal that REAP-pruned models maintain high generative quality even at high compression ratios.

Pruning versus Merging for One-Shot MoE Compression: Theoretical and Empirical Insights

Introduction

The proliferation of sparsely-activated Mixture-of-Experts (SMoE) architectures in LLMs has enabled efficient scaling by decoupling parameter count from inference cost. However, the substantial memory overhead of SMoEs, due to their large number of experts, presents a significant barrier to deployment in resource-constrained environments. This has motivated research into expert compression, with two primary strategies: expert pruning (removal of entire experts) and expert merging (combining multiple experts into a single one). The paper "REAP the Experts: Why Pruning Prevails for One-Shot MoE compression" (2510.13999) provides a rigorous theoretical and empirical analysis of these strategies, introduces the Router-weighted Expert Activation Pruning (REAP) criterion, and demonstrates that pruning, when guided by REAP, consistently outperforms merging for generative tasks.

Theoretical Analysis: Functional Subspace Collapse in Expert Merging

The core theoretical contribution is a formal proof that expert merging introduces an irreducible error due to "functional subspace collapse." In SMoE layers, the router dynamically modulates expert outputs based on input, enabling a high-dimensional, input-dependent functional output space. When two experts are merged, the router's independent control is lost: the merged expert can only realize a static convex combination of the original experts, regardless of input. The paper quantifies the minimal $L^2$ error incurred by merging as proportional to the router's policy variability and the functional gap between experts. This error is strictly positive unless the router's policy is constant and the experts are functionally identical—conditions rarely met in practice, especially in late, highly specialized layers.

In contrast, pruning removes an expert but preserves the router's independent control over the remaining experts. The error from pruning is proportional to the pruned expert's average gate value and is insensitive to policy variability. Thus, pruning low-usage experts can yield strictly lower error than merging, particularly when the router actively mixes between distinct experts.

Empirical Evidence: Subspace Geometry and Output Diversity

Empirical analysis across multiple SMoE architectures (e.g., Qwen3-30B, ERNIE-4.5-21B) corroborates the theoretical predictions. By projecting expert activations onto principal components, the authors show that pruning preserves the geometric structure of the expert manifold, while merging causes a pronounced contraction toward the center—especially in late layers with high policy variability.

Figure 1: PCA of Qwen3-30B Layer 0 expert activations; pruning (blue) preserves manifold geometry, merging (green) collapses it.

This functional collapse under merging is not an artifact of specific architectures or hyperparameters but a fundamental property of the operation. The loss of router control leads to a qualitative change in the output space, which is particularly detrimental for generative tasks requiring diverse, input-dependent outputs.

The REAP Criterion: Router-Weighted Expert Activation Pruning

Building on these insights, the paper introduces REAP, a saliency criterion for expert pruning that considers both the router gate values and the expert activation norms. For each expert $j$, the REAP score is defined as:

$$S_j = \frac{1}{|\mathcal{X}_j|} \sum_{x \in \mathcal{X}_j} g_j(x) \cdot \|f_j(x)\|_2$$

where $\mathcal{X}_j$ is the set of tokens where expert $j$ is active. This score quantifies the average magnitude an expert contributes to the layer output when selected. Experts with the lowest $S_j$ are pruned, ensuring that the least impactful experts are removed.

REAP is robust to outlier activations and does not rely solely on usage frequency, which can be misleading in the presence of high-variance or outlier experts. The method is computationally efficient, requiring only a single pass over a calibration dataset to collect router and activation statistics.

Experimental Results: Generative and Discriminative Benchmarks

The authors conduct extensive experiments on a suite of SMoE models (20B–1T parameters) and tasks, including code generation, mathematical reasoning, creative writing, tool use, and multiple-choice (MC) question answering. Key findings include:

• On generative tasks, REAP-pruned models consistently outperform merged models, especially at high compression ratios (50%). For example, on code generation, REAP achieves a mean decrease in accuracy of only 2.8% (25% compression) and 8.0% (50% compression), while merging methods degrade by >5% and >20%, respectively.
• On MC tasks, merging and pruning perform comparably at moderate compression, but pruning is more robust at higher compression and on generative tasks.
• REAP achieves near-lossless compression on large-scale models (e.g., Qwen3-Coder-480B, Kimi-K2) for code generation and tool-calling, even after pruning 50% of experts.
• Merged models exhibit lower N-gram diversity and higher divergence from the baseline in output distributions, confirming the loss of generative capacity.

  Figure 2: GLM-4.5-Air and Qwen3-30B accuracy vs. task type; REAP offers significant improvements at 50% compression, especially for generative tasks.

  

Figure 3: N-gram diversity for Qwen3-30B at 50% compression; REAP-pruned models maintain diversity close to baseline, while merged models collapse.

Calibration and Scaling Considerations

The quality of compressed models is highly sensitive to the choice of calibration dataset. Domain-specific calibration is essential for maintaining performance in the target domain; general-purpose calibration data leads to severe degradation, especially for fine-grained models.

Figure 4: Coding accuracy vs. calibration dataset; domain-specific calibration is critical for high-quality compression.

REAP is scalable to trillion-parameter models and is compatible with quantized models without requiring re-quantization or reconciliation of block scales. Unlike merging, which necessitates clustering and parameter alignment, pruning with REAP is straightforward and efficient.

Implications and Future Directions

The findings have several important implications:

• Expert pruning, when guided by a criterion that accounts for both router and expert contributions, is the preferred strategy for one-shot SMoE compression in generative LLMs.
• Expert merging is fundamentally limited by the loss of router control and the non-locality of expert clusters, especially in late, specialized layers.
• Evaluation of compression methods must include generative benchmarks, as discriminative metrics (e.g., perplexity, MC accuracy) can be misleading proxies for real-world performance.
• The coordination between router and experts is a critical structural property that must be preserved in any compression scheme.

Theoretically, the work clarifies the limitations of parameter averaging in the context of conditional computation and highlights the importance of preserving input-dependent modulation in modular architectures. Practically, REAP enables the deployment of large SMoE models in memory-constrained settings without sacrificing generative quality.

Future research may explore hybrid compression schemes that combine pruning with quantization or low-rank adaptation, as well as methods for dynamic expert allocation and online pruning. Further investigation into the interplay between expert specialization, router policy, and downstream task performance is warranted, particularly as SMoE architectures continue to scale.

Conclusion

This work establishes that expert pruning, when guided by the REAP criterion, is superior to expert merging for one-shot SMoE compression in generative LLMs. The theoretical analysis of functional subspace collapse, supported by empirical evidence across diverse architectures and tasks, demonstrates that preserving the router's independent control over experts is essential for maintaining generative capacity. REAP provides a robust, scalable, and efficient pruning strategy, facilitating the deployment of accurate, domain-specific SMoE models in resource-constrained environments. The results underscore the necessity of comprehensive, task-relevant evaluation and the preservation of architectural coordination in model compression.