Does a Global Perspective Help Prune Sparse MoEs Elegantly?

Abstract: Empirical scaling laws for LLMs have encouraged the development of ever-larger LLMs, despite their growing computational and memory costs. Sparse Mixture-of-Experts (MoEs) offer a promising alternative by activating only a subset of experts per forward pass, improving efficiency without sacrificing performance. However, the large number of expert parameters still leads to substantial memory consumption. Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs. We propose GRAPE (Global Redundancy-Aware Pruning of Experts, a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy. Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance. On the three main models reported in the paper, it improves average accuracy over the strongest local baseline by 1.40% on average across pruning settings, with gains of up to 2.45%.

• The paper introduces GRAPE, a global pruning strategy that dynamically reallocates pruning budgets across MoE layers based on redundancy measures.
• It employs an entropy-based regularization and a one-shot greedy algorithm to balance expert retention and optimize performance under fixed pruning budgets.
• Empirical results on models like Mixtral and DeepSeek demonstrate that GRAPE improves accuracy by effectively exploiting inter-layer redundancy.

Global Redundancy-Aware Pruning for Sparse Mixture-of-Experts: Summary and Analysis

Introduction

This work investigates the problem of pruning Mixture-of-Experts (MoE) LLMs under memory and computational constraints. The core contribution is the introduction of GRAPE (Global Redundancy-Aware Pruning of Experts), a global pruning strategy that takes into account the heterogeneous layerwise redundancy in sparse MoEs. GRAPE dynamically reallocates pruning budgets based on cross-layer redundancy, departing from traditional methods that prune experts uniformly across all layers. The paper empirically demonstrates that this global perspective leads to consistent improvements in accuracy at fixed pruning budgets across various MoE model architectures.

Motivation: Non-Uniform Redundancy Across Layers

Contemporary pruning techniques for MoEs are predominantly agnostic to discrepancies in redundancy across layers. The paper posits that intra-layer redundancy—in terms of activation similarity—varies significantly both within and across layers of many contemporary MoE architectures, such as Mixtral and DeepSeek-MoE.

Figure 1: Cross-layer redundancy of different MoE models, including Mixtral-8x22B and Deepseek-MoE, revealing substantial variance across layers.

The authors operationalize redundancy using Central Kernel Alignment (CKA) and other similarity measures, and define normalized layer redundancy scores. Their analysis reveals that redundancy is not a monotonic function of depth; some intermediate layers are less redundant than both earlier and later layers. This observation demonstrates that naive uniform per-layer pruning either over-prunes indispensable layers or under-prunes highly redundant ones, resulting in performance degradation or suboptimal compression-performance trade-offs.

The GRAPE Pruning Strategy

GRAPE is a pruning framework that formalizes the allocation of the pruning budget globally rather than locally. It proceeds as follows:

• Expert Similarity Modeling: For each layer, an expert similarity matrix $D^l$ summarizes pairwise expert activation similarities.
• Objective Formulation: The block-diagonal matrix $A$ aggregates expert similarities across all layers. The pruning objective is to minimize the total sum of similarities between retained experts, subject to a layerwise entropy regularization.
• Entropy Constraint: To avoid degenerate solutions—such as allocating most pruning to a few highly redundant layers—GRAPE introduces a global entropy constraint. Entropy is computed over the per-layer distribution of retained experts, and a lower-bound threshold encourages the retained experts to be distributed without severe imbalance.
• One-shot Greedy Algorithm: GRAPE greedily merges the most similar experts in the most redundant un-frozen layer until the entropy threshold is reached, then freezes the layer. If all layers are frozen before the budget is exhausted, frozen sets are reset and pruning continues, which stabilizes pruning allocations and prevents catastrophic collapse for layers with low redundancy.

This method handles the redundancy skew observed in large-scale MoEs and adaptively focuses the pruning where it is least harmful, as measured by inter-expert similarity.

Empirical Evaluation

Comprehensive experiments are conducted on a range of state-of-the-art MoE models, including Mixtral-8x22B, DeepSeek-MoE-16B, Mixtral-8x7B, Qwen-MoE, and GPT-oss, with local and global baselines including router-guided, visitation-guided, and feature similarity-based methods. All methods are evaluated without post-pruning finetuning to isolate pure redundancy exploitation.

Figure 2: Results of Mixtral-8x7B and Qwen-MoE demonstrate that GRAPE yields superior task-wise retained performance across a range of downstream tasks and pruning settings.

On major benchmarks (MMLU, BoolQ, OpenBookQA, RTE), GRAPE consistently achieves the best or near-best accuracy across uniform and adaptive pruning settings. On Mixtral-8x22B, GRAPE improves average accuracy by 1.79%-2.45% over the strongest baseline at comparable pruning budgets. For both DeepSeek-MoE and GPT-oss, the performance advantage is robust (up to 1.84% on DeepSeek and 0.45% on GPT-oss), confirming the advantage of redundancy-aware budget allocation.

It is noted that in pathological cases—such as extremely skewed redundancy in DeepSeek-MoE—aggressive global pruning may collapse some layers when not regulated, emphasizing the necessity of entropy-based regularization and the inherent challenges of cross-layer budget allocation.

Implications and Future Directions

GRAPE’s global perspective on redundancy advances the state of MoE pruning beyond homogeneous heuristics and opens several research horizons:

• Metric Design: A key open question concerns the most effective redundancy and similarity metrics for driving global expert selection.
• Adaptive Entropy Regularization: As observed, entropy-based regularization is critical to prevent collapse, but setting the tolerance parameter remains sensitive and model-specific.
• Model Specialization: By promoting heterogeneous expert retention, GRAPE inherently steers the compressed model towards allocating representational capacity where it is most necessary—a principle with potential relevance for other structured sparsification regimes.
• Theoretical Analysis: Further work is necessary to theoretically characterize the impact of pruning schedule and redundancy distribution on generalization bounds and downstream robustness.

Practically, the method is highly relevant for deploying large MoE LLMs under resource constraints by achieving improved accuracy at equivalent sparsity, especially as large-scale generative models become ubiquitous in memory- and latency-sensitive applications.

Conclusion

This paper presents a principled framework for globally pruning sparse MoE models by dynamically reallocating pruning budgets in accordance with layerwise redundancy. Empirical results substantiate the superiority of global redundancy-aware pruning for both large and modest MoE LLMs, offering better retention of performance under aggressive compression. However, future research is warranted to further refine redundancy metrics, to ensure robust allocation in models with extreme redundancy heterogeneity, and to theoretically ground global pruning schedules in their representational and generalization effects.

The framework introduced here serves as a foundation for more adaptive and robust structured MoE pruning, with broad applicability and several promising directions for extension.