Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

Abstract: Sparsely activated Mixture-of-Experts (MoE) LLMs contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose \textbf{Generic TB-Coverage}, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25\%, 50\%, and 75\% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25\% and 50\% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.

• The paper introduces a novel coverage-aware pruning approach for sparse MoE LMs, ensuring balanced retention of experts across diverse calibration corpora.
• It employs a round-robin algorithm to select top experts from independent per-corpus profiles, mitigating the bias of scalar ranking metrics.
• Empirical evaluations show superior zero-shot accuracy and reduced perplexity, especially under aggressive pruning regimes, compared to standard methods.

Generic Expert Coverage in Sparse MoE LMs: An Analysis of Generic TB-Coverage for Pruning

Motivation and Failure Modes in Scalar Expert Ranking

Sparse Mixture-of-Experts (MoE) LLMs achieve high parameter efficiency by activating only a subset of experts per input token. While this provides computational savings and performance gains, the resulting model contains structured redundancy—many experts can be pruned without substantial quality loss if the correct subset is selected. Crucially, selecting this subset without downstream calibration data is non-trivial due to the risk of selective retention driven by dominant behavior in the calibration corpus.

Standard expert-pruning approaches typically employ a single aggregated scalar criterion: routing frequency, output magnitude, or combinations (e.g., REAP-style metrics). However, when calibration data comprises heterogeneous sources (such as encyclopedic and web text), scalar aggregation overweights experts favored by dominant or more frequent patterns, potentially leading to under-representation of less prevalent but critical behaviors. This scalar approach risks discarding experts necessary for broader generalization, particularly in downstream-free settings—where task-specific calibration is prohibited to prevent task leakage.

To address this, the paper introduces a coverage-aware pruning paradigm based on the insight that preserving high-utility experts across multiple generic calibration corpora provides a more robust prior. By profiling and protecting experts that are uniquely valuable to each corpus, the pruning method prevents the over-concentration of the retained set on a subset of language behaviors, mitigating emergent failure modes of monolithic ranking.

The Generic TB-Coverage Methodology

Generic TB-Coverage executes a principled, coverage-aware expert retention strategy. For each MoE layer, the method:

1. Independent Per-Corpus Expert Profiling: For each calibration corpus (WikiText2 and C4), a REAP-style utility score is computed for every expert. This is the expected router probability times the output norm, averaged over all tokens where the expert is routed—essentially quantifying both selection frequency and computational contribution per corpus.
2. Round-Robin Coverage Selection: Rather than ranking experts by a mean or maximum score, a round-robin algorithm alternately selects top experts from each per-corpus ranking into a protection set, up to a predetermined coverage budget $B$ (with $B \leq K$, where $K$ is the retention count per layer). This guarantees that both corpora contribute equally to the protected subset, addressing selection bias toward any single corpus’ signal.
3. Budget-Preserving Mask Construction: The protected expert set is merged as a hard constraint into a candidate pruning mask (initialized by layerwise output reconstruction minimization on C4). If this combination exceeds the retention budget, the smallest unprotected experts (by average score) are removed until $K$ is met. Protected experts are always retained.

This algorithm requires only forward-passes, incurs minimal compute cost, and eliminates reliance on downstream training data or fine-tuning.

Experimental Setup and Results

Evaluation is conducted on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base, each with 60+ routed experts per layer. Three retention regimes (25%, 50%, 75%) are examined. Calibration is strictly limited to generic corpora, precluding task leakage.

Baselines include random pruning (multi-seed analysis), direct REAP (with task-specific calibration), and ExpertSparsity (reconstruction-based pruning on C4).

Strong performance is reported as both downstream zero-shot task accuracy (Common Avg over six benchmarks) and perplexity (WikiText2, C4). Generic TB-Coverage exhibits consistently superior accuracy and significantly lower perplexity across all settings, with the most marked improvements under high-aggression pruning (25% and 50% retain). This pattern amplifies the value of the coverage mechanism in preserving crucial behavior under tight pruning constraints.

Figure 1: Common Avg accuracy across methods and retain ratios on both models. Generic TB-Coverage (green) consistently achieves the highest average accuracy across all settings.

Figure 2: WikiText2 and C4 perplexity (log scale, lower is better) across methods and retain ratios. Generic TB-Coverage achieves substantially lower PPL than all baselines, with the largest gains under aggressive pruning.

For Qwen1.5-MoE-A2.7B, Common Avg increases by up to +0.080 over ExpertSparsity, and WikiText2/C4 perplexity improvements are sizable, especially at the 25% retain setting (C4 PPL: 259 vs. 358 for ExpertSparsity). For DeepSeek-MoE-16B-Base, similar trends are observed, underscoring the method’s robustness. Random pruning demonstrates substantial seed variance; single-seed results can appear competitive but are not dependable, reaffirming the necessity of principled selection strategies.

Implications and Theoretical Impact

The primary implication of Generic TB-Coverage is a formalization of coverage as an objective in MoE expert pruning, demonstrating empirically that balancing expert utility across generic corpora more effectively preserves a model’s downstream capabilities in a calibration-free regime. This fundamentally challenges the sufficiency of scalar expert ranking for downstream-agnostic compression and aligns with broader modularity and diversity principles in network structure optimization.

Practically, this method offers a low-cost, plug-and-play pruning rule for open MoE models, requiring no downstream access, backpropagation, or gating adaptation—an asset in production environments where calibration and fine-tuning are computationally or ethically constrained.

Theoretically, the coverage rule paradigm may inspire further research on formal diversity objectives, multi-corpus calibration strategies, and the interplay between modularity, specialization, and redundancy in large-scale LLMs. Extensions might include data-driven selection of coverage budgets, adaptive routing adaptation post-pruning, or integration with quantization and distillation techniques.

Limitations and Future Directions

While the experimental evidence supports the efficacy of cross-corpus coverage, several limitations warrant mention. The method is static and does not adapt router behavior or optimize explicit diversity metrics. Only two open-base MoE models and two diverse but generic corpora are considered; extrapolation to larger models, wider expert sets, or domain-specialized corpora remains untested. Further, negligible gains are observed for mathematical reasoning tasks (GSM8K, Math500), suggesting that coverage preservation across generic corpora alone does not suffice for complex, skill-specific reasoning behaviors.

Future work could address optimal coverage budget selection, dynamic routing adaptation post-pruning, ablation on additional corpora, or the integration of explicit diversity regularization. Evaluation of end-to-end inference speed and footprint post-pruning would also be valuable for deployment scenarios.

Conclusion

Generic TB-Coverage provides a coverage-aware pruning strategy that robustly preserves downstream performance and language modeling stability in sparse MoE LMs without requiring downstream calibration data. Its round-robin coverage rule effectively balances expert retention across heterogeneous generic corpora, outperforming existing scalar-based and reconstruction-driven pruning methods, particularly under aggressive compression. These findings promote coverage as a crucial desideratum in the design of scalable, deployment-ready pruning algorithms for large-scale LLMs.