REAM: Merging Improves Pruning of Experts in LLMs

Abstract: Mixture-of-Experts (MoE) LLMs are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.

• The paper presents REAM, a unified framework that combines router-weighted merging with pseudo-pruning to compress MoE LLMs while preserving expert specializations.
• It employs dual metrics integrating activation magnitudes and logit-based similarity, enhanced by sequential layerwise merging and neuron alignment for stable recalibration.
• Empirical results show near-lossless generative performance at 25% compression, outperforming previous methods on code and mathematical reasoning tasks.

REAM: Merging Improves Pruning of Experts in LLMs

Introduction and Motivation

Mixture-of-Experts (MoE) architectures are the foundation of state-of-the-art LLMs operating with massive parameter counts, offering high empirical performance by activating subsets of experts for each token to achieve computational efficiency. However, the deployment of these models is widely constrained by the memory requirements imposed by storing all experts, especially because high redundancy across experts is common as observed empirically in overparameterized MoE architectures.

REAM (Router-weighted Expert Activation Merging) addresses limitations in prior expert-pruning and merging strategies. Existing pruning methods, such as REAP, aggressively drop experts based on saliency, often discarding specialized knowledge. Merging, while more conservative, can degrade performance by forcing heterogeneous experts together and neglecting the router’s role in specialization. REAM is introduced as a unified and data-driven expert compression framework that synthesizes the benefits of both approaches while explicitly balancing the memory-compute trade-off.

Methodology

REAM is distinguished by its coupling of router-aware merging with a pseudo-pruning strategy to preserve expert knowledge more effectively.

1. Expert Saliency and Similarity: REAM advances standard saliency formulations by combining both router-weighted activation magnitudes and logit-based similarities. This dual approach ensures that expert importance reflects both frequency and contextual contribution in alignment with the router’s specialization policy.
2. Pseudo-Pruning and Expert Grouping: Rather than globally clustering all experts, REAM pseudo-prunes by designating the most salient experts as protected centroids and greedily merging less salient, similar experts into these groups while many centroids remain singletons. This preserves high-saliency experts unchanged while benefiting from merging among the redundant subset.

Figure 1: Comparison between merging, pruning, and pseudo-pruning in expert reduction; REAM’s pseudo-pruning selects salient centroids for merging, yielding a hybrid structure.

1. Combined Activation and Weight Alignment: To further stabilize merging, the method computes a neuron-level permutation alignment leveraging a balanced cost matrix from both activation similarity (on calibration data) and raw weight distance. The Hungarian algorithm is applied for neuron matching, ensuring averaged weights are meaningfully aligned.
2. Sequential Merging: Unlike approaches that compress all layers in parallel based on pre-collection, REAM performs per-layer, forward recomputation of activations post-merge. This sequential recomputation captures updated layerwise statistics and mitigates error accumulation across layers, yielding improved post-merge calibration fidelity.

Empirical Evaluation

REAM is evaluated on multiple strong MoE LLMs: Qwen3-30B/80B variants, Qwen3-Coder-Next, and GLM-4.5-Air, using a broad suite of eight multiple-choice (MC) benchmarks and six generative (GEN) reasoning/coding tasks. Compression rates at 25% (e.g., 128 to 96 experts) and 50% (e.g., 128 to 64 experts) are studied.

A unique aspect is the analysis of calibration data mixtures for merging: router activations and expert saliencies are computed on mixtures of general text, math, and code, systematically varying the mix to uncover the dependence of MC/GEN trade-offs on calibration distribution.

Task Trade-Off and Calibration Dynamics

REAM exhibits superior flexibility across the MC and GEN spectrum. Notably, the method establishes that no static calibration mixture universally optimizes both MC and GEN; instead, the optimal trade-off is strongly determined by data choice, with general text favoring MC and code favoring GEN.

Figure 2: Pearson correlation $r$ between calibration dataset composition and MC/GEN scores, highlighting the systematic MC–GEN trade-off.

REAM’s calibration dynamics outperform both pure pruning and previous merging methods, achieving nearly uncompressed GEN performance at 25% compression, particularly on code-centric mixtures, while maintaining or exceeding MC adequacy on general/mathematical mixtures.

Figure 3: Ablation reflecting impact of REAM’s components on average MC and GEN performance; the largest drop arises from using naive frequency-based saliency.

Pareto Frontier Analysis

The paper quantifies each method's MC–GEN Pareto frontier across all calibration mixtures, using the hypervolume (HV) indicator for performance ceiling. REAM achieves the highest HV and offers Pareto-optimal configurations for the majority of mixtures tested, documenting robust calibration sensitivity and overall superiority in balancing MC/GEN trade-offs.

Figure 4: Pareto frontiers over calibration mixtures at 64 experts; REAM dominates the MC$\times$GEN space compared to alternatives.

Results on Large-Scale MoE Models

Experiments on Qwen3-80B and GLM-4.5-Air verify that REAM retains a near-lossless GEN score at standard compression rates, typically outperforming REAP, especially on code and mathematical GEN tasks. The only persistent degradation is noted for knowledge-intensive out-of-domain tasks, suggesting that further domain-aware adaptation or larger calibration sets may provide incremental gains.

Practical and Theoretical Implications

REAM’s explicit modeling of router-aware expert importance, combined with sequential layerwise merging, represents a disciplined shift in expert reduction. The strong empirical findings directly support static, retraining-free deployment—a critical requirement in restricted or confidential environments. The results reaffirm the necessity of careful calibration selection, motivating future automated meta-calibration or even domain-conditional routing/merging policies.

Theoretically, the observed MC–GEN asymmetry raises open questions regarding the specialization and partitioning of expert functionality in large generative models, potentially guiding future architecture designs toward task-adaptive modularity.

Conclusion

REAM defines a new state-of-the-art for compressing MoE LLMs by integrating router-weighted expert merging with a pseudo-pruning framework. The method achieves high compression rates with negligible loss on diverse benchmarks and establishes the importance of data-driven, task-aligned calibration in expert compression. REAM’s approach and empirical findings will drive further work on memory-efficient deployment, calibration-robust compression, and understanding specialization in modular LLMs.