Pruning via Expert-Selection Frequency (PESF)

• The paper demonstrates that PESF, within the EAC-MoE framework, dynamically prunes experts based on runtime selection frequencies to improve inference speed.
• PESF uses online, sequence-local routing statistics to adaptively skip unimportant experts, differing from static, calibration-based pruning methods.
• Empirical results show notable latency reduction and modest accuracy drops, highlighting PESF’s practical role in managing GPU memory and computation in MoE-LLMs.

Pruning based on Expert-Selection Frequency (PESF) is a dynamic, inference-time expert pruning policy for Mixture-of-Experts LLMs (MoE-LLMs) that prunes experts which are rarely selected by the router for the current input sequence. In the EAC-MoE framework, PESF is paired with Quantization with Expert-Selection Calibration (QESC): QESC is used to preserve the expert selection pattern of the full-precision model under low-bit quantization, while PESF exploits the resulting routing pattern to skip computation for experts that are empirically unimportant for the current sequence (Chen et al., 3 Aug 2025). The method is explicitly sequence-specific and task-adaptive, and is motivated by the observation that, although MoE layers activate only a small number of experts per token, long-sequence or batched inference can still incur latency from computing many or all experts across the sequence (Chen et al., 3 Aug 2025).

1. Position within MoE compression and sparsification

PESF was introduced as one of the two core modules of EAC-MoE, an “Expert-Selection Aware Compressor for MoE-LLMs” designed to address two deployment bottlenecks simultaneously: GPU memory consumption and inference latency (Chen et al., 3 Aug 2025). The framework combines static weight-only quantization for memory reduction with dynamic expert pruning during inference for latency reduction. Its central design principle is to preserve and exploit the router-controlled expert selection mechanism rather than treating MoE blocks as ordinary dense feed-forward layers.

Within that framework, PESF serves the latency side of the design. It operates at runtime, on top of either a full-precision or quantized-and-calibrated MoE model, and uses sequence-local routing statistics to decide which experts to prune for that sequence (Chen et al., 3 Aug 2025). The method is therefore distinct from static pruning schemes that determine a single expert subset before deployment. The paper’s stated motivation is that static pruning based on a global calibration set is brittle because “the same expert can be important for one task but useless for another” (Chen et al., 3 Aug 2025).

Related MoE pruning literature places PESF in a broader family of routing-aware methods, but also clarifies its boundaries. “Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts LLMs” implements a frequency-based pruning baseline that drops experts according to activation frequency on calibration data, and reports that this strategy is worse than their reconstruction-loss-based pruning criterion on Mixtral 8×7B (Lu et al., 2024). “Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts” similarly treats visitation frequency as an analyzed signal and as a baseline, but concludes that expert output similarity is more suitable for task-agnostic pruning than routing frequency alone (zhang et al., 2024). Taken together, these results delimit PESF’s intended scope: it is not presented as a universal replacement for all expert pruning criteria, but as a sequence-local runtime heuristic aligned with MoE routing behavior.

2. Formalization of expert-selection frequency

PESF is defined for a decoder-only MoE layer with $N$ experts and top-$K$ routing. Given an input token representation $\mathbf{x}$, the router produces logits $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$ and probabilities

$$\mathbf{s} = \text{Softmax}(\mathbf{r}) \in \mathbb{R}^N.$$

The model then selects the top-$K$ experts by probability (Chen et al., 3 Aug 2025).

For a given sequence of length $l$, PESF counts how many times each expert is selected at a given layer. For expert $i$, let $c_i$ denote the number of tokens in the sequence for which expert $i$ is among the top-$K$0 routed experts. Since the total number of expert selections over the sequence is $K$1, the expected average selection count per expert under perfectly uniform routing is

$K$2

This quantity is the baseline against which observed expert usage is compared (Chen et al., 3 Aug 2025).

PESF introduces a dimensionless pruning hyperparameter $K$3. Its pruning rule is

$K$4

where $K$5 is the expert’s selection count for that sequence in that layer (Chen et al., 3 Aug 2025). The method is therefore based on relative under-selection with respect to the uniform baseline. Experts whose observed selection count is less than an $K$6-fraction of the expected uniform count are pruned.

This definition makes expert-selection frequency in PESF a sequence-level, per-layer statistic rather than a persistent model property. The counts are computed during inference on the current sequence, not on a separate calibration corpus, and they are discarded afterwards (Chen et al., 3 Aug 2025). This sharply contrasts with the “Frequency” baseline in (Lu et al., 2024), where expert counts are estimated offline on calibration data and used to permanently prune the least-visited experts.

3. Empirical rationale: task preference and long-tail expert usage

The empirical rationale for PESF rests on two observations reported in EAC-MoE’s expert-selection analysis. First, the model exhibits task-specific expert preference. For each task type, including QA/CR, Math, Code, and specific language categories such as French, the model tends to use a consistent subset of experts across datasets of that type (Chen et al., 3 Aug 2025). This is quantified by normalized selection frequency vectors $K$7 and cosine similarities,

$K$8

Datasets from the same task category have high similarity, whereas cross-category similarity is much lower (Chen et al., 3 Aug 2025). This implies that expert importance is task-dependent rather than globally fixed.

Second, expert usage within a task displays a long-tail pattern. The paper reports that, for a fixed task, some experts are selected with very high frequency while others are selected very rarely, far below the uniform rate $K$9, especially in models with many experts such as 64-expert DeepSeek-MoE. Appendix figures are described as showing per-layer histograms where a few experts exceed 30–40% selection frequency and many are below 1% (Chen et al., 3 Aug 2025). This observation supports the idea that a given task or sequence often depends heavily on a minority of experts.

These two findings motivate PESF’s sequence-local design. Because importance varies by task, a single globally pruned expert set can over-prune experts that are unimportant on one task but necessary on another. Because usage is long-tailed within a task, there are experts that contribute little on the current sequence but still incur kernel launches and matrix multiplications if left active. PESF therefore measures routing frequency online and prunes experts whose selection counts are significantly below the sequence-wise average (Chen et al., 3 Aug 2025).

Related work frames this same issue differently. In (zhang et al., 2024), expert visitation frequency is treated as too coarse a signal for task-agnostic pruning, because it does not distinguish between experts that are functionally redundant and experts that are low-frequency but unique. In (Lu et al., 2024), the calibration-set frequency baseline underperforms reconstruction-loss-based expert pruning and can even perform worse than random under some settings. These results do not contradict PESF directly, because PESF is not a task-agnostic permanent pruning rule; instead, they suggest that expert-selection frequency is most defensible when it is used online, locally, and conditionally on the current sequence.

4. Inference-time mechanism and integration with QESC

PESF is a post-training, inference-time-only method applied during the prefill stage, that is, while processing the full input sequence or a block of tokens (Chen et al., 3 Aug 2025). For each input sequence of length $\mathbf{x}$0, the model first runs with standard routing. At each MoE layer, the router computes top-$\mathbf{x}$1 experts for each token, and the counts $\mathbf{x}$2 are accumulated for every expert in that layer (Chen et al., 3 Aug 2025).

After counts are obtained, a layer-specific threshold is computed as $\mathbf{x}$3. Experts satisfying $\mathbf{x}$4 are marked as pruned for that sequence (Chen et al., 3 Aug 2025). PESF does not rank experts globally by a continuous importance score; it uses a simple local threshold on counts, independently in each layer. Consequently, effective pruning ratios can differ by layer depending on how skewed the selection distributions are (Chen et al., 3 Aug 2025).

The paper frames pruning primarily as computation skipping. Once an expert is pruned, computation involving that expert is skipped for the sequence: the expert output $\mathbf{x}$5 is not computed, and routing can be adjusted to ignore the pruned experts (Chen et al., 3 Aug 2025). The authors do not define a new router formalism after pruning. Router weights are not retrained or modified, and no fine-tuning after pruning is reported. The text focuses on skipping expert computation rather than redesigning the routing mechanism itself (Chen et al., 3 Aug 2025).

In EAC-MoE, PESF is applied after QESC. The stated pipeline is: MHSA is quantized to 4-bit; experts are quantized to 2, 2.5, or 3 bits, with average bit-widths of 2.06, 2.54, and 3.03 respectively; routers are kept at full precision; and routers are calibrated layer by layer using TopK-MSE to preserve expert-selection patterns of the full-precision model (Chen et al., 3 Aug 2025). PESF is then run on the already quantized and calibrated model, using expert selection frequencies that occur after quantization and router calibration (Chen et al., 3 Aug 2025). The intended logic is that QESC first reduces expert-shift caused by quantization, so the frequencies measured by PESF remain aligned with the model’s true task-specific expert specializations.

The authors state that PESF “introduces only a single-step online computation” and results in “virtually no additional delay” (Chen et al., 3 Aug 2025). Counting requires integer increments when experts are selected, which is $\mathbf{x}$6 per layer per sequence, and threshold comparison is $\mathbf{x}$7 per layer. The paper notes that the counts can be maintained on GPU or CPU and that no large extra memory footprint or extra GPU-host communication is required (Chen et al., 3 Aug 2025).

5. Hyperparameters, measured trade-offs, and comparative results

The central hyperparameter of PESF is $\mathbf{x}$8, the pruning threshold factor in the rule $\mathbf{x}$9. In DeepSeek-MoE experiments, the paper explores $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$0 in 0.1 steps and highlights two operating points: $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$1, described as conservative, and $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$2, described as aggressive (Chen et al., 3 Aug 2025).

The reported trade-offs are summarized below.

Setting	Reported effect	Context
$\mathbf{r} = \{r_0, \dots, r_{N-1}\}$3	~10% latency reduction; average zero-shot accuracy drop within ~0.5%	DeepSeek-MoE results
$\mathbf{r} = \{r_0, \dots, r_{N-1}\}$4	>1.3× average speedup on several models; accuracy loss within ~1.5% on DeepSeek-MoE and Phi3.5-MoE, slightly higher on some tasks	DeepSeek-MoE results
Default in full EAC-MoE	$\mathbf{r} = \{r_0, \dots, r_{N-1}\}$5	Chosen to keep accuracy loss very low

As a standalone pruning method without quantization, PESF is compared against EES and ODP on Mixtral-8x7B, Phi3.5-MoE, DeepSeek-MoE-16B-base, and Qwen1.5-MoE-A2.7B (Chen et al., 3 Aug 2025). With $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$6, the paper reports the following pairs of speedup and average zero-shot accuracy: Mixtral-8x7B, 1.08× and 72.19 versus a 72.64 baseline; Phi3.5-MoE, 1.12× and 69.27 versus 69.62; DeepSeek-MoE-16B, 1.11× and 61.28 versus 61.38; and Qwen1.5-MoE-A2.7B, 1.14× and 64.64 versus 64.72 (Chen et al., 3 Aug 2025). Under the same conditions, PESF is stated to outperform EES and ODP simultaneously in both accuracy and speedup (Chen et al., 3 Aug 2025).

With $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$7, speedups increase but degradation becomes model-dependent. The reported results are: Mixtral-8x7B, 1.13× speedup with accuracy dropping to 58.22; Phi3.5-MoE, 1.30× and 67.95; DeepSeek-MoE-16B, 1.45× and 60.41; and Qwen1.5-MoE-A2.7B, 1.47× and 63.87 (Chen et al., 3 Aug 2025). The Mixtral result is explicitly identified in the paper as a limitation case.

In the full EAC-MoE configuration, Table 9 of (Chen et al., 3 Aug 2025) combines 3.03-bit QESC with PESF at $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$8. For Mixtral-8x7B, parameters are reduced from 93.41 GB to 18.98 GB via QESC, speedup increases from 1.54× with QESC alone to 1.68× with QESC+PESF, and accuracy changes from 72.64 to 72.21 under QESC and to 71.68 under QESC+PESF (Chen et al., 3 Aug 2025). Across four models, the paper reports average speedup of approximately 1.64× with average accuracy loss below 1% (Chen et al., 3 Aug 2025). Against MC-MoE on Mixtral-8x7B, EAC-MoE with QESC+PESF at $\mathbf{r} = \{r_0, \dots, r_{N-1}\}$9 is reported to achieve better or equal perplexity, higher zero-shot accuracy at comparable bit-widths, and similar or slightly better speedup, including 1.82× versus 1.80× at 2.06 bits (Chen et al., 3 Aug 2025).

These results should be interpreted together with the evidence from (Lu et al., 2024) and (zhang et al., 2024). Both papers show that frequency-based pruning is weak as a static, task-agnostic criterion. EAC-MoE’s reported gains therefore suggest that the decisive property of PESF is not frequency alone, but frequency measured online on the actual sequence and used for temporary runtime pruning rather than permanent structural simplification.

6. Limitations, scope, and points of interpretation

The most explicit limitation is that PESF is only applicable during the prefill stage, not during autoregressive generation, because its frequency estimate is defined over the current input sequence and is not meaningful when only a single token is processed at a time (Chen et al., 3 Aug 2025). The paper states this directly and does not provide a generate-stage extension. A plausible implication is that extending PESF to token-by-token decoding would require some alternative accumulation scheme across previous tokens or prior sequences, but such a mechanism is not part of the reported method.

PESF also depends on the presence of skewed expert usage. The paper notes that it works best when the number of experts is large and expert specialization is strong, as in settings with 16 or 64 experts (Chen et al., 3 Aug 2025). Mixtral-8x7B is presented as a counterexample with weaker sparsity in expert selection, and aggressive thresholds such as $$\mathbf{s} = \text{Softmax}(\mathbf{r}) \in \mathbb{R}^N.$$0 cause larger accuracy degradation there (Chen et al., 3 Aug 2025). This suggests that routing skew is not merely a side observation but a precondition for aggressive PESF settings.

Sequence composition matters as well. Because PESF uses sequence-level statistics, short or heterogeneous sequences may produce noisy importance signals, and sequences containing rare but important tokens may be disproportionately affected if those tokens depend on a rarely used expert (Chen et al., 3 Aug 2025). The paper states that this risk is mitigated by choosing conservative $$\mathbf{s} = \text{Softmax}(\mathbf{r}) \in \mathbb{R}^N.$$1 and by using QESC so that routing remains faithful after quantization (Chen et al., 3 Aug 2025). This is mitigation rather than a formal guarantee; the paper explicitly does not provide theoretical guarantees for PESF and presents its justification as empirical and intuitive.

The scope of evaluation is also bounded. PESF is evaluated on Mixtral-8x7B, Phi3.5-MoE, DeepSeek-MoE-16B-base, and Qwen1.5-MoE-A2.7B, with benchmarks including WikiText2 perplexity, eight EleutherAI LM Harness zero-shot tasks, and additional GSM8K and HumanEval results in the quantization analysis (Chen et al., 3 Aug 2025). The method was not tested on larger scales such as DeepSeek-V3 due to resource limits (Chen et al., 3 Aug 2025). Any extrapolation to substantially larger MoE systems is therefore inferential rather than directly established by the reported experiments.

A final point concerns the meaning of “frequency-based pruning” across papers. In (Lu et al., 2024), frequency means activation counts accumulated over a calibration set and then used for permanent expert removal. In (zhang et al., 2024), visitation frequency is likewise a calibration-set statistic and is explicitly criticized as insufficient for task-agnostic pruning. In (Chen et al., 3 Aug 2025), by contrast, expert-selection frequency is a transient, per-sequence runtime signal. The shared terminology can therefore obscure an important methodological distinction. PESF is best understood not as a generic endorsement of frequency as an expert-importance oracle, but as a specific dynamic policy that leverages sequence-local routing sparsity to convert sparse activation into measurable latency reduction.