EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

Abstract: Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leading to suboptimal resource use. We propose EPnG, an adaptive prune-and-grow framework that reallocates LoRA capacity based on expert importance derived from router gate probabilities. EPnG prunes under-utilized experts and expands high-importance experts via rank growth with orthogonal initialization, while maintaining a fixed parameter budget. Across OLMoE and Qwen1.5-MoE, EPnG consistently outperforms LoRA under the same budget and achieves performance comparable to full fine-tuning while updating only 0.55%-0.72% of parameters (up to 140x-180x fewer). These results demonstrate that aligning PEFT with MoE routing yields a more effective and scalable fine-tuning strategy.

• The paper introduces a dynamic prune-and-grow method that reallocates LoRA capacity based on expert utilization statistics.
• It leverages router gate data to prune low-importance experts and boost high-importance ones under a fixed parameter budget.
• Empirical results across tasks show EPnG achieving performance comparable to full fine-tuning with significantly fewer updated parameters.

EPnG: Adaptive Expert Prune-and-Grow for Efficient Mixture-of-Experts Fine-tuning

Introduction and Motivation

Mixture-of-Experts (MoE) architectures are foundational for scaling LLMs via conditional computation, wherein only a selected subset of experts per token is activated during inference and training. Although this strategy yields high computational efficiency for forward and backward passes, adapting MoE-based LLMs for downstream tasks remains resource-intensive due to redundant experts and uniform parameter allocation. Traditional Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), disregard the routing dynamics inherent to MoE architectures, resulting in unoptimized updates—critical experts may receive insufficient adaptation, while underutilized experts are over-parameterized.

The EPnG (Expert Prune-and-Grow LoRA) framework directly addresses these inefficiencies by leveraging router gate statistics to dynamically reallocate LoRA capacity across MoE experts. The method adaptively prunes under-utilized experts and expands the low-rank subspace (LoRA rank) of high-importance experts, preserving a fixed fine-tuning parameter budget.

Figure 1: Standard MoE with uniform LoRA allocation (top) vs. EPnG (bottom). EPnG prunes low-importance experts and increases the ranks of frequently selected ones, yielding an importance-aware allocation under a fixed budget.

EPnG Methodology

EPnG operates in a dynamic loop, guided by expert utilization statistics derived from router gating during fine-tuning. The core steps include:

• Expert Importance Estimation: Aggregate router gate probabilities across the training set to produce importance scores per expert, thus quantifying expert utilization.
• Pruning: Identify and remove LoRA adapters associated with the least important experts, reducing ineffective parameter allocation.
• Growth: Allocate the freed parameter budget to increase the adaptation capacity (i.e., LoRA rank) of highly utilized experts. New components are orthogonally initialized to guarantee non-redundancy.
• Adaptive Scheduling: Periodically repeat the prune-and-grow steps after a warm-up phase, maintaining budget-neutral adaptation and dynamically tracking shifts in expert importance.

  Figure 2: Illustration of EPnG's end-to-end training procedure.

This approach contrasts with standard PEFT and MoE-aware methods by making expert importance an explicit, trainable signal in parameter allocation. The adaptive reallocation not only resolves the issue of rank selection sensitivity seen in static LoRA but also more effectively exploits the MoE architecture’s sparsity by concentrating adaptation capacity on experts aligned with current routing dynamics.

Empirical Results

EPnG is evaluated on OLMoE and Qwen1.5-MoE backbones across mathematical reasoning (GSM8K, MATH), code generation (HumanEval, MBPP), and preference following (PrefEval). The results decisively demonstrate that EPnG:

• Achieves comparable or superior performance to full fine-tuning on GSM8K, approaching or surpassing accuracy levels with only $0.55\%$–$0.72\%$ updated parameters—a drastic reduction (140×–180× fewer) over full model updates.
• Outperforms static LoRA under the same parameterization budget, with up to +1.82 absolute points improvement on GSM8K and notable gains in code evaluation metrics.
• Matches or exceeds the performance of MoE-focused baselines such as ESFT using an order-of-magnitude fewer parameters on both OLMoE and Qwen1.5-MoE.

These strong numerical results empirically support the hypothesis that dynamic, routing-aware PEFT is substantially more effective than uniform allocation for sparse MoE models.

Ablation and Analytical Studies

A systematic ablation uncovers the contributions of the pruning and growth mechanisms:

• Prune-only adaptation yields greater gains than growth-only, highlighting the efficient regularization from suppressing redundancy. However, the prune-and-grow combination consistently produces the highest overall performance, identifying synergy between removing unnecessary expert adaptation and reinforcing essential capacity.

Figure 3: Ablation study demonstrates complementary effects of pruning and growing.

EPnG also demonstrates robust behavior with respect to LoRA rank sensitivity, outperforming baselines particularly at low parameter budgets and reducing variance across ranks. Performance saturates with moderate prune-and-grow fractions (optimal around 0.2), balancing the diversity and efficiency of expert specialization.

Layer-wise and Temporal Adaptation Dynamics

In-depth adaptation analysis reveals that EPnG's capacity reallocation is not layer-uniform:

• Aggressive pruning occurs in shallow layers, while adaptation capacity is reallocated to deeper, more task-relevant layers, resulting in a non-uniform, data-driven rank profile post adaptation.

  Figure 4: Layer-wise behavior of EPnG showing aggressive pruning in shallow layers and capacity redistribution to deeper layers.

Crucially, expert importance scores are not static during fine-tuning; significant shifts occur, with previously unimportant experts becoming vital and vice versa. EPnG tracks and adapts to these changes in situ, strengthening the rationale for dynamic rather than static allocation strategies.

Figure 5: Expert importance before and after fine-tuning reveals dynamic shifts in which experts matter most for the downstream task.

Implications and Future Trajectories

EPnG demonstrates that effective parameter-efficient adaptation in MoE LLMs requires tightly coupling the adaptation path with the model’s internal routing signals. There are several implications:

• Practical: The reduction in trainable parameters—far below 1% of the model—enables efficient, rapid adaptation of state-of-the-art (sparse) LLM backbones on edge devices or platforms with strict resource constraints, with negligible, if any, loss in performance.
• Theoretical: The results challenge the assumption of fixed LoRA configurations, promoting adaptive importance-based reallocation as an optimization subproblem in model adaptation. This suggests that further exploration into dynamic, data-dependent parameter allocation could refine or even supplant extant PEFT strategies.
• Extensibility: The prune-and-grow paradigm is architecture-agnostic and can, in principle, be generalized to other sparse or modular architectures beyond MoE, guiding the broader field toward more sophisticated allocation and adaptation frameworks.
• Limitations: EPnG’s irreversibility in pruning may discard experts that later gain relevance; addressing reversible or soft-pruning variants and improving hyperparameter robustness remains a key area for future research.

Conclusion

EPnG provides a robust, routing-aware paradigm for parameter-efficient fine-tuning of MoE LLMs. By leveraging router-derived expert importance for adaptive prune-and-grow allocation of LoRA capacity, EPnG achieves strong empirical performance with minimal parameter overhead. The approach fundamentally reframes expert-wise parameter allocation as a dynamic optimization problem during adaptation, reinforcing the interplay between MoE routing and adaptation capacity. Future developments in AI model adaptation will likely build upon this principle, seeking ever more fine-grained and responsive control over sparse model parameterization.

---

Reference:

"EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning" (2607.01789)