Less is MoE: Trimming Experts in Domain-Specialist Language Models

Abstract: Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning. We trace this failure to the granularity of compression: important capabilities are distributed across experts but concentrated in FFN sparse intermediate dimensions. To identify these dimensions, we use Fisher importance which outperforms activation-, router-score-, and magnitude-based alternatives, and identifies tiny sets of task-critical dimensions: in Qwen1.5-MoE, removing as few as 12 of 1.35M routed-FFN intermediate dimensions collapses GSM8K accuracy while largely preserving factual-knowledge performance. Building on this, we propose Fisher-MoE, which operates within FFN to remove intermediate dimensions ranked by Fisher importance. At the same 50% MoE compression ratio, Fisher-MoE preserves model capability, while reducing weight memory by ~45% and improving inference throughput by 21%. These findings suggest intermediate dimension granularity is an effective unit for both compression and ranking where capability concentrates in MoE models.

• The paper demonstrates that targeting sparse FFN intermediate dimensions via Fisher information effectively preserves complex reasoning even at a 50% compression ratio.
• It employs fine-grained Fisher scoring over calibration data, outperforming heuristic expert-level pruning on diverse benchmarks by avoiding catastrophic degradation in generation tasks.
• The approach reveals that MoE capacity is concentrated in a few critical neurons, enabling significant memory and throughput gains while maintaining model performance.

Fine-Grained MoE Compression: Fisher-Based Pruning of Intermediate Dimensions

Introduction

Mixture-of-Experts (MoE) architectures are central in scaling LLMs due to their efficiency in leveraging conditional computation for high capacity and throughput. However, their aggregate parameter count remains prohibitive for practical deployment under memory, storage, and inference constraints. While previous compression efforts typically operate at the expert granularity using heuristic importance metrics—such as activation frequency, router scores, or magnitude—these approaches are notably brittle when evaluated on general-purpose and reasoning-heavy benchmarks. The work "Less is MoE: Trimming Experts in Domain-Specialist LLMs" (2606.05538) interrogates the failure of standard MoE pruning schemes and introduces a novel compression paradigm that directly targets the sparse intermediate dimensions within FFN experts, guided by Fisher information as the attribution signal.

Failure of Expert-Level and Heuristic MoE Compression

A rigorous empirical evaluation across MoE models (notably Qwen1.5-MoE-A2.7B) and a suite of diverse benchmarks demonstrates that methods targeting entire experts, regardless of the importance signal, result in catastrophic degradation on generation-oriented tasks (e.g., GSM8K, MATH, HumanEval). The analysis identifies two compounding factors: (1) heuristic or activation-based scores misestimate true parameter impact, and (2) the core assumption that critical capability resides at the expert level is invalid—the actual functional locus is distributed and sharply concentrated at the level of specific FFN intermediate dimensions.

Figure 1: Expert-level pruning using various importance metrics on Qwen1.5-MoE exhibits universal collapses at moderate compression ratios.

Fisher Importance and the Localization of Model Capability

Fisher Information as a Fine-Grained Attribution Mechanism

The paper advances empirical Fisher information as a superior, tractable metric for parameter attribution. Unlike magnitude, activation ratio, or router score—each providing only weak or misaligned correlation with true importance—Fisher scores computed over calibration data surface a small set of task-critical dimensions. This is validated by the finding that zero-masking as few as 12 out of 1.35M intermediate dimensions suffices to fully collapse math reasoning performance, while knowledge recall on multi-choice tasks remains intact.

Figure 2: Intermediate dimension compression for a single MoE expert FFN, where Fisher importance uniquely identifies which dimensions may be pruned without loss.

Figure 3: Fisher distribution over FFN intermediate dimensions is extremely heavy-tailed, highlighting a handful of outlier channels carrying the majority of functional load.

This targeting property is not simply a restatement of high-activation or outlier detection, but reflects an information-theoretic sensitivity of each dimension to perturbation, confirming robustness across code, reasoning, and mathematics domains.

Intermediate Dimension Pruning: The Fisher-MoE Approach

Method Formulation

Instead of removing full experts, Fisher-MoE performs structurally sparse compression by ranking all FFN intermediate dimensions (across all experts) using Fisher scores and physically shrinking the up, gate, and down projection matrices to remove only the least essential dimensions. Multiple allocation strategies—per-expert, per-layer, and global—are explored, with flexible pooling over the global set consistently yielding optimal results at fixed budget.

Comparative Efficacy

Comprehensive experiments at a 50% compression ratio reveal that Fisher-MoE retains downstream utility on high-complexity generation tasks, maintaining parity with uncompressed baselines and universally outperforming all prior expert-level pruning algorithms, including those leveraging Fisher at the expert granularity.

Figure 4: Intermediate dimension compression uniformly outperforms expert-level compression at equal parameter budgets, particularly on reasoning-intensive benchmarks.

Figure 5: Robustness of intermediate dimension pruning across varying compression ratios; expert-level approaches fail even at lower sparsity levels.

Mechanistic Insights: Task Selectivity and Structural Redundancy

Fisher-based removal of the bottom-ranked intermediate dimensions is demonstrated to excise genuinely redundant computation, leaving reasoning capacity and generalization mostly unaffected for out-of-domain tasks. In contrast, excising even a tiny fraction of top-ranked dimensions universally destroys multi-step generation, which is mechanistically traced to the attenuation of the BOS (beginning-of-sequence) attention sink in mid-stack layers. This connects the numeric collapse in downstream performance with the disruption of known LLM attention stabilization circuits.

Deployment Value: Efficiency and Compositionality

The structural reduction enabled by Fisher-MoE directly decreases wall-clock inference latency. At 50% compression, up to 45% memory reduction and 21% throughput gain are observed (on A100/H100 accelerators), outperforming pruning strategies based on expert or head removal, which typically fail to deliver commensurate speedup. Fisher-MoE is compatible with quantization; deploying 4-bit AWQ on top of the Fisher-pruned model multiplies the savings in disk/VRAM while incurring no greater accuracy loss than quantization alone.

Theoretical and Practical Implications

Structural Redundancy and Capability Localization

This work empirically verifies that MoE functional capacity is not localized at the level of experts or attention heads, but sharply at a sparse set of intermediate neurons. This suggests that large MoEs remain overparameterized at the structural FFN level and that standard routing, activation, and magnitude proxies are insufficient to expose and target this redundancy. This insight can inform new lines in model attribution, structural pruning of non-MoE LLMs, and the study of "skill" neurons in transformer representations [dai2022knowledge, wang2022finding].

Generalization and Calibration

Fisher-MoE's reliance on calibration data introduces a controllable tradeoff: the score is most effective when computed on representative downstream samples, but generalizes sufficiently well that even domain-mismatched calibration preserves OOD robustness—contradicting concerns of excessive specialization or mode collapse.

Composability

The approach composes well with quantization and domain-specific post-training (SFT), facilitating further reductions in deployment cost with minimal framework modification. This complements recent advances in MoE quantization and low-rank compression [lin2024awq, yang-etal-2024-moe].

Limitations and Future Directions

Scalability to 100B+ parameter MoE architectures and the impact of longer or more diverse calibration/post-training on compression-induced gaps are highlighted as open questions. Additional investigation into the intersection between Fisher-rich intermediate dimensions and circuits underpinning specific linguistic, factual, or reasoning capabilities is warranted. Extending the methodology to dense LLMs, or other structured sparsity forms (e.g., grouped convolutions, attention blocks), also remains unexplored but promising.

Conclusion

By shifting the compression granularity from the expert to the Fisher-ranked intermediate FFN dimension, this work establishes a new regime for MoE compression that robustly preserves, and occasionally enhances, general and specialist task performance at parameter and memory budgets previously unattainable with heuristic approaches. Fisher-MoE not only advances practical model deployment but also illuminates the substrate of capability concentration in sparse LLMs, providing an actionable framework for future model design and interpretability studies.