GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

Abstract: Mixture-of-Experts LLMs (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation. Source code is available at https://github.com/jndeng/GEMQ .

• The paper introduces a global bit allocation approach that minimizes quantization loss by leveraging a linear program across all experts.
• It employs efficient, parameter-light global router fine-tuning to realign routing distributions and restore performance after quantization.
• The progressive quantization strategy incrementally refines expert importance estimation, enabling aggressive memory reduction with minimal accuracy loss.

Global Expert-Level Mixed-Precision Quantization for MoE LLMs

Problem Formulation and Motivation

Mixture-of-Experts (MoE) LLMs have become increasingly predominant in scaling LLM capacity efficiently via sparsely-gated expert subnetworks. Despite this efficiency in compute, the requirement to hold all expert parameters in memory remains a deployment bottleneck, especially for models such as Mixtral-8×7B, whose expert parameters dominate model size and prevent single-device inference even on high-end GPUs. Quantization—mapping floating-point model weights to low-precision fixed-point representations—mitigates memory usage, but existing approaches to MoE-LLM quantization exhibit key deficiencies.

Current state-of-the-art methods adopt local, layer-wise bit allocation heuristics for selecting the precision assigned to each expert. This locality fails to capture the substantial heterogeneity of expert and layer sensitivities, as confirmed by empirical gradient analyses. Moreover, naive quantization distorts expert router behaviors; low-bit quantization substantially alters routing distributions, resulting in degraded downstream performance. GEMQ is proposed to address these limitations by introducing a global, task-driven, quantization error-aware bit allocation across all experts and an efficient post-quantization global router adaptation.

Figure 1: (a) Squared-gradient analysis reveals inter- and intra-layer expert sensitivity heterogeneity; (b) Over 40% of tokens are routed to different experts following 1.5-bit quantization, indicating severe router distribution distortion.

Methodology

GEMQ’s contribution is threefold: (1) formulation of expert-level bit allocation as a global linear program minimizing loss proxy under a bit budget, (2) parameter-efficient global router fine-tuning to adapt to quantized expert weights, and (3) a progressive multi-stage quantization strategy that incrementally aligns expert importance estimation to the true loss behavior under low-precision.

Figure 2: Schematic of the GEMQ workflow, which integrates global importance-driven expert quantization and router fine-tuning in a progressive, multi-stage loop.

Global Expert Bit-Width Allocation

The proposed bit allocation strategy leverages a closed-form Taylor expansion of the quantization-induced loss increase, utilizing a diagonal Fisher Information Matrix (FIM) approximation for scalability. Crucially, by computing importance estimates $\Delta\widetilde{\mathcal{L}}_{ij}$ across all experts and bit settings (not just within each layer), GEMQ’s linear program selects per-expert bit-width assignments that optimize for minimum aggregate proxy loss under the global bit budget. Unlike heuristic approaches with hand-tuned coefficients, this procedure is hyperparameter-free and allows for more nuanced inter-layer allocation—allocating more bits to globally critical experts independent of their layer.

Figure 3: Analysis and visualization of global expert-level bit-width assignments, showing that the allocation adapts to layer-wise sensitivity variation.

Global Router Fine-Tuning

Subsequent to expert quantization, GEMQ decouples all router weights and fine-tunes only these parameters on a small calibration dataset (typically 128-2048 sequences of 2048 tokens). Because routers comprise less than 0.04% of model parameters, this process is highly efficient and does not incur the cost associated with full-scale quantization-aware training. Empirically, this fine-tuning process dramatically reduces the substantial router distribution shifts observed post quantization, restoring routing accuracy and thus model performance.

Figure 4: Ablation on router fine-tuning settings demonstrates convergence within a minimal number of training steps and negligible overhead.

Figure 5: Router distribution statistics from selected layers show that fine-tuning realigns quantized router outputs toward the original full-precision distribution.

Progressive Quantization

The local-minimum assumption underlying the Taylor-series loss approximation falters at extreme low-bit settings, as large quantization errors may induce abrupt loss changes, especially where router-triggered discrete mode switches occur. GEMQ introduces a progressive procedure, wherein importance estimation at any target bit-width leverages the previously fine-tuned quantized model at a nearby higher bit-width. This mitigates estimation errors due to nonlinearity and routing discontinuity, further stabilizing performance in aggressive low-bit regimes.

Figure 6: Depicts cases of expert importance estimation under varying quantization errors and highlights the advantage of using fine-tuned, progressively quantized weights as estimation anchors.

Empirical Evaluation

Extensive evaluation across four representative MoE LLMs—DeepseekV2-Lite, Qwen1.5-MoE, Qwen3-30B-A3B, and Mixtral-8×7B—demonstrates the superiority of GEMQ over both uniform and state-of-the-art mixed-precision quantization approaches such as PMQ and SpQR.

• For Mixtral-8×7B, quantization from FP16 (87 GB) to 2.5 bits/expert (16 GB) leads to an 82% reduction in memory, with only a 7% accuracy drop on 5-shot MMLU. This outperforms PMQ by over 3% in accuracy.
• The framework achieves deployment of Mixtral-8×7B at 2.5 bits/expert on a single H100 GPU, enabling a decoding speed of 82.5 tokens/s.
• GEMQ consistently maintains higher average zero-shot and in-context learning accuracy relative to all tested baselines in all bit-width settings.

This performance is robust to calibration data sample size (plateauing at ~128 sequences) and is further enhanced by domain-aligned calibration (e.g., supplementing C4 with MATH for math reasoning tasks).

Figure 7: Ablation of router fine-tuning (“RFT”) and progressive quantization (“PQ”) on perplexity and zero-shot accuracy, showing each technique’s additive contributions.

Analysis and Ablations

The ablation studies confirm:

• Local (layer-wise) versus global bit allocation: Only global allocation can leverage inter-layer heterogeneity, yielding statistically superior allocations and performance. Ad hoc regularization (e.g., enforced high-bit experts per layer) further enhances extreme low-bit stability.
• Router adaptation: Rigid alignment of router distributions to FP values yields lower improvements than loss-optimized fine-tuning for the quantized regime.
• Progressive quantization: Loss estimation from the immediately prior fine-tuned quantized anchor produces lower expert selection change ratios and improved perplexity, especially in ultra-low-bit settings.

Figure 8: Robustness analysis of expert importance proxies and corresponding bit-width allocation across multiple calibration datasets and seeds, showing consistency.

Figure 9: Router selection change ratio (percentage of tokens assigned to different experts) across estimation anchor models, quantifying router shift.

Practical Implications and Future Directions

By compressing MoE LLMs to aggressive low-precision, GEMQ lowers deployment cost and enables on-device or edge deployment scenarios previously infeasible for multi-billion parameter models. The global formulation demonstrates that cross-layer heterogeneity is essential and that per-expert importance estimation provides a substantially more efficient Pareto frontier. The router calibration procedure is essentially plug-and-play with negligible additional resource cost.

The progressive quantization and task-level loss analysis framework provides a reusable methodology that could extend to other weight-sharing or sparse-activation architectures. Future research directions include the optimization of calibration datasets for broader domain generality, integrating hardware cost co-optimization at the allocation stage, and unifying expert pruning and quantization within a single global compression procedure.

Conclusion

GEMQ resolves critical weaknesses in MoE LLM quantization by globally optimizing bit assignments and repairing routing adaptation post-quantization. This results in a principled approach that achieves substantial compression without sacrificing core LLM capabilities. The empirical and analytical evidence establishes global expert importance modeling and minimal router adaptation as essential components for pushing the compression-performance Pareto boundary for MoE-LLMs.