Global Expert-level Mixed-Precision Quantization

• GEMQ is a post-training quantization framework for MoE-LLMs that assigns expert-level bit-widths under a global memory budget using a global LP approach.
• The method integrates Hessian/Fisher-based loss proxy estimation with router fine-tuning to correct routing shifts caused by quantization.
• GEMQ employs a progressive quantization strategy and ablation findings show that securing at least one high-bit expert per layer is critical for stability.

Searching arXiv for the specified GEMQ paper and closely related mixed-precision/MoE quantization work to ground the article. Global Expert-level Mixed-precision Quantization (GEMQ) is a post-training quantization framework for mixture-of-experts LLMs (MoE-LLMs) that assigns bit-widths at the expert level under a global budget, rather than using a uniform precision or a purely layer-wise allocation. In the 2026 formulation, GEMQ is designed to address two failure modes of prior mixed-precision MoE quantization: suboptimal expert allocation caused by layer-wise importance estimation, and router shifts induced by quantization that alter token-to-expert assignments. Its core components are a global linear-programming allocation based on quantization-error analysis, router fine-tuning after quantization, and a progressive quantization procedure that refines both importance estimation and allocation across stages (Deng et al., 21 May 2026).

1. Problem setting and conceptual scope

MoE-LLMs achieve strong performance but incur substantial memory overhead because expert parameters dominate model size. Mixed-precision quantization mitigates this cost by allocating different bit-widths to different experts according to their importance, with the aim of approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization (Deng et al., 21 May 2026).

In GEMQ, the quantization unit is the expert. Let the full set of experts be denoted by $\mathcal{E}$, and let $\mathcal{B}$ be a discrete set of candidate bit-widths such as $\{1,2,3\}$. The task is to select exactly one bit-width for each expert so that the total expert-bit footprint does not exceed a global budget $B$. This makes the allocation problem intrinsically model-wide: the decision for one expert affects the remaining budget for all others.

The framework is explicitly positioned against two simpler practices. First, uniform quantization assigns the same precision to all experts and therefore cannot exploit heterogeneous expert sensitivity. Second, expert-wise mixed precision based only on layer-wise importance can overlook interactions across the model and can fail to account for routing changes caused by quantization (Deng et al., 21 May 2026). A closely related but distinct line of work, MxMoE, addresses mixed precision at the level of linear blocks inside each expert and jointly optimizes for algorithmic and system objectives, including mixed-precision GroupGEMM kernel generation (Duanmu et al., 9 May 2025). This suggests that “GEMQ” can refer both to a specific expert-level method and, more broadly, to a design space in which global allocation is the primary organizing principle.

2. Global allocation by loss-proxy minimization

GEMQ models quantizing expert $i$ to $b$ bits as adding a perturbation $\Delta w_{i,b}$ to the full-precision weights $w_i$, and approximates the expected loss increase by a second-order expansion. Under the Gauss–Newton and diagonal Fisher Information Matrix approximation, the method uses the proxy

$$\Delta\widetilde{\mathcal{L}}_{i}(b) \;=\; \mathbb{E}_{x\in\mathcal{D}}\big[\Delta z_{i,b}^T\,\mathrm{diag}(g_i^{(z)}g_i^{(z)\,T})\,\Delta z_{i,b}\big] \approx \mathbb{E}[\mathcal{L}(w_i+\Delta w_{i,b})-\mathcal{L}(w_i)] ,$$

where $\Delta z_{i,b}=\hat z_{i,b}-z_i$ is the change in the aggregated MoE layer output when expert $\mathcal{B}$0 is quantized to $\mathcal{B}$1 bits, $\mathcal{B}$2 is the gradient of the loss with respect to that output, and the expectation is taken over a small calibration set $\mathcal{B}$3 (Deng et al., 21 May 2026).

The decision variable is binary:

$\mathcal{B}$4

with $\mathcal{B}$5 indicating that expert $\mathcal{B}$6 is assigned $\mathcal{B}$7 bits. GEMQ then solves the global optimization problem

$\mathcal{B}$8

In words, the LP selects one bit-width per expert so as to minimize the proxy loss increase under the total bit budget. The method also “typically add[s] a mild regularizer—for instance, ‘each MoE layer must contain at least one expert at the highest bit-width’—to avoid degenerate low-bit assignments in ultra-low-bit regimes” (Deng et al., 21 May 2026). That constraint is not incidental: an ablation reported for 1.5 bpe shows that removing the “one high-bit per layer” condition causes the loss to blow up catastrophically (Deng et al., 21 May 2026).

GEMQ’s loss proxy differs from the simpler $\mathcal{B}$9 sensitivity models that are often used in dense-LLM mixed-precision work. The paper’s characterization is that the layer-output FIM produces a data-driven $\{1,2,3\}$0 that “effectively plays the role of $\{1,2,3\}$1 but without hand-tuned hyperparameters” (Deng et al., 21 May 2026).

3. Router adaptation and progressive quantization

A central premise of GEMQ is that quantization changes more than the expert weights: it also perturbs the routing landscape. Quantizing experts changes the router’s input-output mapping, which in turn shifts token-to-expert assignments in ways that a full-precision-trained router cannot correct (Deng et al., 21 May 2026). GEMQ therefore adds router fine-tuning after expert quantization.

The fine-tuning procedure is deliberately narrow. The starting point is a quantized weight-only model in which experts are quantized, while attention and routers are dequantized to float. All expert and attention parameters are frozen, and only router parameters are updated using a cross-entropy loss,

$\{1,2,3\}$2

on a small calibration set of $\{1,2,3\}$3 sequences. The optimizer is AdamW with lr $\{1,2,3\}$4, weight decay $\{1,2,3\}$5, batch size $\{1,2,3\}$6 per sequence, for $\{1,2,3\}$7 epoch (Deng et al., 21 May 2026).

The reported effect is twofold. First, the mean router-logit shift $\{1,2,3\}$8 realigns the router’s top-$\{1,2,3\}$9 selections to exploit higher-bit experts. Second, the fraction of changed assignments relative to the full-precision model drops dramatically. Empirically, the paper reports that router fine-tuning yields a $B$0–$B$1 drop in perplexity in the 1.5 bpe regime (Deng et al., 21 May 2026).

Because Hessian-based estimates become less accurate as $B$2 grows large, especially at ultra-low precision, GEMQ embeds allocation and router adaptation inside a progressive quantization (PQ) framework. For descending budgets $B$3, the procedure iterates:

1. collect $B$4 and $B$5 on the current model to form $B$6,
2. solve the global LP under budget $B$7,
3. quantize the full-precision model’s experts per the LP assignment via GPTQ,
4. fine-tune the routers,
5. set the current model to the quantized and fine-tuned output of that stage (Deng et al., 21 May 2026).

The practical recommendation given is explicit: for extreme low bits $B$8, use 3-stage PQ $B$9 with router fine-tuning at each stage; for moderate bits $i$0, a single stage suffices (Deng et al., 21 May 2026).

4. Empirical behavior and ablation findings

The principal reported benchmark is Mixtral-8×7B. GEMQ is evaluated on memory-accuracy trade-offs, allocation ablations, router adaptation, and the effect of progressive quantization (Deng et al., 21 May 2026).

Configuration	Reported metrics	Interpretation
FP16	87 GB model, WT2 PPL = 3.84, 0-shot MMLU = 70.97%	Full-precision reference
2.5 bpe GEMQ	16 GB, WT2 PPL = 5.03, MMLU = 65.13%	82% memory reduction
1.5 bpe GEMQ	12 GB, WT2 PPL = 7.93, MMLU = 52.00%	86% memory reduction

On allocation, GEMQ’s global LP outperforms a layer-wise baseline denoted PMQ: in the 1.5–2.5 bpe range, GEMQ lowers WT2 perplexity by approximately $i$1–$i$2 relative to PMQ (Deng et al., 21 May 2026). The same set of ablations shows that the high-bit-per-layer constraint is structurally important in ultra-low-bit settings; removing it at 1.5 bpe leads to catastrophic degradation.

On router adaptation, the paper reports a stepwise improvement at 1.5 bpe: without router adaptation, WT2 PPL is approximately $i$3; with “FP logits” alignment, approximately $i$4; with global fine-tuning, approximately $i$5 (Deng et al., 21 May 2026). On progressive quantization, the one-shot 1.5 bpe result without PQ is approximately $i$6, which improves to approximately $i$7 with PQ alone and to approximately $i$8 with PQ plus router fine-tuning (Deng et al., 21 May 2026).

The runtime profile is also itemized. For one stage on Mixtral, LP solving takes less than $i$9 on a single CPU with negligible memory; gradient computation for $b$0 takes approximately $b$1 with $b$2 peak on H100; GPTQ quantization takes approximately $b$3 with $b$4 peak; router fine-tuning takes approximately $b$5 with $b$6 peak. The total is approximately $b$7, i.e. less than $b$8 minutes per stage, and less than $b$9 overhead beyond GPTQ time (Deng et al., 21 May 2026). The same summary notes that gradient computation requires approximately $\Delta w_{i,b}$0 peak in aggregate but is spreadable over $\Delta w_{i,b}$1–$\Delta w_{i,b}$2 GPUs.

The paper further gives deployment-oriented budget guidance: $\Delta w_{i,b}$3–$\Delta w_{i,b}$4 bpe is a “safe zone” with minimal accuracy drop $\Delta w_{i,b}$5; $\Delta w_{i,b}$6 bpe yields a mild drop $\Delta w_{i,b}$7–$\Delta w_{i,b}$8; and $\Delta w_{i,b}$9 bpe is aggressive and requires full PQ plus router fine-tuning to avoid collapse (Deng et al., 21 May 2026).

5. Relation to adjacent mixed-precision quantization frameworks

GEMQ sits inside a broader literature on global mixed-precision allocation, but its exact granularity and corrective mechanisms distinguish it from nearby methods.

MxMoE formulates mixed-precision quantization for MoE as a joint algorithm-system problem over expert, linear-block, and quantization-scheme variables. Its objective combines a distortion term

$w_i$0

with an estimated execution-time term

$w_i$1

and then minimizes $w_i$2 under memory constraints. It also generates fused mixed-precision GroupGEMM kernels and reports both accuracy and speed improvements, including up to $w_i$3 speedup over full precision and up to $w_i$4 speedup over uniform quantization at equivalent accuracy with 5-bit weight-activation quantization (Duanmu et al., 9 May 2025). Relative to GEMQ, this is a finer-grained and more explicitly hardware-aware formulation.

Another method titled “GEMQ” in a different sense proposes a theoretically grounded expert-wise strategy based primarily on router-norm change during training and a MaxVar correction. Experts are sorted by ascending

$w_i$5

with outliers promoted when

$w_i$6

That work provides theoretical generalization guarantees and reports negligible bit-assignment overhead relative to calibration-heavy baselines (Chowdhury et al., 7 Apr 2026). By contrast, the 2026 GEMQ method relies on a global LP over Hessian/Fisher-derived loss proxies and router fine-tuning (Deng et al., 21 May 2026).

MoPEQ also operates at per-expert granularity, but uses Hessian-trace approximation and K-means clustering over expert importances to map experts to $w_i$7 bits, rather than solving a global LP and adapting the router (Chitty-Venkata et al., 2 Sep 2025). DyMoE, in turn, studies dynamic mixed-precision quantization for edge inference and uses importance-aware prioritization, depth-adaptive scheduling, and look-ahead prefetching to reduce TTFT and TPOT under edge memory constraints (Huang et al., 19 Mar 2026). Outside the MoE setting, HAQ uses reinforcement learning with hardware-simulator feedback for mixed-precision quantization (Wang et al., 2020), MixLLM performs global mixed precision across output features with system co-design (Zheng et al., 2024), GAMMA learns module-wise precision preferences via an augmented Lagrangian and reuses scores across arbitrary budgets (Yao et al., 18 May 2026), and WINDQuant uses PPO to assign bit-widths to fine-grained column chunks under a global storage budget (Nguyen et al., 26 May 2026). These works collectively show that “global” mixed-precision quantization has multiple realizations: LP-based, RL-based, feature-level, module-level, expert-level, and hardware-aware.

6. Interpretation, practical implications, and common misunderstandings

A common misunderstanding is to treat GEMQ as merely an expert-wise bit allocator. In the 2026 formulation, allocation is only one component. The method’s distinct claim is that model-wide expert importance must be estimated globally and then corrected after quantization by adapting routing (Deng et al., 21 May 2026). The reported ablations support this interpretation: router fine-tuning produces a large additional gain in the 1.5 bpe regime, and progressive quantization further improves stability in the same range.

Another misconception is that the LP itself is the computational bottleneck. The reported implementation profile suggests the opposite. LP solving is less than $w_i$8 on CPU with negligible memory, whereas gradient computation, GPTQ quantization, and router fine-tuning dominate total time and memory (Deng et al., 21 May 2026). A plausible implication is that GEMQ’s algorithmic complexity is concentrated in proxy estimation and post-allocation refinement rather than in the combinatorial solve.

The framework also clarifies that extreme low-bit MoE quantization is not only an allocation problem but a routing problem. Existing methods that ignore router shifts may assign bits suboptimally and then evaluate a model whose routing no longer corresponds to the original expert specialization (Deng et al., 21 May 2026). This observation is consistent with the broader MoE literature summarized here: expert activation dynamics matter for runtime in MxMoE (Duanmu et al., 9 May 2025), while expert specialization and relative importance matter for theoretical or empirical allocation in expert-ranking methods (Chowdhury et al., 7 Apr 2026).

For deployment, the most conservative reading of the published guidance is budget-stratified. The 3.0–2.5 bpe regime is described as a safe zone with minimal accuracy drop; 2.0 bpe is mild; 1.5 bpe is aggressive and should use 3-stage progressive quantization with router fine-tuning at each stage (Deng et al., 21 May 2026). This does not imply that GEMQ is restricted to those budgets. Rather, it indicates where the authors report qualitatively different stability regimes.

In summary, GEMQ is best understood as a global, expert-level post-training quantization framework for MoE-LLMs whose novelty lies in combining Hessian/Fisher-informed global bit allocation with router repair and progressive refinement. Within the current mixed-precision literature, it occupies the specific niche where expert-wise memory reduction, ultra-low-bit robustness, and routing-aware post-quantization adaptation are treated as a single coupled problem (Deng et al., 21 May 2026).