---
title: Global Expert-level Mixed-Precision Quantization
url: https://www.emergentmind.com/topics/global-expert-level-mixed-precision-quantization-gemq
type: topic
---

# Global Expert-level Mixed-Precision Quantization

Searching arXiv for the specified GEMQ paper and closely related mixed-precision/MoE quantization work to ground the article.
Global Expert-level Mixed-precision Quantization (GEMQ) is a post-training quantization framework for mixture-of-experts large language models (MoE-LLMs) that assigns bit-widths at the expert level under a global budget, rather than using a uniform precision or a purely layer-wise allocation. In the 2026 formulation, GEMQ is designed to address two failure modes of prior mixed-precision MoE quantization: suboptimal expert allocation caused by layer-wise importance estimation, and router shifts induced by quantization that alter token-to-expert assignments. Its core components are a global linear-programming allocation based on quantization-error analysis, router fine-tuning after quantization, and a progressive quantization procedure that refines both importance estimation and allocation across stages [2605.23078].

## 1. Problem setting and conceptual scope

MoE-LLMs achieve strong performance but incur substantial memory overhead because expert parameters dominate model size. Mixed-precision quantization mitigates this cost by allocating different bit-widths to different experts according to their importance, with the aim of approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization [2605.23078].

In GEMQ, the quantization unit is the expert. Let the full set of experts be denoted by $\mathcal{E}$, and let $\mathcal{B}$ be a discrete set of candidate bit-widths such as $\{1,2,3\}$. The task is to select exactly one bit-width for each expert so that the total expert-bit footprint does not exceed a global budget $B$. This makes the allocation problem intrinsically model-wide: the decision for one expert affects the remaining budget for all others.

The framework is explicitly positioned against two simpler practices. First, uniform quantization assigns the same precision to all experts and therefore cannot exploit heterogeneous expert sensitivity. Second, expert-wise mixed precision based only on layer-wise importance can overlook interactions across the model and can fail to account for routing changes caused by quantization [2605.23078]. A closely related but distinct line of work, MxMoE, addresses mixed precision at the level of linear blocks inside each expert and jointly optimizes for algorithmic and system objectives, including mixed-precision GroupGEMM kernel generation [2505.05799]. This suggests that “GEMQ” can refer both to a specific expert-level method and, more broadly, to a design space in which global allocation is the primary organizing principle.

## 2. Global allocation by loss-proxy minimization

GEMQ models quantizing expert $i$ to $b$ bits as adding a perturbation $\Delta w_{i,b}$ to the full-precision weights $w_i$, and approximates the expected loss increase by a second-order expansion. Under the Gauss–Newton and diagonal Fisher Information Matrix approximation, the method uses the proxy
$$
\Delta\widetilde{\mathcal{L}}_{i}(b)
\;=\;
\mathbb{E}_{x\in\mathcal{D}}\big[\Delta z_{i,b}^T\,\mathrm{diag}(g_i^{(z)}g_i^{(z)\,T})\,\Delta z_{i,b}\big]
\approx
\mathbb{E}[\mathcal{L}(w_i+\Delta w_{i,b})-\mathcal{L}(w_i)] ,
$$
where $\Delta z_{i,b}=\hat z_{i,b}-z_i$ is the change in the aggregated MoE layer output when expert $i$ is quantized to $b$ bits, $g_i^{(z)}=\nabla_{z_i}\mathcal{L}$ is the gradient of the loss with respect to that output, and the expectation is taken over a small calibration set $\mathcal{D}$ [2605.23078].

The decision variable is binary:
$$
x_{i,j}\in\{0,1\},
$$
with $x_{i,j}=1$ indicating that expert $i$ is assigned $j$ bits. GEMQ then solves the global optimization problem
$$
\begin{aligned}
\min_{x_{i,j}}
\quad & \sum_{i\in\mathcal{E}}\sum_{j\in\mathcal{B}}
\Delta\widetilde{\mathcal{L}}_{i,j}\,x_{i,j} \\
\mathrm{s.t.}
\quad & \sum_{i,j} j\,x_{i,j}\le B, \\
& \sum_{j\in\mathcal{B}}x_{i,j}=1 \quad \forall i, \\
& x_{i,j}\in\{0,1\}.
\end{aligned}
$$

In words, the LP selects one bit-width per expert so as to minimize the proxy loss increase under the total bit budget. The method also “typically add[s] a mild regularizer—for instance, ‘each MoE layer must contain at least one expert at the highest bit-width’—to avoid degenerate low-bit assignments in ultra-low-bit regimes” [2605.23078]. That constraint is not incidental: an ablation reported for 1.5 bpe shows that removing the “one high-bit per layer” condition causes the loss to blow up catastrophically [2605.23078].

GEMQ’s loss proxy differs from the simpler $\alpha_i 2^{-b}$ sensitivity models that are often used in dense-LLM mixed-precision work. The paper’s characterization is that the layer-output FIM produces a data-driven $\Delta\widetilde{\mathcal{L}}_{i,b}$ that “effectively plays the role of $w_iE_i(b)$ but without hand-tuned hyperparameters” [2605.23078].

## 3. Router adaptation and progressive quantization

A central premise of GEMQ is that quantization changes more than the expert weights: it also perturbs the routing landscape. Quantizing experts changes the router’s input-output mapping, which in turn shifts token-to-expert assignments in ways that a full-precision-trained router cannot correct [2605.23078]. GEMQ therefore adds router fine-tuning after expert quantization.

The fine-tuning procedure is deliberately narrow. The starting point is a quantized weight-only model in which experts are quantized, while attention and routers are dequantized to float. All expert and attention parameters are frozen, and only router parameters are updated using a cross-entropy loss,
$$
\mathcal{L}_{\mathrm{CE}}(\mathrm{router}(x), y),
$$
on a small calibration set of $|\mathcal{D}|$ sequences. The optimizer is AdamW with lr $=1\mathrm{e}{-4}$, weight decay $=1\mathrm{e}{-4}$, batch size $=1$ per sequence, for $1$ epoch [2605.23078].

The reported effect is twofold. First, the mean router-logit shift $\Delta r_i(x)$ realigns the router’s top-$K$ selections to exploit higher-bit experts. Second, the fraction of changed assignments relative to the full-precision model drops dramatically. Empirically, the paper reports that router fine-tuning yields a $10$–$20\,\%$ drop in perplexity in the 1.5 bpe regime [2605.23078].

Because Hessian-based estimates become less accurate as $\Delta w$ grows large, especially at ultra-low precision, GEMQ embeds allocation and router adaptation inside a progressive quantization (PQ) framework. For descending budgets $(B_1>B_2>\dots>B_K)$, the procedure iterates:
1. collect $\Delta z_{i,j}$ and $g_i^{(z)}$ on the current model to form $\Delta\widetilde{\mathcal{L}}_{i,j}$,
2. solve the global LP under budget $B_k$,
3. quantize the full-precision model’s experts per the LP assignment via GPTQ,
4. fine-tune the routers,
5. set the current model to the quantized and fine-tuned output of that stage [2605.23078].

The practical recommendation given is explicit: for extreme low bits $(< 2\,\mathrm{bpe})$, use 3-stage PQ $(2.5\rightarrow 2.0\rightarrow 1.5)$ with router fine-tuning at each stage; for moderate bits $(2.5\,\mathrm{bpe}+)$, a single stage suffices [2605.23078].

## 4. Empirical behavior and ablation findings

The principal reported benchmark is Mixtral-8×7B. GEMQ is evaluated on memory-accuracy trade-offs, allocation ablations, router adaptation, and the effect of progressive quantization [2605.23078].

| Configuration | Reported metrics | Interpretation |
|---|---|---|
| FP16 | 87 GB model, WT2 PPL = 3.84, 0-shot MMLU = 70.97% | Full-precision reference |
| 2.5 bpe GEMQ | 16 GB, WT2 PPL = 5.03, MMLU = 65.13% | 82% memory reduction |
| 1.5 bpe GEMQ | 12 GB, WT2 PPL = 7.93, MMLU = 52.00% | 86% memory reduction |

On allocation, GEMQ’s global LP outperforms a layer-wise baseline denoted PMQ: in the 1.5–2.5 bpe range, GEMQ lowers WT2 perplexity by approximately $0.3$–$1.0$ relative to PMQ [2605.23078]. The same set of ablations shows that the high-bit-per-layer constraint is structurally important in ultra-low-bit settings; removing it at 1.5 bpe leads to catastrophic degradation.

On router adaptation, the paper reports a stepwise improvement at 1.5 bpe: without router adaptation, WT2 PPL is approximately $9.66$; with “FP logits” alignment, approximately $9.29$; with global fine-tuning, approximately $7.69$ [2605.23078]. On progressive quantization, the one-shot 1.5 bpe result without PQ is approximately $11.72$, which improves to approximately $11.30$ with PQ alone and to approximately $8.71$ with PQ plus router fine-tuning [2605.23078].

The runtime profile is also itemized. For one stage on Mixtral, LP solving takes less than $0.1\,\mathrm{s}$ on a single CPU with negligible memory; gradient computation for $\Delta\widetilde{\mathcal{L}}$ takes approximately $400\,\mathrm{s}$ with $30\,\mathrm{GB}$ peak on H100; GPTQ quantization takes approximately $1700\,\mathrm{s}$ with $21\,\mathrm{GB}$ peak; router fine-tuning takes approximately $60\,\mathrm{s}$ with $136\,\mathrm{GB}$ peak. The total is approximately $2300\,\mathrm{s}$, i.e. less than $40$ minutes per stage, and less than $5\,\%$ overhead beyond GPTQ time [2605.23078]. The same summary notes that gradient computation requires approximately $190\,\mathrm{GB}$ peak in aggregate but is spreadable over $2$–$4$ GPUs.

The paper further gives deployment-oriented budget guidance: $3.0$–$2.5$ bpe is a “safe zone” with minimal accuracy drop $(< 5\,\%)$; $2.0$ bpe yields a mild drop $(5$–$10\,\%)$; and $1.5$ bpe is aggressive and requires full PQ plus router fine-tuning to avoid collapse [2605.23078].

## 5. Relation to adjacent mixed-precision quantization frameworks

GEMQ sits inside a broader literature on global mixed-precision allocation, but its exact granularity and corrective mechanisms distinguish it from nearby methods.

MxMoE formulates mixed-precision quantization for MoE as a joint algorithm-system problem over expert, linear-block, and quantization-scheme variables. Its objective combines a distortion term
$$
L = \sum_{j=1}^E \sum_{i=1}^N \sum_{k\in S} \Delta_{i,j,k}\,x_{i,j,k}
$$
with an estimated execution-time term
$$
T \approx (1/P)\cdot \sum_{j,i,k,t} c_{i,j,k,t}\,y_{i,j,k,t}\,x_{i,j,k},
$$
and then minimizes $L^r \cdot T^{1-r}$ under memory constraints. It also generates fused mixed-precision GroupGEMM kernels and reports both accuracy and speed improvements, including up to $3.4\times$ speedup over full precision and up to $29.4\%$ speedup over uniform quantization at equivalent accuracy with 5-bit weight-activation quantization [2505.05799]. Relative to GEMQ, this is a finer-grained and more explicitly hardware-aware formulation.

Another method titled “GEMQ” in a different sense proposes a theoretically grounded expert-wise strategy based primarily on router-norm change during training and a MaxVar correction. Experts are sorted by ascending
$$
\Delta_s := \|w_s^{(T)}\|_2 - \|w_s^{(0)}\|_2,
$$
with outliers promoted when
$$
\mathrm{MaxVar}_s \ge \zeta \cdot \mathrm{MaxVar}_{s'}
\quad (\zeta=3).
$$
That work provides theoretical generalization guarantees and reports negligible bit-assignment overhead relative to calibration-heavy baselines [2604.06515]. By contrast, the 2026 GEMQ method relies on a global LP over Hessian/Fisher-derived loss proxies and router fine-tuning [2605.23078].

MoPEQ also operates at per-expert granularity, but uses Hessian-trace approximation and K-means clustering over expert importances to map experts to $\{2,3,4\}$ bits, rather than solving a global LP and adapting the router [2509.02512]. DyMoE, in turn, studies dynamic mixed-precision quantization for edge inference and uses importance-aware prioritization, depth-adaptive scheduling, and look-ahead prefetching to reduce TTFT and TPOT under edge memory constraints [2603.19172]. Outside the MoE setting, HAQ uses reinforcement learning with hardware-simulator feedback for mixed-precision quantization [2008.04878], MixLLM performs global mixed precision across output features with system co-design [2412.14590], GAMMA learns module-wise precision preferences via an augmented Lagrangian and reuses scores across arbitrary budgets [2605.18475], and WINDQuant uses PPO to assign bit-widths to fine-grained column chunks under a global storage budget [2605.26660]. These works collectively show that “global” mixed-precision quantization has multiple realizations: LP-based, RL-based, feature-level, module-level, expert-level, and hardware-aware.

## 6. Interpretation, practical implications, and common misunderstandings

A common misunderstanding is to treat GEMQ as merely an expert-wise bit allocator. In the 2026 formulation, allocation is only one component. The method’s distinct claim is that model-wide expert importance must be estimated globally and then corrected after quantization by adapting routing [2605.23078]. The reported ablations support this interpretation: router fine-tuning produces a large additional gain in the 1.5 bpe regime, and progressive quantization further improves stability in the same range.

Another misconception is that the LP itself is the computational bottleneck. The reported implementation profile suggests the opposite. LP solving is less than $0.1\,\mathrm{s}$ on CPU with negligible memory, whereas gradient computation, GPTQ quantization, and router fine-tuning dominate total time and memory [2605.23078]. A plausible implication is that GEMQ’s algorithmic complexity is concentrated in proxy estimation and post-allocation refinement rather than in the combinatorial solve.

The framework also clarifies that extreme low-bit MoE quantization is not only an allocation problem but a routing problem. Existing methods that ignore router shifts may assign bits suboptimally and then evaluate a model whose routing no longer corresponds to the original expert specialization [2605.23078]. This observation is consistent with the broader MoE literature summarized here: expert activation dynamics matter for runtime in MxMoE [2505.05799], while expert specialization and relative importance matter for theoretical or empirical allocation in expert-ranking methods [2604.06515].

For deployment, the most conservative reading of the published guidance is budget-stratified. The 3.0–2.5 bpe regime is described as a safe zone with minimal accuracy drop; 2.0 bpe is mild; 1.5 bpe is aggressive and should use 3-stage progressive quantization with router fine-tuning at each stage [2605.23078]. This does not imply that GEMQ is restricted to those budgets. Rather, it indicates where the authors report qualitatively different stability regimes.

In summary, GEMQ is best understood as a global, expert-level post-training quantization framework for MoE-LLMs whose novelty lies in combining Hessian/Fisher-informed global bit allocation with router repair and progressive refinement. Within the current mixed-precision literature, it occupies the specific niche where expert-wise memory reduction, ultra-low-bit robustness, and routing-aware post-quantization adaptation are treated as a single coupled problem [2605.23078].

Source: https://www.emergentmind.com/topics/global-expert-level-mixed-precision-quantization-gemq