MoPEQ: Mixed Precision Quantized Experts

• MoPEQ is a post-training quantization method for MoE models that assigns expert-specific bit widths (2, 3, or 4) based on Hessian-trace sensitivity.
• It clusters experts by importance scores rather than relying solely on activation frequency, ensuring a structured, data-free bit allocation.
• Experiments show MoPEQ achieves up to 1.5× model size reduction with less than 5% accuracy loss across diverse vision-language benchmarks.

Searching arXiv for the specified paper and closely related work to ground the article in current literature. MoPEQ, short for Mixture of Mixed Precision Quantized Experts, is a post-training quantization algorithm for Mixture-of-Experts (MoE) large language and vision models that assigns an expert-specific bit width from $\{2,3,4\}$ in order to balance memory reduction and accuracy retention (Chitty-Venkata et al., 2 Sep 2025). It is introduced for the deployment setting in which MoE architectures achieve high capacity by sparsely activating a small set of expert subnetworks per token, yet still incur large memory and compute overhead. The central design choice in MoPEQ is to rank expert importance using Hessian-trace sensitivity rather than relying primarily on activation frequency, then cluster experts by relative importance and map those clusters to distinct precisions. The method supports both layer-wise and global precision assignment and is evaluated on state-of-the-art vision-language MoE models using VLMEvalKit benchmark tasks (Chitty-Venkata et al., 2 Sep 2025).

1. Problem setting and design objective

MoE models activate only a subset of experts for each token, but deployment remains difficult because the full parameter set must still be stored, and the expert matrices dominate memory usage in large systems (Chitty-Venkata et al., 2 Sep 2025). In this setting, uniform low-bit quantization of all experts may degrade accuracy, while full-precision wastes resources on rarely used experts. MoPEQ is formulated specifically to address this trade-off by assigning 2, 3, or 4 bits to each expert.

The method is positioned as mixed-precision quantization at per-expert granularity rather than conventional uniform expert quantization. Its stated objective is to reduce model size while maintaining competitive validation accuracy on downstream tasks. Two organizational choices are central. First, expert importance is estimated with a data-free Hessian-trace approximation. Second, experts with similar importance are clustered so that groups of experts, rather than each expert independently, receive a shared bit width. This suggests that MoPEQ treats bit allocation as a structured approximation problem rather than as a purely frequency-driven routing heuristic.

A common misconception addressed by the reported analysis is that activation frequency alone is an adequate proxy for expert importance. The experiments explicitly distinguish activation frequency from Hessian sensitivity and report that, in well-balanced models, activation frequency is uniform and fails to distinguish experts, whereas Hessian sensitivity yields better bit allocations (Chitty-Venkata et al., 2 Sep 2025).

2. Post-training quantization workflow

MoPEQ is defined as a wrapper around a post-training quantization engine. The base quantizer used in the reported experiments is SignRound in the AutoRound framework, which for each weight tensor $W$ solves

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$

while minimizing $\operatorname{mse}(WX, \tilde W X)$ (Chitty-Venkata et al., 2 Sep 2025).

Within that PTQ substrate, MoPEQ follows a three-stage workflow. It first computes an importance score $I_i$ for each expert $E_i$. It then chooses a bit width $b_i \in \{2,3,4\}$ for that expert. Finally, it quantizes the expert weights to $b_i$ using the chosen PTQ engine. Non-MoE layers are uniformly quantized to 4 bits in the reported setup.

The high-level procedure is given as:

1. For each expert $E_i$, compute Hessian trace $H_i$ via Hutchinson’s estimator.
2. Optionally compute activation frequency $W$0 on a small calibration set.
3. Compute the expert importance score $W$1.
4. Collect $W$2 and run $W$3-means with $W$4 clusters.
5. Compute cluster means $W$5 and sort clusters by descending mean importance.
6. Assign precision levels so that the highest-importance cluster receives 4 bits, the next receives 3 bits, and the last receives 2 bits.
7. Quantize each expert with the resulting bit assignment (Chitty-Venkata et al., 2 Sep 2025).

Although the paper presents an optimization statement, it also emphasizes that the actual implementation uses clustering as a heuristic surrogate. This is significant because it frames MoPEQ as a practical approximation to an otherwise discrete combinatorial allocation problem.

3. Hessian-trace sensitivity and expert importance

The core technical mechanism in MoPEQ is expert sensitivity estimation via Hessian trace approximation. Let $W$6 denote the weights of an expert and define the Hessian

$W$7

where $W$8 is a proxy loss (Chitty-Venkata et al., 2 Sep 2025). Since direct computation of $W$9 is infeasible for large $$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$0, the method uses Hutchinson’s estimator:

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$1

with random vectors $$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$2 or Rademacher.

In practice, the surrogate loss is chosen as the Frobenius norm of each FC-layer weight,

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$3

and the computation proceeds through

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$4

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$5

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$6

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$7

For each expert $$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$8, the final sensitivity aggregates the traces of the gating, up-projection, and down-projection FC layers:

$$\tilde W = s \cdot \operatorname{clip}(W/s + zp + V, n, m),$$9

This construction makes the importance score second-order and data-free. The paper contrasts this with activation frequency, which depends on routing statistics from a calibration set. A plausible implication is that MoPEQ is designed to remain informative even when routing frequencies are nearly uniform and therefore weakly discriminative.

An optional hybrid importance score combines normalized activation frequency and normalized Hessian sensitivity:

$\operatorname{mse}(WX, \tilde W X)$0

The inclusion of this hybrid form indicates that MoPEQ does not exclude frequency information outright; rather, it treats frequency as supplementary when load imbalance makes it informative.

4. Clustering and precision allocation

Once importance values are computed, MoPEQ collects them into $\operatorname{mse}(WX, \tilde W X)$1 and applies $\operatorname{mse}(WX, \tilde W X)$2-means with $\operatorname{mse}(WX, \tilde W X)$3 to partition experts into clusters $\operatorname{mse}(WX, \tilde W X)$4 (Chitty-Venkata et al., 2 Sep 2025). For each cluster, it computes the mean

$\operatorname{mse}(WX, \tilde W X)$5

sorts clusters by descending $\operatorname{mse}(WX, \tilde W X)$6, and maps them to precision levels in descending order of importance: 4 bits, then 3 bits, then 2 bits.

The paper also gives an optimization statement for the underlying allocation problem:

$\operatorname{mse}(WX, \tilde W X)$7

It further describes an equivalent formulation as minimizing total memory plus a weighted error approximation over the discrete precision assignments. The optimization is explicitly labeled intractable in its direct form, and clustering is used as a heuristic surrogate: similar-importance experts share the same bit width, and more important clusters receive higher precision.

MoPEQ supports both layer-wise and model-wise precision assignment. The distinction matters empirically because the reported results show that, across 63 model-wise scenarios, Hessian-MPQ outperforms layer-wise in 63 vs 42 cases (Chitty-Venkata et al., 2 Sep 2025). This suggests that global allocation captures cross-layer heterogeneity in expert importance more effectively than independent per-layer decisions.

5. Experimental configuration

The experimental study evaluates MoPEQ on both MolmoE and DeepSeek-VL2 variants using VLMEvalKit tasks and measures validation accuracy together with total model size in GB (Chitty-Venkata et al., 2 Sep 2025). The reported models are:

Model	Configuration	Active experts
MolmoE-1B	7.2 B parameters, 16 layers, 64 experts	8 active per token
DeepSeek-VL2-tiny	3 B, 12 layers, 64 experts	6 active
DeepSeek-VL2-small	16 B, 27 layers, 64 experts	6 active
DeepSeek-VL2-base	27 B, 30 layers, 72 experts	6 active

The PTQ engine is AutoRound + SignRound, with expert bit-width search space $\operatorname{mse}(WX, \tilde W X)$8 and uniform 4-bit quantization for non-MoE layers. The tasks drawn from VLMEvalKit are MME (reasoning, perception), TextVQA, AI2D, DocVQA, MMMU, InfoVQA, RealWorldQA, ScienceQA, and BLINK. The evaluation metrics are validation accuracy for each task and total model size.

This setup is tailored to vision-language MoE models rather than text-only MoE systems. That focus is important because the benchmarks span multimodal reasoning, perception, document understanding, and general visual question answering, thereby testing whether expert-specific precision changes preserve utility across heterogeneous VLM workloads.

6. Quantitative findings, interpretation, and limitations

Under model-wise Hessian sensitivity allocation, MoPEQ reduces memory relative to uniform 4-bit baselines across all reported models (Chitty-Venkata et al., 2 Sep 2025). The reported size changes are:

Model	Uniform 4-bit	MoPEQ
MolmoE-1B	4.08 GB	3.41 GB
DeepSeek-VL2-tiny	2.64 GB	2.35 GB
DeepSeek-VL2-small	8.73 GB	7.14 GB
DeepSeek-VL2-base	14.35 GB	10.49 GB

For MolmoE-1B, the hybrid sensitivity-plus-frequency variant further reduces size to 2.79 GB, corresponding to a larger reduction than Hessian-only allocation. On the accuracy side, the reported examples show heterogeneous behavior. For MolmoE-1B on MME-Perception, uniform 4-bit gives 1300.09, Hessian MPQ gives 1338.09, and the hybrid variant gives 1245.82. For DeepSeek-VL2-base on MME-Reasoning, uniform 4-bit gives 585.71 and Hessian MPQ gives 576.43.

The paper summarizes the broader trade-off by stating that mixed-precision expert quantization enables up to $\operatorname{mse}(WX, \tilde W X)$9 model size reduction with $I_i$0 average accuracy loss, making MoE models more feasible for memory-constrained inference (Chitty-Venkata et al., 2 Sep 2025). It also reports that, in the well-balanced DeepSeek variants, activation frequency is uniform and Hessian sensitivity yields better bit allocations and higher accuracy in 7/9 tasks for the VL2-base model. Conversely, the hybrid combination reduces memory further under load-imbalance cases such as MolmoE-1B.

These findings support two interpretive conclusions stated in the paper. First, Hessian trace-based sensitivity provides a data-free, second-order measure of expert importance superior to simple activation counts, especially in balanced MoE. Second, precision assignment benefits from model-wide reasoning over expert importance rather than purely layer-local ranking.

The reported limitations are operational rather than conceptual. Current frameworks lack hardware support for per-expert mixed bit-width, and the impact on latency has not yet been measured. Future work proposed in the paper includes integrating mixed-precision kernels into inference engines such as vLLM, extending clustering to dynamic per-batch bit assignments, and exploring finer bit granularity such as 1–8 bits or per-channel quantization. These directions indicate that the present contribution is primarily an allocation and quantization methodology; deployment efficiency in production inference stacks remains an open systems problem.