MoE-based Parallel Quantization Codebook

• MPQ is a design pattern combining MoE-based expert routing with adaptive quantization to address heterogeneity using parallel processing.
• It integrates methods like Taylor expansion surrogates, mixed-precision scheduling, and dynamic codebook generation to optimize low-bit training and inference.
• Empirical comparisons from MEC-Quant, MxMoE, and RQ-MoE highlight improved accuracy, reduced computational load, and enhanced throughput in quantized models.

Searching arXiv for the cited papers to ground the article in current literature. MoE-based Parallel Quantization Codebook (MPQ) is best understood as a non-canonical umbrella for several recent ways of using mixture-of-experts structure to make quantization or codebook construction adaptive, heterogeneous, and parallelizable. In the current literature, the term does not denote a single standardized module. Instead, closely related mechanisms appear in at least three forms: a MoE-based reformulation of an entropy surrogate for extremely low-bit quantization-aware training in MEC-Quant (Pang et al., 19 Sep 2025), a mixed-precision assignment-and-kernel framework for MoE inference in MxMoE (Duanmu et al., 9 May 2025), and an explicitly dynamic, input-dependent codebook generator with parallel decoding in RQ-MoE (Zhong et al., 14 May 2026). By contrast, Tiny-QMoE is conceptually adjacent but does not introduce a distinct MPQ mechanism (Cashman et al., 26 Sep 2025).

1. Conceptual scope and usage

The literature does not present MPQ as a universally agreed formal term. A more precise characterization is that it names a family resemblance across methods that combine quantization with MoE-style specialization and some form of parallel structure. In that family, “parallel” can refer to parallel surrogate objectives, parallel mixed-precision execution, or parallel decoding; “codebook” can refer either to literal vector-quantization codebooks or to codebook-like surrogate families; and “MoE-based” can refer either to soft routing among experts or to expert-conditioned transformations.

Work	Core mechanism	Relation to MPQ
MEC-Quant	MoE mixture of Taylor-expansion experts for coding-length estimation	Parallel surrogate codebook analogue
MxMoE	Linear-block-level mixed precision plus mixed-precision GroupGEMM	Parallel quantization/execution co-design
RQ-MoE	Dynamic input-dependent codebooks via two-level MoE	Most literal MPQ instance
Tiny-QMoE	8-bit quantization plus dictionary compression	Not a distinct MPQ method

This usage matters because superficially similar phrases can describe materially different objects. In MEC-Quant, the MoE component is not a discrete vector-quantization codebook; it is a soft mixture of entropy estimators (Pang et al., 19 Sep 2025). In MxMoE, the primary object is not a codebook at all, but a mixed-precision allocation and execution framework for MoE linear blocks (Duanmu et al., 9 May 2025). In RQ-MoE, by contrast, the codebook is literal and dynamic: codewords are constructed on demand from a base codebook plus expert-conditioned deformations (Zhong et al., 14 May 2026).

2. Shared design principles across MPQ-style methods

Despite their differences, these methods are organized around a common diagnosis: uniform quantization is too coarse for structurally heterogeneous models or data. MEC-Quant attributes the failure mode of extremely low-bit quantization-aware training to representation bias, including feature collapse or low-rank tendencies, accumulated backbone error, and poor generalization at low bit-widths (Pang et al., 19 Sep 2025). MxMoE identifies two heterogeneous factors inside MoE LLMs: linear blocks have different quantization sensitivities, and expert activation frequencies vary by more than $10\times$ (Duanmu et al., 9 May 2025). RQ-MoE argues that static codebooks assume homogeneous local geometry across the embedding space, which is unrealistic for heterogeneous real-world data manifolds (Zhong et al., 14 May 2026).

A second shared principle is conditional specialization. MEC-Quant uses multiple Taylor expansion points, with a gating network selecting among experts to better approximate a long-tailed feature distribution. MxMoE allocates precision at the level of individual linear blocks such as Gate_proj, Up_proj, and Down_proj, rather than treating the expert as the atomic unit. RQ-MoE uses quantization indices as implicit routing signals and then applies a second-level weighted MoE to deform a base codebook into an input-specific codebook.

A third principle is that parallelism is introduced to remove a bottleneck. In MEC-Quant, the bottleneck is the log-determinant computation inside the coding-length objective, and MoE reduces reliance on a high-order single Taylor expansion. In MxMoE, the bottleneck is inefficient execution of heterogeneous expert GEMMs, addressed through mixed-precision GroupGEMM kernels. In RQ-MoE, the bottleneck is the strictly sequential dependency of dynamic codebook generation in QINCo-like decoding, addressed through dual-stream quantization and precomputable instructions (Pang et al., 19 Sep 2025, Duanmu et al., 9 May 2025, Zhong et al., 14 May 2026).

3. MEC-Quant: MPQ as a parallel entropy-surrogate family

MEC-Quant introduces Maximum Entropy Coding Quantization for extremely low-bit quantization-aware training and motivates it by the claim that quantization inevitably introduces biases into learned representations, especially in the extremely low-bit setting (Pang et al., 19 Sep 2025). The method explicitly optimizes the information structure of the learned representation by maximizing entropy through a tractable surrogate based on minimal coding length in lossy coding.

For learned representation $Z$, the coding length is defined as

$$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$

with

$$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$

Because direct computation of the log-determinant is expensive, the paper uses a Taylor expansion surrogate: $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$ under the convergence condition

$\| \lambda Z^\top Z \|_2 < 1.$

The MoE reformulation is introduced because a single expansion point around zero is insufficient for long-tailed distributions. Defining

$$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$

the paper expands $f(X)$ around multiple points $a_i \cdot A$, where $A$ is the identity matrix. The $Z$0-th expert loss is

$Z$1

Routing weights are produced by

$Z$2

and the final MEC objective is

$Z$3

This is the clearest reason MEC-Quant is only an analogue of MPQ rather than a literal codebook method. Each expert corresponds to an approximation regime or expansion center for the same entropy objective, so the resulting structure behaves like a parallel surrogate codebook family rather than a set of discrete learned codewords. The overall training objective is

$Z$4

with $Z$5 in supervised classification and $Z$6 in distillation or label-free settings.

The method is trained end-to-end by feeding quantized inputs through the model, obtaining final output $Z$7 and backbone output $Z$8, computing task loss plus MEC loss, and backpropagating through both model parameters and quantization parameters. A warm-up schedule gradually increases $Z$9, keeping the MEC regularizer small early in training and stronger later. In ablations, plain MEC with 4th-order Taylor and MEC+MoE with only 2nd-order Taylor achieve similar or better accuracy, supporting the claim that MoE reduces the required Taylor order and lowers computational load. On CIFAR-10 W2A2, the reported figures are 89.85 for MEC 4th order, 89.67 for MEC 2nd order, and 89.98 for MEC+MoE with 2nd order. The same paper reports a W2A2 Hessian maximum eigenvalue of 155.73 for MEC-Quant versus 520.24 for LSQ, which it interprets as evidence of flatter minima and better generalization (Pang et al., 19 Sep 2025).

4. MxMoE: MPQ as mixed-precision assignment and parallel execution

MxMoE addresses a different problem: mixed-precision quantization for MoE models under simultaneous accuracy and system-performance constraints (Duanmu et al., 9 May 2025). Its central claim is that quantization for MoE deployment cannot be solved solely as an algorithmic compression problem, because experts differ not only in quantization sensitivity but also in activation frequency and therefore in hardware behavior.

The framework searches over precision assignments for each linear block and tile configurations for execution, while respecting a memory budget. Its global objective is formulated as

$$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$0

which is simplified to a per-block form

$$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$1

Here $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$2 is quantization or output loss, $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$3 is execution time, $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$4 prioritizes accuracy, and $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$5 prioritizes speed.

For an MoE block with $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$6 experts, $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$7 linear blocks per expert, and quantization scheme set $$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$8, the quantization loss is modeled as

$$L = \left( \frac{m+d}{2} \right) \log \det \left( I_m + \frac{d}{m\epsilon^2} Z^\top Z \right),$$9

where $$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$0 indicates scheme choice and $$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$1 is Euclidean output deviation,

$$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$2

Runtime is modeled through profiled tile costs: $$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$3 The optimization is posed as an ILP-like problem with one precision per block, one tile configuration per precision choice, and a total memory constraint.

What makes MxMoE relevant to MPQ is not a learned codebook in the vector-quantization sense, but a form of parallel quantization co-design: heterogeneous precisions are assigned at linear-block granularity and then made executable in parallel through automatically generated mixed-precision GroupGEMM kernels. The system contribution includes CTA-level CUDA micro-kernel specialization, compatibility of different micro-kernels at the CTA level through shared warp-count and shared-memory constraints, k-dimension tiling (slice-K), and a greedy precision-aware tile scheduler that prioritizes heavier tiles. This architecture is specifically designed to support parallel expert GEMMs with different precisions.

The reported results are explicitly dual: accuracy and throughput. The paper states that MxMoE achieves 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit, up to $$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$4 speedup over full precision, and up to 29.4% speedup over uniform quantization at equivalent accuracy with 5-bit weight-activation quantization. It also reports 1.6–2.7× throughput increase for memory-bound workloads and 3–3.4× throughput increase for compute-bound workloads. This suggests that, in the MoE inference setting, an MPQ-style design may be less about learned codewords than about matching precision heterogeneity to execution heterogeneity (Duanmu et al., 9 May 2025).

5. RQ-MoE: dynamic codebook generation and parallel decoding

RQ-MoE is the most literal instance of an MoE-based parallel quantization codebook in the surveyed literature (Zhong et al., 14 May 2026). It begins from residual quantization, where an embedding $$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$5 is represented by multiple codebooks $$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$6. Standard residual quantization selects

$$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$7

then updates

$$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$8

RQ-MoE identifies the limitation of this baseline as the use of static codebooks that assume homogeneous local geometry.

The method introduces a two-level MoE with dual-stream quantization. Each codebook entry is augmented into a hyper-dimensional vector

$$\mu = \frac{m+d}{2}, \qquad \lambda = \frac{d}{m\epsilon^2}.$$9

where $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$0 is the reconstruction component and $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$1 is an expert component storing local manifold information. The first-level MoE is an implicit router: nearest-neighbor quantization itself selects both the reconstruction codeword and the expert component. Because routing is embedded in the chosen quantization index, the paper states that there is zero additional bit overhead.

The instruction stream maintains a latent instruction vector $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$2, initialized as $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$3, and updated by

$$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$4

This stream is distinct from the reconstruction itself. Given $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$5, the quantization stream constructs a dynamic codebook from the base codebook $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$6: $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$7 The transformation $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$8 is implemented as a weighted MoE: $$L = \mathrm{Tr}\left( \mu \sum_{k=1}^{\infty} \frac{(-1)^{k+1}}{k} \left(\lambda Z^\top Z \right)^k \right),$$9

$\| \lambda Z^\top Z \|_2 < 1.$0

$\| \lambda Z^\top Z \|_2 < 1.$1

The second-level experts are residual MLPs: $\| \lambda Z^\top Z \|_2 < 1.$2

Quantization then proceeds against the dynamic codebook,

$\| \lambda Z^\top Z \|_2 < 1.$3

with residual updates

$\| \lambda Z^\top Z \|_2 < 1.$4

The key structural advance is dual-stream quantization. Because instruction vectors $\| \lambda Z^\top Z \|_2 < 1.$5 can be computed by simple lookups and additions using indices and stored expert components, they can be prepared independently of the reconstruction path. Once instructions are available, the specialized codebooks $\| \lambda Z^\top Z \|_2 < 1.$6 can be instantiated independently, and the final reconstruction becomes

$\| \lambda Z^\top Z \|_2 < 1.$7

This is the paper’s basis for inter-step parallelism and intra-step parallelism.

Theoretical results formalize RQ-MoE as a generalization of both standard residual quantization and QINCo. If $\| \lambda Z^\top Z \|_2 < 1.$8 or $\| \lambda Z^\top Z \|_2 < 1.$9, then $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$0 and the method reduces to memoryless residual quantization. If $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$1, the instruction stream collapses into the partial reconstruction state, recovering the coupled-state behavior of QINCo. The paper also gives a guideline for expert dimensionality: $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$2 is sufficient, and empirical performance saturates once $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$3 reaches $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$4. For training, it uses a Normalized Residual Loss,

$$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$5

Empirically, RQ-MoE is reported to match or slightly exceed QINCo on Deep1M, BigANN1M, FB-ssnpp1M, and Contriever1M under 8-byte and 16-byte budgets, while providing 6x–14x faster decoding than prior vector quantization methods. On an NVIDIA A800 GPU, when $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$6 at $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$7, decoding drops to $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$8, which is about 6.6× faster than QINCo; with $$f(X) = \log(I_m + X), \qquad X = \frac{d}{m\varepsilon^2} Z^\top Z,$$9, the speedup grows to 14.8× in some configurations (Zhong et al., 14 May 2026).

6. Boundary cases, non-examples, and recurrent misconceptions

A recurrent misconception is that any quantization method associated with MoE models is automatically an MPQ method. Tiny-QMoE is a clear counterexample. It is described as a mobile/edge-oriented quantization plus dictionary-compression system for LLaMA 3.2, not as a faithful implementation of the original QMoE method and not as a distinct MoE-based Parallel Quantization Codebook (Cashman et al., 26 Sep 2025).

Its pipeline has two stages: 8-bit layer-wise or parameter-wise affine quantization, followed by compression of quantized byte sequences using a frequency-based table over fixed-length subsequences, with default sequence_length = 4 and top_k = 2**16 - 1. Known 4-byte sequences are replaced by short codewords; unknown sequences emit a special escape marker 0xFFFF followed by the raw sequence. The paper explicitly states that it does not describe expert-wise compression, routing-aware quantization, parallel codebook learning across experts, or any actual MoE-specific handling. The only codebook-like object is a global compression_table over repeated quantized sequences (Cashman et al., 26 Sep 2025).

A second misconception is that MPQ must always mean vector quantization with learned centroids. MEC-Quant contradicts that reading: its “parallel codebook” analogue is a soft mixture of entropy estimators built from Taylor expansions around different expansion points (Pang et al., 19 Sep 2025). A third misconception is that the value of MoE-based quantization lies only in accuracy retention. MxMoE demonstrates that, for deployed MoE inference, the quantization problem is also a hardware scheduling and kernel-generation problem (Duanmu et al., 9 May 2025). A fourth misconception is that dynamic codebooks necessarily imply sequential decoding. RQ-MoE is specifically designed to avoid that dependency by decoupling instruction from reconstruction (Zhong et al., 14 May 2026).

7. Significance and research trajectory

Taken together, these works suggest that MPQ is better treated as a design pattern than as a single architecture. The pattern appears whenever MoE structure is used to allocate representational capacity non-uniformly across data regions, linear blocks, or approximation regimes, and parallelism is used to remove a computational bottleneck. In MEC-Quant, the bottleneck is scalable entropy-aware regularization for extremely low-bit QAT; in MxMoE, it is heterogeneous-precision execution of sparse expert workloads; in RQ-MoE, it is the sequential decoding bottleneck of dynamic vector quantization (Pang et al., 19 Sep 2025, Duanmu et al., 9 May 2025, Zhong et al., 14 May 2026).

The main technical significance lies in the way these methods reframe “codebook” and “routing.” RQ-MoE uses index reuse so that the chosen quantization index simultaneously selects a reconstruction codeword and an expert signal, eliminating explicit routing overhead. MEC-Quant uses routing weights from a gating network to combine approximation experts for a coding-length objective, effectively turning entropy estimation into a region-wise piecewise surrogate. MxMoE routes precision budgets and execution resources according to quantization sensitivity and expert activation dynamics. This suggests a broader interpretation in which MPQ encompasses conditional quantization mechanisms that are both distribution-aware and execution-aware.

At the same time, the literature marks clear boundaries. MEC-Quant’s MoE component is not a discrete product codebook or a vector-quantization routing architecture. MxMoE is not primarily a codebook-learning method. Tiny-QMoE is not an MoE-aware MPQ method despite its name. The most literal MPQ formulation among the surveyed works is therefore RQ-MoE, while MEC-Quant and MxMoE are best read as adjacent realizations of the same organizing idea: replacing monolithic quantization with parallel, specialized, and conditionally activated quantization mechanisms (Zhong et al., 14 May 2026, Pang et al., 19 Sep 2025, Duanmu et al., 9 May 2025).