CodeQuant: 4-Bit MoE Quantization

• CodeQuant is a unified post-training quantization framework that employs an orthogonal rotation to smooth activation outliers in MoE models.
• It uses row-wise clustering and centroid fine-tuning to absorb weight outliers, allowing for accurate 4-bit precision deployment.
• Dedicated LUT-based kernels on GPU and CPU boost inference efficiency, achieving significant speedups with minimal loss of accuracy.

Searching arXiv for the specified paper and closely related quantization baselines for citation support. [arXiv search] Query: (Yin et al., 12 Apr 2026) CodeQuant is a post-training compression framework specifically designed to enable accurate, low-precision inference of Mixture-of-Experts (MoE) LLMs. It addresses the two distinct sources of large quantization error in MoE under post-training quantization (PTQ)—activation outliers and weight outliers—by combining learnable orthogonal rotation for activation smoothing with row-wise clustering and centroid fine-tuning for weights. The method is formulated as a unified quantization-and-clustering scheme, and is coupled with dedicated GPU and CPU kernels based on lookup tables (LUTs), with the stated goal of preserving accuracy at very low bit-widths such as 4 bits while improving inference efficiency (Yin et al., 12 Apr 2026).

1. Problem setting and motivation

Low-precision deployment of large models is constrained by outliers, which the paper characterizes as a fundamental bottleneck in preserving accuracy, particularly within MoE architectures that are increasingly central to large-scale language modeling (Yin et al., 12 Apr 2026). Under PTQ, activation outliers dramatically expand the dynamic range of intermediate activations, so uniform quantization steps become too coarse. Weight outliers create an analogous problem on the weight side, magnifying errors in matrix-vector products.

The work is positioned against recent rotation-based smoothing techniques. According to the paper, such methods alleviate the problem by redistributing outlier magnitudes, but residual errors remain and continue to impede reliable low-precision deployment. CodeQuant is therefore introduced not as a replacement for smoothing alone, but as a unified response to both activation-side and weight-side error sources.

A central implication of this formulation is that MoE quantization is treated as a two-bottleneck problem rather than a single-bottleneck problem. This suggests that improving activation quantization without explicitly handling residual weight outliers is insufficient for reliable A4W4 deployment in MoE settings. The method’s design reflects that diagnosis directly.

2. Unified four-stage pipeline

CodeQuant addresses both outlier classes simultaneously through a four-stage offline calibration pipeline (Yin et al., 12 Apr 2026).

First, Activation-Oriented Outlier Smoothing (AOS) learns a single orthogonal rotation $R$ that, when applied to every layer’s activation $X$, reduces quantization error. Second, Permutation-Invariant Outlier Grouping (POG) optionally permutes the columns of the rotated weight matrix for block-wise clustering so as to form more “clusterable” groups. Third, Adaptive Weight Clustering with Centroid Finetuning (ACCF) performs row-wise $K$-centroid clustering of the rotated weights $W_R = R^\top W$, with centroid fine-tuning under a loss designed to preserve both MLP outputs and MoE router assignments. Fourth, LUT-Based Deployment implements 4-bit activations multiplied by 4-bit codebook weights through a shared-memory lookup table on GPU or a table-lookup GEMM on CPU.

The pipeline is explicitly offline. The rotation $R$ is learned once and then folded into the pretrained weights, and clustering changes only the stored weight representation rather than the graph. The paper states that inference remains exactly the same except that every multiply-accumulate is replaced with a 2-level LUT lookup. This point is important because a common misconception in low-precision methods is that auxiliary transforms necessarily introduce persistent deployment overhead. In CodeQuant, the orthogonal transforms are intended to be absorbed into existing linear layers at deploy time rather than executed as separate runtime operators.

The optional role of POG also matters. The method does not require permutation for all variants; rather, POG is presented as an intermediate step that can be applied for block-wise clustering. This indicates that the framework contains both a core mechanism—AOS plus ACCF—and an optional grouping strategy that improves within-block variance structure.

3. Mathematical formulation

The AOS component begins with activations $X \in \mathbb{R}^{N \times d_{\rm in}}$ and introduces a learnable orthogonal matrix $R \in \mathbb{R}^{d_{\rm in} \times d_{\rm in}}$ so that quantizing the rotated activations $X_R = X R$ incurs minimal error (Yin et al., 12 Apr 2026). To parameterize $R$ without constraints, the method forms a skew-symmetric matrix

$S = \tfrac12 (M - M^\top),$

where $X$0 is unconstrained, and then applies the Cayley transform

$X$1

The AOS objective is

$X$2

where $X$3 denotes the 4-bit quantizer.

After folding the learned rotation into the weights, the method defines $X$4. ACCF then clusters each output row $X$5 of $X$6 into $X$7 centroids $X$8. A one-hot assignment tensor

$X$9

produces the reconstructed weight

$K$0

The basic clustering objective is

$K$1

For MoE FFN layers, specifically the “gate” and “up” projections, the loss is augmented with a KL-divergence term to preserve the expert-routing distribution:

$K$2

Here $K$3 is the original MoE weighted sum on the calibration set, $K$4 is the original router probability matrix, and $K$5 is the router output after clustering. The parameter $K$6 trades off output matching versus routing fidelity.

Optimization proceeds by alternation: fixing $K$7 and updating centroids $K$8 with gradient descent on $K$9, then fixing $W_R = R^\top W$0 and updating $W_R = R^\top W$1. The assignment update is derived from a distance-based approximation involving

$W_R = R^\top W$2

where $W_R = R^\top W$3 and $W_R = R^\top W$4. The resulting assignment rule is

$W_R = R^\top W$5

The overall calibration problem is therefore staged:

$W_R = R^\top W$6

with POG optionally applied between them. This formulation makes explicit that activation smoothing and weight clustering are sequentially coupled rather than jointly solved in a single monolithic optimization.

4. Outlier handling mechanism and invariance properties

The paper’s explanation of how outliers become manageable is divided into activation outliers and weight outliers (Yin et al., 12 Apr 2026). For activations, a single orthogonal rotation $W_R = R^\top W$7 redistributes large-magnitude activation directions into coordinates with smaller peak values, allowing a tight 4-bit symmetric quantizer to cover the range with lower error. The same $W_R = R^\top W$8 is shared across all self-attention and FFN layers and is folded into the linear weights at deploy time.

For weights, once activations are smooth, the residual output variability is borne by the weights. Clustering each row of $W_R = R^\top W$9 into a small codebook means that extreme weights are no longer forced into a uniform quantization grid; instead, they are represented by one of the $R$0 learned centroids. Those centroids are then fine-tuned directly against end-to-end matrix-multiply error or router KL loss. The paper describes this as “absorbing” weight outliers into fine-tuned cluster centroids.

Two invariance properties are central to the deployment argument. First, the rotation is orthogonal and folded into the pretrained weights. Second, the permutation used in POG is paired with a compensating transformation, written in the pseudocode as applying $R$1 and $R$2 to preserve invariance. A practical consequence is that model topology is not changed. This suggests that CodeQuant should be understood less as an architectural modification than as a representation transformation combined with hardware-conscious execution.

5. Kernel design on GPU and CPU

CodeQuant couples its offline compression scheme with dedicated kernel designs for both GPU and CPU deployment (Yin et al., 12 Apr 2026). On modern NVIDIA GPUs, the weight matrix is tiled by the clustering block size. For each tile, the kernel stores the 16 centroids in shared memory and precomputes a $R$3 sub-LUT of centroid-by-activation-value products, where activation values are 4-bit integers. A two-level MUX hierarchy supplies each thread with a pointer to the correct subtable and then the correct entry, thereby masking out the need for any floating-point multiply.

To avoid bank conflicts, the LUT is replicated across more banks—for example, 64 banks rather than the native 32—and shared-memory layouts are rearranged so that simultaneous threads index different banks. Using Accel-Sim calibrated to an A100/H100, the paper reports a net $R$4 speedup over FP16/BF16 on GPU.

On CPU, the A8W4 variant is integrated into Llama.cpp using the T-MAC LUT GEMM kernel. In this implementation, each 4-bit activation and 8-bit weight-centroid combination is reduced to a single table lookup and accumulation in 32-bit. On an Intel Xeon w7-3445, the reported result is up to $R$5 end-to-end inference speedup with the same accuracy.

These kernels are not an auxiliary detail; they are part of the method’s definition as presented in the paper. A plausible implication is that CodeQuant’s reported efficiency gains depend not only on the compression scheme itself but also on execution paths specifically designed around codebook access patterns and shared-memory behavior.

6. Experimental methodology and results

The evaluation covers four representative MoE models: Phi-mini-MoE-Instruct (1.3 B), DeepSeek-V2-Lite (6 B), Qwen3-30B-A3B (30 B), and Mixtral 8×7B (56 B) (Yin et al., 12 Apr 2026). The tasks and metrics are language modelling perplexity on WikiText2 and C4, zero-shot question answering on ARC-Ch/E, HellaSwag, MMLU, PIQA, and WinoGrande using average accuracy, and few-shot math on GSM8K (8-shot) and MATH500 (4-shot). Calibration uses 1 024 WikiText2 samples for AOS with 128 iterations, and 512 samples for ACCF with 64 iterations and $R$6.

The baselines are RTN, SmoothQuant, QuaRot, DuQuant, SpinQuant, SqueezeLLM (A4W4), plus Hadamard-online variants. The representative embedding-wise A4W4 results reported against BF16 and QuaRot show that CodeQuant preserves substantially more accuracy than QuaRot across all four MoE models.

For Phi-mini-MoE, WikiText2 perplexity is 6.83 in BF16, 7.93 for QuaRot, and 7.63 for CodeQuant; C4 perplexity is 13.06, 14.44, and 13.94 respectively; and average zero-shot accuracy is 0.731, 0.694, and 0.700. For DeepSeek-V2-Lite, the corresponding values are 6.69, 7.75, and 7.08 on WikiText2 perplexity; 9.32, 10.75, and 9.85 on C4 perplexity; and 0.682, 0.640, and 0.664 on average accuracy.

For Qwen3-30B-A3B, WikiText2 perplexity is 9.04 in BF16, 16.04 for QuaRot, and 10.31 for CodeQuant; C4 perplexity is 14.05, 24.27, and 15.75; and average accuracy is 0.735, 0.581, and 0.694. For Mixtral 8×7B, WikiText2 perplexity is 4.01 in BF16, 16.79 for QuaRot, and 4.65 for CodeQuant; C4 perplexity is 7.41, 24.29, and 8.06; and average accuracy is 0.747, 0.497, and 0.725.

On mathematical reasoning, the paper reports that A4W4 embedding-wise CodeQuant on Qwen3-30B-A3B achieves 86.7% GSM8K accuracy, compared with 92.4% for BF16 and 50.8% for QuaRot, and 24.1% on MATH500, compared with 32.2% for BF16 and 12.8% for QuaRot. Kernel speed-ups normalized to BF16 = 1.0 average $R$7 on GPU and reach up to $R$8 on CPU with T-MAC.

Taken together, the reported results indicate that the method’s benefit is most visible in regimes where pure rotation-based A4W4 quantization degrades sharply. This suggests that the centroid-based handling of residual weight outliers is particularly consequential in larger or more heterogeneous MoE models.

7. Interpretation, scope, and practical implications

The paper characterizes CodeQuant as a practical, hardware-aware path to deploy very large MoE LLMs in ultra-low precision, achieving state-of-the-art accuracy-efficiency trade-offs without changing model topology or requiring full retraining (Yin et al., 12 Apr 2026). More specifically, it argues that a carefully unified approach—smoothing activation outliers via a single shared rotation and absorbing weight outliers via per-row clustering—can push MoE models down to 4-bit codebook weights and 4-bit activations without retraining or mixed precision.

Several clarifications follow from the paper’s design. First, the method is a post-training procedure rather than a retraining strategy. Second, its transformations are intended to be folded into existing layers, not executed as standalone operators at inference time. Third, clustering modifies the stored representation of weights, but the computation graph is unchanged. These properties distinguish it from methods that rely on online transforms, mixed-precision exceptions, or architectural surgery.

At the same time, the scope is specific. The evaluation is conducted on MoE LLMs, the low-precision setting emphasized is 4-bit activations with codebook weights, and the deployment claims are tied to dedicated LUT-based kernels. A plausible implication is that the method’s effectiveness depends on the conjunction of algorithmic compression and kernel co-design rather than on either component in isolation.

In summary, CodeQuant formalizes low-precision MoE deployment as a coupled problem of activation smoothing and weight-outlier absorption. Its contribution lies in making those two operations mathematically explicit, deployment-compatible, and hardware-conscious within a single PTQ framework.