Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference

Abstract: Two-bit weight quantization is attractive for memory-efficient LLM inference, but the standard W2 level set {-2,-1,0,+1} often collapses under aggressive W2A4/KV4 settings. We study the scalar level-set geometry of two-bit weights in a Hadamard-rotated quantization pipeline. Conventional asymmetric W2 substantially improves over the standard level set, indicating that W2A4 failure is not only a bit-width problem but also a reconstruction-level problem. Across all 224 linear modules in each of LLaMA-2-7B and LLaMA-3.1-8B, pretrained weights are already nearly zero-centered, while Hadamard rotation primarily Gaussianizes their standardized shape: excess kurtosis and Q-Q error drop by orders of magnitude. Based on this approximate zero-centered Gaussian-like source model, we propose Qift, a fixed no-zero W2 level set for rotated W2A4/KV4 inference. The main level set is {+/-0.5, +/-1.5}, equivalently {+/-1, +/-3} under a half-scale reparameterization; a power-of-two variant uses {+/-1, +/-4} for sign-and-shift decoded weight application. Qift redesigns the fixed two-bit code-to-level mapping and is training-free, learned-codebook-free, group-grid-free, and zero-point-free, retaining the standard per-channel scale. A scale-invariant ratio analysis identifies an effective inner/outer centroid ratio range of 0.25 to 0.33, explaining why mirror no-zero (MNZ), Lloyd, NF2, and PoT-MNZ perform well while {+/-1, +/-2} does not. On both models, the no-zero level sets consistently improve pure W2A4 perplexity, L-layer mixed W2/W4 perplexity, downstream accuracy, and GPTQ residual behavior over the standard W2 level set. At L=16 mixed precision, they substantially narrow the gap to W3A4 while keeping half of the transformer layers at two-bit precision, giving a simple, source-aware, and deployment-friendly alternative to more complex learned W2 codebooks.

• The paper presents a novel post-training quantizer that avoids zero-centroid misallocation, greatly enhancing 2-bit weight precision.
• It leverages rotated weight distributions and fixed modular grids (MNZ and PoT-MNZ) to optimize centroid geometry for efficient inference.
• Empirical evaluations on LLaMA models show significant reductions in perplexity and improvements in accuracy without the need for retraining.

Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference

Introduction and Motivation

Qift introduces a targeted intervention in post-training quantization (PTQ) for LLMs operating in the highly aggressive W2A4/KV4 regime (2-bit weights, 4-bit activations and KV-cache). As modern deployment scenarios for LLMs are memory-limited and memory bandwidth, rather than arithmetic, dominates inference throughput in decode phase, weight and KV-cache bitwidth reduction is paramount for practical, cost-effective inference. While W3A4 and W4A4 quantization with rotation have approached near-lossless performance, pushing to W2A4 remains fraught with significant accuracy degradation. The literature has largely inherited the standard symmetric integer 2-bit grid $\{-2, -1, 0, +1\}$ without re-examining its geometric fit to the distributions induced by orthogonal rotations such as Hadamard transforms. This work frames the W2A4/KV4 challenge as not just a limited-bitwidth problem, but fundamentally one of centroid geometry mismatch.

Foundations: Rotated Weight Distributions and Level-Set Design

Recent advances in quantization, including activation smoothing (e.g., SmoothQuant), rotation-based PTQ (e.g., QuaRot, SpinQuant), and compensation (e.g., GPTQ, GPTAQ) have leveraged equivalent transformations to improve quantizability. However, with only four reconstruction levels, the allocation of centroids in the presence of a zero-centered, nearly-Gaussian, low-skewed, and low-kurtosis weight distribution is critical. The authors empirically confirm that Hadamard rotation Gaussianizes per-channel weight distributions in pretrained LLaMA-2 and LLaMA-3.1 models: per-channel mean remains nearly zero, while excess kurtosis, skewness, and Q–Q error relative to a normal distribution fall by several orders of magnitude post-rotation. This motivates viewing rotated weights as approximately zero-centered Gaussian sources, and the structure of the scalar quantizer as a first-class lever.

From the quantization perspective, Lloyd-Max theory for a zero-mean Gaussian dictates that, with four quantization levels, optimal placement occurs with two inner centroids symmetric around zero and two outer centroids in the tails—crucially, not allocating a centroid exactly at zero. Thus, the use of the mid-tread (zero-included) standard grid results in significant representational waste, with one quarter of quantization range dedicated to the near-zero region inappropriately.

Qift Methodology

Qift proposes a fixed, modular redesign of the 2-bit weight code-to-level mapping, focusing on hardware-favorable and source-aware level sets:

• Mirror No-Zero (MNZ) grid: $\{-1.5, -0.5, +0.5, +1.5\}$ (also representable as odd integers $\{\pm1, \pm3\}$ under rescaling).
• PoT-MNZ grid: $\{\pm1, \pm4\}$, enforcing power-of-two structure, facilitating sign-and-shift decoding.

Both configurations are globally fixed and used per-channel with only a single learned scaling factor. They are zero-point-free, codebook-free, group-grid-free, and compatible with existing pipelines (e.g., RTN, GPTQ, GPTAQ).

A scale-invariant inner/outer centroid ratio $r$ is identified as the critical design knob ($\{-1, -r, +r, +1\}$). Empirical analysis demonstrates that good empirical performance is obtained for $r\in[0.25, 0.33]$ (e.g., MNZ at $r=1/3$, PoT-MNZ at $1/4$), consistent with Lloyd-Max and NormalFloat2 (NF2) centroids.

The Qift method requires no retraining, codebook learning, or metadata adaptation, making it a strictly post-training quantizer plug-in within existing rotation-based PTQ pipelines.

Empirical Evaluation and Numerical Findings

Quantitative evaluations were conducted on LLaMA-2-7B and LLaMA-3.1-8B under pure W2A4/KV4, mixed W2/W4 precision, and a variety of downstream tasks. The baseline SYM-INT grid yields severe degradation or collapse in perplexity and accuracy (e.g., LLaMA-2-7B PPL from 5.471 to 53.849 and LLaMA-3.1-8B from 6.277 to over 3000). Merely asymmetric W2 brings some recovery but is far from optimal.

Adoption of Qift (MNZ) under identical conditions yields:

• LLaMA-2-7B: pure W2A4 GPTAQ PPL reduced from 12.118 (SYM-INT) to 9.294, with downstream average accuracy increased from 0.4211 to 0.4794.
• LLaMA-3.1-8B: pure W2A4 GPTAQ PPL drops from 29.695 to 19.515, and downstream accuracy improves from 0.3683 to 0.4064.
• At L=16 mixed-precision (16 layers at W4, rest at W2), MNZ W2/W4 narrows the W3A4 gap by closing nearly half the perplexity difference attributable to W2 grid misallocation.

Ablations confirm that performance gain derives not solely from the absence of a zero centroid, but a well-chosen $r$. Evaluations with Far-MNZ ($\{-1.5, -0.5, +0.5, +1.5\}$0) demonstrate failure—removing zero but placing inner centroids too far from the dense bulk sacrifices accuracy. The RTN bucket diagnostic substantiates more balanced assignment and significant reduction in aggregate reconstruction error for MNZ and PoT-MNZ vs. SYM-INT. Notably, the Qift grids remain competitive or superior to learned or Gaussian-optimal level sets (Lloyd-Max, NF2), but without the overhead or hardware disadvantages (Table: ratio-sensitivity).

Across all settings, Qift provides strictly improved GPTQ residual behavior and downstream metric alignment, while maintaining code simplicity and deployment modularity.

Theoretical and Practical Implications

Qift recasts the extreme low-bit quantization problem as the joint solution of optimal bit allocation and centroid geometry, rather than accepting the inherited mid-tread grid from symmetric integer quantizers. Its separation of rotation/calibration pipeline and quantization level design enables easy integration with advanced compensation schemes, kernel optimizations, and future hardware-aware mixed-precision policies. The method's hardware friendliness—sign-and-shift decoded values, no codebook or zero-point overhead—renders it immediately deployable in edge/server environments targeting aggressive memory reduction.

In the broader context, this work highlights that for deep LLMs operating near the quantization/accuracy cliff, fine-grained quantizer design at the scalar level is an untapped avenue for meaningful improvements in efficiency-accuracy tradeoffs, surpassing what simple bitwidth increments or naive codebook learning regimes can offer.

Future Directions

Future work can extend Qift to sub-channel or structured quantizer designs, hardware-specific kernel implementations exploiting power-of-two decoding, and adaptive ratio selection mechanisms. There is also the potential for joint optimization of centroid placement and adaptive mixed-precision policies, aligning quantizer geometry with resource-aware layer sensitivity. Additionally, systematic study of centroid geometry for ternary quantization (W1A4) and integration with model-retraining regimes remain open avenues.

Conclusion

Qift demonstrates that careful redesign of the fixed two-bit code-to-level mapping, grounded in empirical analysis of rotated weight statistics, is both necessary and sufficient to recover most accuracy lost at W2A4 quantization, achieving strong results without any increase in algorithmic or deployment complexity. This finding decouples the "bitwidth barrier" from centroid geometry mismatches and opens the path for further source-aware, hardware-friendly quantizer designs in LLM inference pipelines.