Per-Channel Key Quantization in LLMs

• Per-channel key quantization is a technique that replaces a shared scale with individual channel scales to mitigate the effects of distributional outliers in transformer architectures.
• It is applied in transformer-based large language models to enhance quantization fidelity at ultra-low bit-widths, achieving significant memory compression and speed improvements.
• Method extensions such as adaptive grouping, latent basis, and polar transforms offer tailored solutions to overcome channel-specific quantization challenges.

Per-channel key quantization is a class of quantization techniques in which each channel (dimension) of a key vector or key-projection matrix—most often arising in transformer-based LLMs and attention-based architectures—is assigned a distinct quantization scale (and, for asymmetric schemes, zero-point). The primary motivation is to mitigate the deleterious effects of distributional outliers along individual channels, thus improving quantization fidelity at ultra-low bit-widths and enabling hardware-efficient deployment at scale. These methods are now standard for compressing the key (K) or value (V) cache in LLM inference, as well as for resilient post-training quantization of linear and attention modules.

1. Fundamental Principles and Canonical Schemes

Per-channel quantization replaces a shared (per-tensor or per-token) quantization scale with a vector of scales—typically one per channel or group of channels—thereby localizing dynamic range adaptation. For a key cache matrix $K\in\mathbb{R}^{l\times d}$ with $l$ tokens and $d$ channels, per-channel $b$-bit asymmetric quantization is defined by

$$\begin{aligned} &\text{Calibration:}\quad \min_c = \min_{i} K_{i,c},\qquad \max_c = \max_{i} K_{i,c} \ &s_c = \frac{\max_c - \min_c}{2^b - 1}, \quad z_c = \min_c\ &\text{Quantization:}\quad Q(K_{i,c}) = \mathrm{clip}(\mathrm{round}((K_{i,c} - z_c)/s_c),\,0,\,2^b-1)\ &\text{Dequantization:}\quad \hat K_{i,c} = s_c\,Q(K_{i,c}) + z_c \end{aligned}$$

This decoupling of channel ranges is critical for attenuating channel-specific outlier effects, as demonstrated in multiple empirical studies (Liu et al., 2024, Hooper et al., 2024).

For weight matrices, per-channel quantization is most commonly applied along the output axis ($M\times N$ matrix: per-row), while for activations and cached keys, it is applied per hidden dimension.

The extreme of this approach is groupwise per-channel, where channels are partitioned (coarsely or adaptively) and each group is quantized separately (Heo et al., 2023, Qin, 2024).

2. Motivation: Outliers and Failure Cases in Per-Tensor Quantization

The principal rationale for per-channel key quantization is the presence of stark, persistent channel-level outliers. In transformer and LLMs, key (and sometimes value) channels exhibit heavy-tailed activations or weight magnitudes, especially before positional embeddings or in initial transformer blocks (Hooper et al., 2024, Qin, 2024, Liu et al., 2024). Under per-tensor quantization, a single errant channel with large range dominates the global scale, destroying the effective quantization resolution for all other (well-behaved) channels.

Empirical evidence is robust:

• KIVI reports 4.55% reconstruction error with per-channel 2-bit quantization versus 13.67% for per-token (tied scale) at equal bit width. Attention-score error is reduced five-fold in per-channel mode (Liu et al., 2024).
• In LLaMA3-70B, a handful of layers exhibit "max_abs" weight values $\sim1000\times$ larger than their neighbors; per-channel quantization there causes catastrophic accuracy collapse unless outlier-tolerant mechanisms (per-group quantization, smoothing) are introduced (Qin, 2024).
• In KVQuant, 3-bit per-channel key quantization yields $\sim 4$ point perplexity improvement over per-token (per-tensor) (Hooper et al., 2024).

3. Methodological Extensions: Grouping, Polar, and Latent Bases

Beyond the basic per-channel paradigm, several works propose structural generalizations:

• Per-group quantization: Instead of individual scales for every channel, nearby channels are merged into groups (e.g., size 1024), balancing error smoothing and hardware complexity. LLaMA3-70B anomaly mitigation uses mixed per-group-per-channel quantization in $<3\%$ of layers, fully restoring performance with negligible throughput loss (Qin, 2024).
• Adaptive grouping (AdaDim): Input channels are partitioned using a Hessian-aware sensitivity metric, allocating bit-width where quantization error is likely to be most detrimental. This is especially effective for attention key projections and sub-4-bit quantization (Heo et al., 2023).
• Latent basis quantization (SVDq): SVDq performs SVD on the key cache, quantizes the most significant singular vector projections at higher precision, and low-energy directions at lower precision or truncation. This achieves $\leq$0.1$\times$ the quantization error (in MSE) of vanilla per-channel at the same bit rate, and enables up to $410\times$ compression with negligible loss (Yankun et al., 21 Feb 2025).
• Polar/rotary quantization: In the context of RoPE, cartesian channel outliers are transformed into the polar domain, in which radius and angle are quantized separately per pair of rotated dimensions, yielding both outlier-robustness and streamlined decoding (Wu et al., 1 Feb 2025).
• Coupled quantization: Recognizing statistical dependence across key channels, small channel groups are quantized jointly via vector quantization; this method allows pushing precision down to 1 bit per channel with high information retention (Zhang et al., 2024).

4. Interactions with Positional Embeddings and Attention

In LLMs and transformer decoders, key and value vectors are modified by positional embeddings, most notably rotary positional embedding (RoPE). Pre-RoPE and post-RoPE per-channel quantization exhibit markedly different statistical properties:

• Pre-RoPE quantization is preferred in most schemes due to the stationarity of channel statistics; if quantization is performed post-RoPE, the rotation mixes channel pairs, amplifying distributional variance and reducing quantization accuracy (Hooper et al., 2024, Liu et al., 2024).
• PolarQuant quantizes in the 2D polar subspace post-RoPE, harnessing the geometric structure of the rotated vectors and bypassing the outlier problem (Wu et al., 1 Feb 2025).

These interactions dictate when and how calibration is conducted, and whether specialized kernels are needed for decoding and attention matmuls.

5. Empirical Performance, Pathologies, and Special Cases

Quantitative results demonstrate dramatic improvements from per-channel quantization and its enhancements:

Method	Model/Task	Bits	Perplexity / Accuracy Loss	Throughput/Memory Impact	Citation
Per-channel 2-bit	Llama2 KV, CoQA	2	$<1$ pt drop	$2.6\times$ memory cut, $2-3.5\times$ speedup	(Liu et al., 2024)
Per-channel 3-bit	LLaMA-7B, Wikitext2	3	$\Delta$ PPL 0.07	$4.8\times$ KV compression	(Hooper et al., 2024)
Per-group 8-bit	LLaMA3-70B, 8 tasks	8	FP16-level (from $-28$\%)	$1-2\%$ slow only on $<3\%$ layers	(Qin, 2024)
SVDq ($\bar b=1.25$)	LLaMA3.1-8B, RULER/LongBench	1.25	$\sim$–3.3 pts (vs 3-bit vanilla, –11 pts)	$410\times$ compression	(Yankun et al., 21 Feb 2025)
PolarQuant-m4n4	Qwen2.5/KV, LongBench	4.16	$<$0.3 pts vs FP16	Speedup $1.27\times$ vs matmul	(Wu et al., 1 Feb 2025)
CQ-8c8b 1 bit/ch	LLaMA-7B, WikiText2	1	PPL 8.09 (vs 5.68 FP16)	$16\times$ memory, $8-12\times$ speed	(Zhang et al., 2024)

Certain pathologies exist: in LLaMA3-70B, extreme channel outliers in early attention blocks can defeat per-channel quantization, requiring targeted application of per-group or smoothing strategies (Qin, 2024). All schemes benefit from a minimal calibration phase (offline), with 8–32 examples often sufficient to estimate stable channel statistics (Qin, 2024, Hooper et al., 2024).

6. Software, Hardware, and Implementation Considerations

Per-channel key quantization offers substantial advantages in compatibility with hardware accelerators, as scale/zero-point arrays enable SIMD and tensor-core fused quantized matmuls (Qin, 2024, Wang et al., 7 Mar 2025). However, methods such as SVDq or coupled quantization introduce additional memory for codebooks or basis storage, though these are asymptotically negligible for $d\ll s$.

Per-group and per-pair schemes may entail minor kernel fragmentation and require small runtime scale corrections or lookup tables; the overhead remains sublinear in practice since most compute is still dominated by large GEMM operations (Qin, 2024, Wu et al., 1 Feb 2025).

For static quantization (e.g., MergeQuant), quantization steps can be entirely fused into norm and linear layers, eliminating runtime quantize/dequant calls and yielding up to $2\times$ decoding speedup and $4\times$ memory reduction without accuracy loss at 4 bits (Wang et al., 7 Mar 2025).

7. Extensions, Limitations, and Research Frontiers

Recent trends push beyond independent per-channel quantization:

• Mixed-precision assignment using Hessian or Fisher information allows ultra-low bitwidth regimes without catastrophic accuracy collapse (Heo et al., 2023, Yankun et al., 21 Feb 2025, Zhang et al., 2024).
• Channel coupling (multivariate vector quantization) exploits statistical dependencies to reach the 1 bit/channel limit, achieving up to $16\times$ KV cache compression while outperforming naive per-channel at similar rates (Zhang et al., 2024).
• Latent basis (SVDq) strategies indicate that quantizing projected channels is fundamentally superior in models where singular value spectra are sharply decaying (Yankun et al., 21 Feb 2025).

Limitations include: higher calibration burden in highly nonstationary networks, marginal kernel complexity in per-group/polar transforms, and reliance on the persistence of outlier structure (pathological key distributions may require manual intervention) (Qin, 2024).

Research continues to focus on methods for robust, information-optimal quantization under changing data and model regimes, and hardware co-designs that exploit the channel-local structure of activations and weights.

---

References

• "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache" (Liu et al., 2024)
• "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization" (Hooper et al., 2024)
• "The Uniqueness of LLaMA3-70B Series with Per-Channel Quantization" (Qin, 2024)
• "Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of LLMs" (Heo et al., 2023)
• "MergeQuant: Accurate 4-bit Static Quantization of LLMs by Channel-wise Calibration" (Wang et al., 7 Mar 2025)
• "KV Cache is 1 Bit Per Channel: Efficient LLM Inference with Coupled Quantization" (Zhang et al., 2024)
• "SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention" (Yankun et al., 21 Feb 2025)
• "PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration" (Wu et al., 1 Feb 2025)