Per-Head Residual Vector Quantization

Updated 12 July 2026

Per-head residual vector quantization is a compression technique that decomposes transformer KV caches into independent attention-head or latent-group blocks, enabling additive residual encoding.
It employs spectral denoising to extract low-rank shared structures before quantizing full-rank residuals with methods like TurboQuant for near-lossless reconstruction.
Empirical results demonstrate reduced bits-per-entry and preserved model performance while simplifying training stability and codebook management.

Searching arXiv for the cited papers to ground the article in current literature. {"query":"(Su, 6 Apr 2026) eOptShrinkQ Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization arXiv"} {"query":"(Kumar, 2024) Residual vector quantization for KV cache compression in LLM arXiv"} {"query":"(Zandieh et al., 28 Apr 2025) TurboQuant Online Vector Quantization with Near-optimal Distortion Rate arXiv"} {"query":"per-head residual vector quantization arXiv KV cache audio codecs"} Per-head residual vector quantization denotes a family of quantization schemes in which residual coding is organized around a head-structured decomposition. In transformer compression, the head is an attention head, and the quantizer operates on head-local key-value blocks or on groups derived from head slices. In neural codecs, the head is often a residual quantization stage or a grouped latent partition, and successive heads encode progressively finer residual information. The literature therefore uses the term across two closely related but not identical settings: stacked codebook-based residual vector quantization, and per-head residual compression pipelines that first isolate a structured component and then quantize the residual. A prominent recent formulation is eOptShrinkQ, which models each per-head KV block as a low-rank shared component plus a full-rank residual, removes the shared structure by optimal spectral shrinkage, and then quantizes the residual row-by-row with TurboQuant (Su, 6 Apr 2026).

1. Conceptual scope of “per-head”

In transformer KV-cache compression, per-head processing means that each attention head is handled independently. For a transformer layer with $h$ attention heads and head dimension $d$ , each head maintains key and value caches $K \in \mathbb{R}^{T \times d}$ and $V \in \mathbb{R}^{T \times d}$ . eOptShrinkQ processes each head independently, in blocks of consecutive tokens of size $n \approx d$ such as $n=d=128$ , forming per-head matrices $X_K \in \mathbb{R}^{n \times d}$ and $X_V \in \mathbb{R}^{n \times d}$ and compressing each block separately (Su, 6 Apr 2026).

In residual vector quantization for audio and related latent codecs, per-head has two common meanings. In the sequential interpretation, each residual codebook is itself a head, so stage $k$ is head $k$ . In the grouped interpretation, the latent vector is partitioned into groups, each group has its own RVQ stack, and each group’s codebook or codebooks form a head operating on a disjoint feature subspace (Zheng et al., 2024).

Transformer KV-cache RVQ introduces a third nuance. In "Residual vector quantization for KV cache compression in LLM," quantization is applied to the output of a key or value projection before RoPE; the projected vector is flattened, scaled by its standard deviation, partitioned into groups of size $d$ 0, and quantized groupwise with a residual quantizer shared across all groups for that matrix within a layer. Since $d$ 1 is smaller than a head dimension and non-contiguous grouping may mix channels from multiple heads, the method is only per-head in an indirect sense: it acts on a flattened per-layer vector whose groups may or may not align with head boundaries (Kumar, 2024).

This suggests that per-head residual vector quantization is best understood as a structural principle rather than a single implementation. The relevant invariant is that residual coding respects a head-like decomposition, but the head may be an attention head, a residual stage, a grouped latent partition, or an expert-selected branch.

2. Canonical residual quantization formalism

The standard residual vector quantization recursion is additive. Let $d$ 2 denote an input vector, let $d$ 3, and let stage $d$ 4 select an embedding $d$ 5 from codebook $d$ 6. The residual update is

$d$ 7

The reconstructed vector is the sum of the selected codewords across stages. This coarse-to-fine recursion is the common backbone of classical RVQ, ERVQ, VRVQ, and PURE Codec (Zheng et al., 2024, Chae et al., 2024, Shi et al., 27 Nov 2025).

Generalized residual vector quantization formulates the same idea as an additive reconstruction problem over multiple codebooks. A vector $d$ 8 is approximated by

$d$ 9

and the learning objective minimizes the total squared reconstruction error over the dataset. In the per-head setting, the feature space is partitioned into disjoint sub-vectors

$K \in \mathbb{R}^{T \times d}$ 0

with per-head reconstruction

$K \in \mathbb{R}^{T \times d}$ 1

Because heads act on disjoint dimensions, the global squared error decomposes additively, which permits independent optimization across heads and head-wise asymmetric distance computation without cross-head interaction (Liu et al., 2016).

This decomposition clarifies an important distinction. In classical codebook RVQ, the residual is represented by additional codewords. In eOptShrinkQ, by contrast, the residual is not recursively encoded by further codebooks. Instead, a low-rank shared component is first extracted, and the remaining full-dimensional residual is quantized directly. The method remains residual-centric, but the residual object is a denoised vector field rather than a sequence of codebook errors (Su, 6 Apr 2026).

3. Per-head RVQ for transformer KV caches

A direct codebook-based KV-cache instantiation is given in "Residual vector quantization for KV cache compression in LLM" (Kumar, 2024). The method quantizes the output of the key and value projection matrices before RoPE. Each projected vector $K \in \mathbb{R}^{T \times d}$ 2 is scaled by its standard deviation $K \in \mathbb{R}^{T \times d}$ 3, producing $K \in \mathbb{R}^{T \times d}$ 4. The scaled vector is divided into $K \in \mathbb{R}^{T \times d}$ 5 groups of dimension $K \in \mathbb{R}^{T \times d}$ 6, and each group is quantized with an $K \in \mathbb{R}^{T \times d}$ 7-stage residual quantizer:

$K \in \mathbb{R}^{T \times d}$ 8

The quantized group is $K \in \mathbb{R}^{T \times d}$ 9, and the full vector is reconstructed by concatenating the groups and rescaling:

$V \in \mathbb{R}^{T \times d}$ 0

The reported default configuration uses $V \in \mathbb{R}^{T \times d}$ 1 residual stages, codebook size $V \in \mathbb{R}^{T \times d}$ 2, and group dimension $V \in \mathbb{R}^{T \times d}$ 3. Each attention block learns two residual quantizers, one for keys and one for values. Within a layer, the same residual quantizer is shared across all channel groups and thus across all heads. Codebooks are not shared across layers. Codebooks are initialized by k-means on the first batch of inputs and updated by exponential moving average with decay $V \in \mathbb{R}^{T \times d}$ 4; there are no encoder or decoder projection matrices and no commitment loss (Kumar, 2024).

Grouping strategy is asymmetric between keys and values. For keys, non-contiguous grouping works better than contiguous grouping. For values, contiguous grouping is preferred. The paper reports that a residual depth of $V \in \mathbb{R}^{T \times d}$ 5 recovers most of the performance of the unquantized model, that $V \in \mathbb{R}^{T \times d}$ 6 outperforms $V \in \mathbb{R}^{T \times d}$ 7, and that the method is competitive with existing KV-cache quantization methods while remaining much simpler. The reported compression is roughly $V \in \mathbb{R}^{T \times d}$ 8 compared to half precision (Kumar, 2024).

This formulation corrects a common misconception: per-head RVQ does not necessarily imply one learned codebook stack per attention head. In this KV-cache design, the quantizer is shared per matrix per layer, and head structure enters through grouping and through the downstream reshaping back to $V \in \mathbb{R}^{T \times d}$ 9.

4. Spectral denoising and residual quantization in eOptShrinkQ

eOptShrinkQ defines per-head residual vector quantization through a low-rank-plus-residual model for each per-head KV block:

$n \approx d$ 0

where $n \approx d$ 1 is a low-rank signal representing shared contextual structure across tokens in the block, and $n \approx d$ 2 is a full-rank residual capturing per-token idiosyncratic content. For keys, the decomposition arises from

$n \approx d$ 3

and for values the same decomposition holds without RoPE:

$n \approx d$ 4

The residual is modeled as colored noise with separable covariance, while $n \approx d$ 5 is low-rank and the detectable singular spikes are governed by a BBP phase transition (Su, 6 Apr 2026).

The spectral stage computes the SVD

$n \approx d$ 6

and reconstructs only the detected outlier components with a nonlinear shrinker:

$n \approx d$ 7

Automatic rank selection is obtained from the estimated bulk edge $n \approx d$ 8 and the BBP threshold

$n \approx d$ 9

This differs from truncated SVD in two ways stated explicitly in the paper: eOptShrink corrects the inflation of each outlier singular value via optimal shrinkage, and it selects rank automatically from the spectrum (Su, 6 Apr 2026).

After shrinkage, the residual

$n=d=128$ 0

is quantized row-by-row with TurboQuant. TurboQuant operates per vector by separating norm and direction, applying a random orthogonal rotation, quantizing each rotated coordinate with a Lloyd–Max codebook optimized for $n=d=128$ 1, and reconstructing by inverse rotation. The generic scalar mapping can be written as

$n=d=128$ 2

although TurboQuant uses codebook bins rather than uniform steps (Zandieh et al., 28 Apr 2025). eOptShrinkQ uses TurboQuant-MSE rather than TurboQuant-prod because, after shrinkage, the residual becomes close to isotropic and delocalized:

$n=d=128$ 3

The paper further gives a residual inner-product bias reduction bound

$n=d=128$ 4

which is used to argue that outlier handling and dedicated inner-product bias correction are unnecessary after spectral denoising (Su, 6 Apr 2026).

The storage budget reflects this two-part decomposition. Residual entries use $n=d=128$ 5 bits per entry, with a small shared cost for one FP16 norm per row. Low-rank factors contribute an overhead

$n=d=128$ 6

so the MSE-variant total is

$n=d=128$ 7

Runtime decoding reconstructs $n=d=128$ 8 from stored singular vectors and singular values, dequantizes residual rows, and adds the two terms to form $n=d=128$ 9 (Su, 6 Apr 2026).

This architecture departs sharply from codebook-based RVQ. Traditional VQ or RVQ often relies on learned codebooks or sub-vector codebooks, coordinate subgrouping, and special treatment of outlier channels. eOptShrinkQ first removes low-rank shared structure, then applies scalar quantization to a residual whose geometry is deliberately restored to match the assumptions under which scalar quantization is near-optimal (Su, 6 Apr 2026).

A large part of the per-head RVQ literature concerns training stability and codebook utilization rather than inference-time KV-cache compression. ERVQ augments RVQ with intra-codebook and inter-codebook optimization. Intra-codebook optimization tracks exponentially decayed codeword usage,

$X_K \in \mathbb{R}^{n \times d}$ 0

refreshes underused entries by anchor blending, and regularizes the empirical assignment distribution toward the uniform prior with a balancing loss. Inter-codebook optimization minimizes SSIM between adjacent quantized outputs,

$X_K \in \mathbb{R}^{n \times d}$ 1

to reduce redundancy between successive heads. On APCodec with $X_K \in \mathbb{R}^{n \times d}$ 2 VQs of size $X_K \in \mathbb{R}^{n \times d}$ 3, ERVQ reports $X_K \in \mathbb{R}^{n \times d}$ 4 utilization versus $X_K \in \mathbb{R}^{n \times d}$ 5– $X_K \in \mathbb{R}^{n \times d}$ 6 without ERVQ, perplexity improvements from $X_K \in \mathbb{R}^{n \times d}$ 7 to $X_K \in \mathbb{R}^{n \times d}$ 8, and bitrate efficiency improvement from $X_K \in \mathbb{R}^{n \times d}$ 9 to $X_V \in \mathbb{R}^{n \times d}$ 0 (Zheng et al., 2024).

VRVQ makes the number of active residual heads variable across frames. It uses an importance network to produce a per-frame score $X_V \in \mathbb{R}^{n \times d}$ 1, maps that score to a monotone top- $X_V \in \mathbb{R}^{n \times d}$ 2 binary mask, and applies only the first $X_V \in \mathbb{R}^{n \times d}$ 3 heads on that frame. Training uses a straight-through estimator with a smooth surrogate

$X_V \in \mathbb{R}^{n \times d}$ 4

and a rate–distortion objective

$X_V \in \mathbb{R}^{n \times d}$ 5

In the DAC-based setup reported, $X_V \in \mathbb{R}^{n \times d}$ 6, each codebook has size $X_V \in \mathbb{R}^{n \times d}$ 7, frame rate is approximately $X_V \in \mathbb{R}^{n \times d}$ 8 Hz, and signaling $X_V \in \mathbb{R}^{n \times d}$ 9 costs $k$ 0 bits per frame, or approximately $k$ 1 kbps (Chae et al., 2024).

SwitchCodec replaces fixed residual stacks with a shared base quantizer and a pool of expert quantizers chosen by a router. The shared head is always active; routed experts are selected by TopK affinity scores and then applied in ascending expert index order to preserve a stable residual hierarchy. The paper reports a single model covering approximately $k$ 2 kbps to $k$ 3 kbps without retraining, and gives a routing-mask overhead example of less than $k$ 4 at $k$ 5 kbps when $k$ 6, $k$ 7, and the routing window is $k$ 8 seconds (Wang et al., 28 Jan 2026).

PURE Codec modifies the first residual head rather than the head count. The first stage is trained to approximate a denoised embedding $k$ 9, while later stages encode the residual relative to the original encoder output. The enhancement-guided loss is

$k$ 0

The reported motivation is that the enhanced signal has lower perceptual entropy, empirically reduced by $k$ 1 on average compared to noisy inputs. PURE uses a DAC-style backbone with embedding dimension $k$ 2, $k$ 3 quantizers, and codebook size $k$ 4, and reports large gains over a DAC baseline on noisy corpora such as URGENT and CommonVoice (Shi et al., 27 Nov 2025).

Taken together, these papers show that per-head RVQ has evolved along three axes: codebook refresh and balancing, variable head activation, and semantically guided residual decomposition. The common problem is not merely compression efficiency but stable allocation of information across residual stages or head partitions.

6. Empirical behavior, practical guidance, and limitations

eOptShrinkQ provides the most explicit empirical characterization of per-head residual quantization in transformer KV caches. On Llama-3.1-8B-Instruct keys, eOptShrinkQ_MSE at $k$ 5 bpe achieves $k$ 6 L2 and near-zero IP bias/std, versus TurboQuant_MSE at $k$ 7 bpe with $k$ 8 L2; at approximately $k$ 9 bpe, eOptShrinkQ_MSE reaches $d$ 00 L2 versus TurboQuant_MSE at $d$ 01 bpe with $d$ 02 L2. On values, eOptShrinkQ_MSE at $d$ 03 bpe achieves $d$ 04 L2 versus TurboQuant_MSE at $d$ 05 bpe with $d$ 06 L2, and at $d$ 07 bpe achieves $d$ 08 versus $d$ 09. On LongBench, eOptShrinkQ_MSE at approximately $d$ 10 bpe achieves $d$ 11 average on Llama-3.1-8B and $d$ 12 on Ministral-8B, outperforming both TurboQuant_MSE and TurboQuant_prod at $d$ 13 bpe. On multi-needle retrieval, eOptShrinkQ_MSE at $d$ 14 bpe averages $d$ 15 recall on Llama-3.1-8B, exceeding FP16 at $d$ 16, and $d$ 17 on Ministral-8B, nearly matching FP16 at $d$ 18 (Su, 6 Apr 2026).

The eOptShrinkQ paper also provides explicit implementation guidance. A recommended starting point is $d$ 19 bits for the residual, with $d$ 20 and $d$ 21, giving a typical total of $d$ 22– $d$ 23 bpe. Keys and values can use the same $d$ 24, because post-shrinkage isotropy is stated to suffice for both. Compression is integrated on-the-fly per chunk in prefill, new tokens are buffered until a block is formed, and the most recent at most $d$ 25 tokens remain FP16 (Su, 6 Apr 2026).

The codebook-based KV-cache RVQ paper identifies different practical trade-offs. Its guidance is to use non-contiguous grouping for keys, contiguous grouping for values, $d$ 26 stages, and $d$ 27. The paper notes that $d$ 28 codebooks per quantizer may introduce computational challenges, especially in compute-bound phases such as prefill and large-batch decoding, and recommends fused kernels to reduce the overhead (Kumar, 2024).

Several limitations recur across the literature. eOptShrinkQ assumes block-wise shared structure in KV caches and separable residual covariance; if the residual is highly structured or anisotropic, isotropy may not be fully restored and gains may reduce. Its BBP criterion can be conservative when $d$ 29 is very small or signals are extremely weak. The paper also states that tasks requiring global context integration, such as summarization, may benefit from some shared structure, so spectral denoising can slightly lower scores relative to FP16. Truly token-by-token streaming remains future work because the implementation compresses in blocks (Su, 6 Apr 2026).

Other limitations are modality-specific. TurboQuant’s theory becomes stronger as dimensionality increases, and the recommendations favor head dimension at least $d$ 30; very small head dimensions weaken the Beta-to-Gaussian approximation and may require slightly larger bit-widths or stronger mixing transforms (Zandieh et al., 28 Apr 2025). PURE explicitly depends on a speech enhancement model and is therefore speech-specific; the paper notes that severe domain mismatch between enhancement model and data can make the first-head anchor suboptimal (Shi et al., 27 Nov 2025).

A final source of confusion concerns the relationship between quality and regularization. In eOptShrinkQ, the claim that spectral denoising may improve retrieval-intensive tasks by acting as a regularizer is presented as a hypothesis rather than a theorem. The reported retrieval gains therefore support, but do not establish, a general principle (Su, 6 Apr 2026). More broadly, the literature indicates that per-head residual vector quantization should not be treated as a single fixed recipe. It is a design family whose members differ in whether heads are attention heads, residual stages, latent groups, or routed experts; whether residuals are represented by codewords or by denoise-then-quantize pipelines; and whether the primary objective is inference-time KV-cache compression, training-time codebook health, variable bitrate control, or structured decomposition of residual entropy.