eOptShrinkQ: Near-Lossless KV-Cache Compression

• eOptShrinkQ is a KV-cache compression method for transformers that separates a low-rank shared-context component from a full-rank residual using optimal spectral denoising.
• It employs a two-stage process with singular value shrinkage and residual quantization to preserve inner-product fidelity essential for attention.
• Empirical results on models like Llama-3.1 and Ministral-8B demonstrate reduced memory footprint and improved retrieval performance through near-lossless compression.

eOptShrinkQ is a KV-cache compression method for transformers that combines spectral denoising with quantization. It is motivated by the observation that a KV-cache block is not an arbitrary dense matrix but can be modeled as a superposition of a low-rank shared context component and a full-rank per-token residual, well described by a spiked random matrix model. The method therefore applies optimal singular value shrinkage to extract the shared structure and then quantizes only the residual. In the formulation of "eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization," the denoising stage restores the isotropy assumed by scalar quantization, reduces inner-product bias, and enables compression at roughly $2$–$3$ bits per entry while maintaining strong downstream attention fidelity (Su, 6 Apr 2026).

1. Problem setting and motivation

In autoregressive LLM inference, the KV cache stores all past keys and values so that attention can reuse them rather than recomputing them. For $L$ layers, $h$ heads, head dimension $d$, and context length $T$, the cache stores about $2LhTd$ half-precision values. At long contexts, this becomes a dominant memory bottleneck, so compression is operationally central rather than incidental (Su, 6 Apr 2026).

The compression objective is not simply low reconstruction error. Attention quality depends on the scaled inner product

$\langle q, k \rangle / \sqrt{d},$

so a useful compressor must preserve inner-product fidelity as well as matrix fidelity. This is the immediate limitation of applying a purely vectorwise quantizer without accounting for structured correlations inside a KV block. The paper argues that TurboQuant alone is strongest when vectors are approximately uniform on the sphere, whereas KV-cache vectors within a block often share a strong low-dimensional contextual structure. Under that geometry, direct scalar quantization incurs larger MSE, inner-product bias, and additional engineering overhead such as outlier handling or dedicated inner-product bias correction (Su, 6 Apr 2026).

2. Structural model of KV-cache blocks

The central statistical model writes a KV-cache block $\widetilde S \in \mathbb R^{n\times d}$ as

$\widetilde S = S + Z,$

where $3$0 is a low-rank shared-context signal and $3$1 is a full-rank token-specific residual. The signal component is expressed as

$3$2

with singular strengths $3$3, left singular vectors $3$4, and right singular vectors $3$5. The residual is modeled as

$3$6

where $3$7 has independent centered entries, $3$8 captures temporal dependence across tokens, and $3$9 captures coordinate covariance in head space (Su, 6 Apr 2026).

This formulation is a colored-noise, rectangular spiked model. That distinction matters because the noise is not assumed white and the matrix is not assumed square. A plausible implication is that simple white-noise shrinkers or fixed-rank truncation rules are structurally mismatched to the KV-cache regime. The eOptShrinkQ construction instead treats the low-rank component as the recoverable outlier structure and the residual as the object to be made quantization-friendly (Su, 6 Apr 2026).

The paper also reports that the effective rank varies across layers, heads, model families, and input sequences. eOptShrinkQ is therefore designed around automatic rank selection rather than a globally fixed truncation level. This is important because the shared-context geometry is not uniform across the model (Su, 6 Apr 2026).

3. Two-stage compression pipeline

The method consists of two sequential stages.

First, eOptShrink estimates and subtracts the shared low-rank component. Given the SVD

$L$0

the estimator takes the form

$L$1

For Frobenius loss, the asymptotically optimal shrunken value is

$L$2

where

$L$3

The paper states that eOptShrink estimates $L$4, $L$5, and $L$6 from the observed spectrum using the $L$7-transform and a data-driven estimate of the noise spectral distribution $L$8, thereby making the shrinkage automatic (Su, 6 Apr 2026).

Second, the residual

$L$9

is quantized with TurboQuant. For a vector $h$0, the quantization procedure is: compute $h$1 and $h$2, apply a random Haar rotation $h$3 to obtain $h$4, quantize the coordinates with a Lloyd–Max scalar quantizer tuned to $h$5, and reconstruct. The paper distinguishes $h$6, which is MSE-optimal at a given bit-width, from $h$7, which adds a 1-bit QJL correction to remove inner-product bias (Su, 6 Apr 2026).

The defining claim of eOptShrinkQ is that the denoising stage changes the geometry of the object being quantized. By removing the shared low-rank signal before quantization, the residual is made much closer to the isotropic, delocalized regime assumed by TurboQuant. The paper therefore argues that eOptShrinkQ can often omit QJL-style bias correction and reallocate those bits to improved reconstruction (Su, 6 Apr 2026).

4. Random-matrix basis and spectral guarantees

The spectral denoising stage is grounded in the BBP phase transition. If a signal singular value $h$8 is above a threshold $h$9, it creates an outlier singular value separated from the noise bulk and can be recovered; if $d$0, it is buried in the bulk and is effectively unrecoverable. In the paper,

$d$1

where $d$2, with $d$3 and $d$4 the limiting Stieltjes transforms of $d$5 and $d$6. The number of singular values above the estimated bulk edge defines the automatically selected rank $d$7 (Su, 6 Apr 2026).

The theoretical justification includes three guarantees emphasized in the abstract: automatic rank selection via the BBP phase transition, provably near-zero inner-product bias on the residual, and coordinate delocalization ensuring near-optimal quantization distortion. The paper further states that after eOptShrink the residual satisfies the following properties:

1. Residual spectrum matches noise: the spectrum of $d$8 converges to that of $d$9, so no outliers remain.
2. Residual energy matches noise level:

$T$0

1. Inner-product bias is reduced by roughly

$T$1

A corollary gives the row-wise delocalization bound

$T$2

with high probability. This is the precise geometric condition used to justify why scalar quantization should behave near-optimally on the residual (Su, 6 Apr 2026).

A common misconception is that eOptShrinkQ is merely a low-rank truncation scheme preceding quantization. The paper instead frames it as loss-aware spectral denoising: outlier directions are estimated and shrunk, while the non-outlier component is preserved as a full-rank residual to be quantized. Another misconception is that improved MSE alone explains the method’s effect; in fact, the theoretical discussion centers on restoring isotropy and reducing inner-product bias, because those quantities matter directly for attention (Su, 6 Apr 2026).

5. Empirical behavior

The empirical study evaluates eOptShrinkQ on Llama-3.1-8B-Instruct and Ministral-8B-Instruct. The experiments use $T$3, $T$4 blocks, and all layers and heads: $T$5 heads for Llama and $T$6 heads for Ministral. The comparison set includes KIVI, $T$7, $T$8, $T$9, $2LhTd$0, and $2LhTd$1. The SVD factors are quantized at 4 bits (Su, 6 Apr 2026).

At the per-head level, the paper reports relative $2LhTd$2 error, inner-product bias $2LhTd$3 standard deviation, average rank $2LhTd$4, and bits per entry. One representative example is Llama keys at $2LhTd$5: $2LhTd$6 uses 2.35 bits, achieves 17.7\% $2LhTd$7 error, and has inner-product bias $2LhTd$8 with standard deviation 0.014; KIVI uses 2.50 bits, achieves 24.8\% $2LhTd$9, and has bias $\langle q, k \rangle / \sqrt{d},$0 with standard deviation 0.022; $\langle q, k \rangle / \sqrt{d},$1 at 2.00 bits yields 34.1\% $\langle q, k \rangle / \sqrt{d},$2. The paper summarizes this behavior as saving nearly one bit per entry over TurboQuant at equivalent quality (Su, 6 Apr 2026).

On LongBench (16 tasks), $\langle q, k \rangle / \sqrt{d},$3 at about 2.2 bits per entry reaches 47.4 on Llama-3.1-8B and 48.3 on Ministral-8B. The corresponding values for FP16 are 49.0 and 50.5; for $\langle q, k \rangle / \sqrt{d},$4, 44.9 and 46.6; for $\langle q, k \rangle / \sqrt{d},$5, 45.2 and 46.6; and for $\langle q, k \rangle / \sqrt{d},$6, 47.6 and 48.6. The paper therefore states that eOptShrinkQ at $\langle q, k \rangle / \sqrt{d},$7 bits per entry outperforms TurboQuant at 3.0 bits in the end-to-end setting (Su, 6 Apr 2026).

On multi-needle retrieval, the paper reports especially strong behavior. For Llama, $\langle q, k \rangle / \sqrt{d},$8 at 2.22 bits attains 0.981 average recall, compared with 0.972 for FP16 and 0.940 for $\langle q, k \rangle / \sqrt{d},$9. For Ministral, $\widetilde S \in \mathbb R^{n\times d}$0 reaches 0.992, versus 1.000 for FP16 and 0.938 for $\widetilde S \in \mathbb R^{n\times d}$1. The paper suggests that spectral denoising can act as a beneficial regularizer for retrieval-intensive tasks by removing redundant shared structure and improving discrimination among similar tokens (Su, 6 Apr 2026).

The same experiments also qualify the method’s scope. The reported gains are strongest on MultiQA and Code, whereas summarization can be slightly harmed. This suggests that the benefit of shared-context removal is task-dependent rather than universal (Su, 6 Apr 2026).

6. Relation to prior shrinkage methods and known limitations

eOptShrinkQ inherits its denoising logic from the OptShrink line of work. OptShrink is a low-rank matrix denoiser that keeps the empirical singular vectors of the observation matrix but optimally reweights them rather than retaining the noisy singular values. Its central estimator is

$\widetilde S \in \mathbb R^{n\times d}$2

and the 2026 paper explicitly presents eOptShrink as an extension of this perspective to non-square matrices, separable colored covariance, automatic estimation of the bulk edge, and data-driven rank detection in KV-cache blocks (Nadakuditi, 2013). In that sense, eOptShrinkQ is not a generic quantization heuristic but a composition of optimal spectral denoising and residual quantization tailored to the spiked-matrix structure of transformer caches (Su, 6 Apr 2026).

Several limitations are stated explicitly. The method operates on $\widetilde S \in \mathbb R^{n\times d}$3-token blocks, so recent tokens may need to be buffered in FP16 until a block is full. The SVD introduces additional prefill-time computation, although the paper argues that this is comparable to or smaller than attention cost at long contexts. The theory is asymptotic; while the paper states that it matches observed behavior well for $\widetilde S \in \mathbb R^{n\times d}$4–$\widetilde S \in \mathbb R^{n\times d}$5, the assumptions are still approximations for fixed inference inputs. The downstream effect is task-dependent: denoising helps retrieval and code, but can slightly hurt tasks requiring the full shared context, such as summarization. Finally, streaming extensions remain open, so online updates without block buffering are not yet solved (Su, 6 Apr 2026).

Within the broader shrinkage literature, eOptShrinkQ is notable for translating a random-matrix shrinkage principle into an inference-time systems problem. Its distinctive claim is that KV-cache compression should first separate recoverable low-rank context from quantization-ready residual structure. That decomposition, rather than quantization alone, is what underwrites the reported combination of low bit-rate, low bias, and strong retrieval fidelity (Nadakuditi, 2013).