MixKVQ: Query-Aware KV Cache Quantization

• MixKVQ is a query-aware mixed-precision KV cache quantization method that allocates precision based on both quantization difficulty and query relevance.
• It employs per-channel mixed precision for keys and uniform low-bit quantization for values to alleviate memory and bandwidth constraints during long-context inference.
• Empirical evaluations show that MixKVQ nearly matches BF16 performance at effective precisions of 2.3–2.7 bits while enabling larger batch sizes and higher throughput.

MixKVQ is a training-free, query-aware mixed-precision KV cache quantization method for long-context and long Chain-of-Thought (CoT) reasoning in autoregressive LLM inference. Introduced by Tao Zhang, Ziqian Zeng, Hao Peng, Huiping Zhuang, and Cen Chen, it addresses the memory and bandwidth cost of the Key-Value cache by combining per-channel mixed precision for keys with uniform low-bit per-token quantization for values, while using a lightweight query-aware criterion to decide which key channels should remain in higher precision (Zhang et al., 22 Dec 2025). Its central claim is that effective low-bit KV compression must jointly account for intrinsic quantization difficulty and query relevance; under this design, the method reports reasoning performance close to BF16 at effective precisions around 2.3–2.7 bits, together with larger feasible batch sizes and higher throughput (Zhang et al., 22 Dec 2025).

1. Problem setting and formulation

MixKVQ is situated in the standard autoregressive decoding regime in which every newly generated token appends one key and one value per layer and head to the KV cache. The method is motivated by two coupled bottlenecks. First, KV cache size grows linearly with sequence length, which becomes acute for long-context and long-CoT generation. Second, decoding repeatedly reads the growing cache, so inference becomes memory-bandwidth-bound rather than purely compute-bound (Zhang et al., 22 Dec 2025).

The supplied manuscript gives a concrete scale example: a 32B Qwen2.5 model with batch size 64 and sequence length 32,768 tokens requires about 512 GB of GPU memory for KV cache, which is 8.59× larger than the memory used to store model weights (Zhang et al., 22 Dec 2025). This is the deployment regime MixKVQ targets: long reasoning traces, large active batch sizes, and KV traffic dominating decode-time cost.

The paper adopts the usual asymmetric distinction between Keys and Values. Keys are treated as more sensitive because they perturb the pre-softmax attention logits,

$$\mathbf{a}_i^l = \mathrm{softmax}\left(\frac{\mathbf{Q}_i^l (\mathbf{K}^l)^\top}{\sqrt{D}}\right),$$

whereas Values affect the post-attention output,

$\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$

In the reported implementation, Keys are quantized per-channel, Values per-token, with group size $G=32$ and residual length $R=128$ (Zhang et al., 22 Dec 2025). The paper’s broader claim is that low-bit degradation in reasoning tasks is driven primarily by a small subset of problematic key channels with outlier-induced error, rather than by uniformly distributed quantization noise across the entire cache (Zhang et al., 22 Dec 2025).

2. Query-aware salience criterion

The defining idea of MixKVQ is that a key channel should receive high precision only when two conditions hold simultaneously: it is hard to quantize and it is important to the current query (Zhang et al., 22 Dec 2025). This is presented as a correction to two earlier tendencies: fixed-precision quantization, which cannot accommodate outlier channels, and prior mixed-precision heuristics that rank channels only by scale or quantization difficulty.

The paper formalizes the attention-logit error from key quantization as

$$\mathbf{E}_{attn} = \mathbf{Q}(\mathbf{K} - \tilde{\mathbf{K}})^T,$$

with elementwise contribution

$$E_{i,j} = \sum_{d=1}^{D} \mathbf{Q}_{i,d}\cdot \epsilon_{j,d},$$

where $\epsilon_{j,d}$ is the quantization noise in key channel $d$ (Zhang et al., 22 Dec 2025). This motivates a channel salience score built from two terms.

The importance score is the average absolute query activation on channel $d$,

$$\mathcal{I}_d = \frac{1}{L_q}\sum_{i=1}^{L_q} |\mathbf{Q}_{i,d}|,$$

and the sensitivity score is the channel scale,

$\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$0

MixKVQ multiplies them to obtain

$\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$1

Channels with large $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$2 are therefore both attention-relevant and quantization-sensitive (Zhang et al., 22 Dec 2025).

Two empirical observations support this formulation. First, the paper reports that the Pearson correlation between query magnitude and key scale is only 0.16, so scale alone is a weak proxy for actual attention influence. Second, key scales are said to be densely clustered; for example, 80% of values in Head 0 lie in $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$3, which further limits the discriminative value of scale-only ranking (Zhang et al., 22 Dec 2025). This suggests that channel protection based solely on outlier magnitude allocates precision inefficiently.

The paper’s strongest direct validation is a query-awareness ablation on AIME 2024-2025. Replacing the full salience rule $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$4 with an error-only baseline $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$5 lowers performance from 60.00 to 53.33 on R1-Qwen-14B, and from 40.00 to 33.33 on R1-Llama-8B (Zhang et al., 22 Dec 2025). This suggests that in long-form reasoning, preserving channels that are merely hard to quantize is insufficient; what matters is preserving the channels that are hard to quantize and currently active in the query-conditioned attention path.

3. Quantization policy and runtime mechanism

MixKVQ uses a three-tier precision policy for Keys and a simpler uniform low-bit policy for Values (Zhang et al., 22 Dec 2025). For each key channel, two thresholds determine the assigned precision:

• BF16 if $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$6
• UINT4 if $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$7
• UINT2 if $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$8

Values are quantized uniformly with 2-bit per-token quantization (Zhang et al., 22 Dec 2025). The paper explicitly motivates this asymmetry by arguing that Value-cache error is empirically more uniform and less dominated by pathological outlier channels, whereas Key-cache error is heterogeneous and concentrated in a small number of channels.

Operationally, the method maintains a full-precision residual buffer of length $\mathbf{o}_i^l = \mathbf{a}_i^l \mathbf{V}^l.$9. Newly generated KV states are first stored in this residual buffer, and channel salience is updated only every $G=32$0 tokens. When the buffer fills, MixKVQ computes the per-channel statistics, assigns key precision, quantizes the block, and packs it into structured storage consisting of low-bit quantized cache, sparse BF16 outlier channels, and the residual buffer (Zhang et al., 22 Dec 2025). The paper characterizes this as a lazy update strategy rather than per-token online rescoring.

The method includes implementation-specific accommodations for common transformer variants. For Grouped Query Attention (GQA), importance scores are computed at the KV head group level by aggregating query magnitudes from the query heads associated with the same KV head. For models using RoPE, salience is computed after rotary embedding transformation, so channel relevance is measured in the rotated attention space actually used by decoding (Zhang et al., 22 Dec 2025).

Although MixKVQ is described as plug-and-play and without tuning, the supplied appendix description includes an Optuna threshold search over $G=32$1 and $G=32$2 in $G=32$3, using 30 trials with a TPE sampler, jointly optimizing GSM8K accuracy and effective average bit-width (Zhang et al., 22 Dec 2025). The paper’s own phrasing therefore supports a narrower interpretation of “plug-and-play”: it is training-free and fine-tuning-free, but not necessarily configuration-free.

4. Empirical performance and efficiency

The principal reported result is that MixKVQ maintains reasoning performance close to BF16 at effective bit-widths that are substantially below 4-bit fixed precision (Zhang et al., 22 Dec 2025). On DeepSeek-R1-Distill-Llama-8B, the average score is 53.77 for BF16 and 51.89 for MixKVQ-C2.7. On DeepSeek-R1-Distill-Qwen-14B, BF16 averages 64.73, whereas MixKVQ-C2.3 reaches 63.10. On DeepSeek-R1-Distill-Qwen-32B, BF16 averages 67.84, while MixKVQ-C2.3 reaches 66.04 (Zhang et al., 22 Dec 2025). In the same tables, MixKVQ consistently exceeds KIVI KV2, KVQuant KV2, and KVTuner under comparable low-bit settings, and also improves over RotateKV KV4 on the reported reasoning averages (Zhang et al., 22 Dec 2025).

On LongBench, the method is presented as nearly lossless at 2.7 effective bits. For Mistral-7B-Instruct-v0.3, BF16 averages 53.84, while MixKVQ C2.7 reaches 53.68. For Llama-3.1-8B-Instruct, BF16 averages 54.00, while MixKVQ C2.7 reaches 53.71 (Zhang et al., 22 Dec 2025). The paper uses these results to argue that long-context understanding tolerates the proposed mixed-precision policy with minimal degradation, provided that key-channel protection is query-aware.

A complementary perplexity simulation on Qwen2.5-7B-Instruct is used to isolate the role of Key precision. The reported progression from KIVI-K4V2 to KIVI-K2V4 shows that lowering Key precision hurts more severely than lowering Value precision: on WikiText2, perplexity rises from 6.81 to 8.13, and on C4 from 2.77 to 3.15 (Zhang et al., 22 Dec 2025). This is consistent with MixKVQ’s design choice to spend most of the precision budget on key-channel selectivity rather than on value-side complexity.

The paper also reports concrete serving benefits on Llama2-13B-chat in ShareGPT-style workloads. Under similar maximum memory usage, MixKVQ allows up to 2.25× larger batch size and achieves 2.63× to 2.81× higher throughput relative to FP16 (Zhang et al., 22 Dec 2025). Runtime overhead from the query-aware channel-selection logic is described as small: for R1-Qwen-7B, Channel Selection accounts for 2.17% of per-layer execution time, compared with 64.62% for Attention and 33.21% for the MLP, and channel selection is invoked on only 3.13% of decode steps because of the lazy $G=32$4-token update schedule (Zhang et al., 22 Dec 2025).

5. Position within KV-cache compression research

MixKVQ belongs to a broader lineage of KV-cache compression methods, but it differs in the axis along which it allocates precision. Earlier work such as QAQ argued that Keys and Values exhibit distinct sensitivities and proposed quality-adaptive quantization with separate strategies for K and V, together with attention-aware bit allocation and outlier handling (Dong et al., 2024). KVmix shifted the focus to gradient-based layer importance, assigning higher precision to layers whose Key and Value projection matrices have larger loss gradients and supplementing this with a dynamic Recent Pivotal Context retention rule (Li et al., 18 May 2025). PM-KVQ extended mixed precision to long-CoT by introducing progressive quantization, block-wise memory allocation, and positional interpolation for RoPE-aware calibration (Liu et al., 24 May 2025).

Other nearby work addresses low-bit KV compression through different mechanisms rather than query-aware key-channel selection. RotateKV targets 2-bit KV quantization by using Outlier-Aware Rotation, Pre-RoPE Grouped-Head Rotation, and Attention-Sink-Aware Quantization, reporting less than 0.3 perplexity degradation on WikiText-2 for LLaMA-2-13B and strong long-context behavior at 2-bit precision (Su et al., 25 Jan 2025). CommVQ replaces scalar low-bit quantization with Commutative Vector Quantization, using a RoPE-commutative key codebook and additive quantization to reach 87.5% KV-cache reduction at 2-bit and viable 1-bit operation for long-context LLMs (Li et al., 23 Jun 2025).

Against that background, MixKVQ is best understood as a method that preserves the standard scalar quantization setting while making its mixed-precision allocation explicitly query-conditioned. It does not redesign the KV representation as vector quantization does, nor does it progressively change precision over time in the manner of PM-KVQ. Its novelty is narrower and more specific: mixed precision is assigned to key channels according to an online proxy for their expected effect on current attention-logit error (Zhang et al., 22 Dec 2025).

A plausible implication is that these lines of work are not mutually exclusive. The data support the view that KV compression design space spans at least four orthogonal axes: query awareness, layer sensitivity, progressive temporal scheduling, and representation choice. MixKVQ occupies the first axis most directly.

6. Limitations, caveats, and interpretation

The supplied manuscript identifies several limitations. It explicitly states that the study does not cover all attention architectures, especially Multi-Head Latent Attention (MLA), and that its latency analysis concentrates on memory-bound generation/decode rather than prompt-processing bottlenecks such as batch compression during prefill (Zhang et al., 22 Dec 2025). It also notes residual runtime overhead from tensor transformation, even though the GPU-optimized implementation keeps the channel-selection cost small (Zhang et al., 22 Dec 2025).

A second caveat concerns configurability. Although MixKVQ is framed as training-free and plug-and-play, the appendix-level threshold search indicates that architecture-specific configuration remains important (Zhang et al., 22 Dec 2025). This suggests that “plug-and-play” should be interpreted operationally as no retraining or fine-tuning, not as a guarantee that one pair of thresholds transfers unchanged across all models and workloads.

A third limitation is scope. The reported strength of MixKVQ is clearest on long-context and reasoning-oriented benchmarks, where small perturbations in attention can corrupt long dependency chains (Zhang et al., 22 Dec 2025). The paper does not claim a universal advantage on every inference workload; rather, it argues that fixed precision becomes unreliable when contexts are long, reasoning traces are extended, and KV memory is the dominant deployment constraint.

In that sense, MixKVQ’s historical significance lies in sharpening one particular thesis in KV-cache research: precision should be spent where the query is actually using the cache, not merely where the cache appears difficult to quantize in isolation. The method’s salience score,

$G=32$5

is a compact expression of that thesis. Within the literature summarized here, it marks a shift from static low-bit KV compression toward online, query-conditioned precision allocation for long-context reasoning (Zhang et al., 22 Dec 2025).