MixKV: KV Cache Compression in LVLMs

• MixKV is a key-value cache compression method for LVLMs that jointly optimizes token importance and semantic diversity under a fixed KV budget.
• It combines extrinsic attention scores with intrinsic value norms and head-wise normalized key diversity to select critical KV pairs.
• Empirical results show MixKV can boost performance by 5–9% on multimodal benchmarks with negligible latency overhead.

Searching arXiv for the cited MixKV-related papers to ground the article in current literature. MixKV most commonly denotes a key-value cache compression method for large vision-LLMs (LVLMs) that jointly optimizes token importance and semantic diversity under a fixed KV budget, with head-wise adaptation to modality-specific redundancy patterns (Liu et al., 23 Oct 2025). In the 2025 literature, the same query term also appears informally in discussion of KVmix, a distinct method for mixed-precision quantization of the KV cache in LLMs; the paper itself explicitly states that the correct name of that method is KVmix (Li et al., 18 May 2025). The shared emphasis across both lines of work is the KV cache as the dominant inference-time memory bottleneck, but the mechanisms differ fundamentally: MixKV selects which KV pairs to retain, whereas KVmix changes the precision at which KV tensors are stored.

1. Nomenclature and disambiguation

The term MixKV is primarily associated with "Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-LLMs" (Liu et al., 23 Oct 2025). That method is plug-and-play, operates during KV cache compression after pre-filling, and preserves the original attention computation except for pruning the retained KV set.

A separate paper, "KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache," notes that “MixKV” is sometimes used informally to denote its mixed-precision KV approach, but it explicitly names the method KVmix (Li et al., 18 May 2025). This distinction matters because the two methods address different compression regimes: one is selection-based, the other quantization-based.

Method	Core operation	Primary setting
MixKV	Top-$B$ selection of KV pairs by joint importance-diversity score	LVLM and extensible to LLMs
KVmix	Layer-specific mixed-precision quantization with RPC retention	LLM inference

A common misconception is that the names are interchangeable. They are not. MixKV in the strict sense refers to the head-wise importance-diversity selection method for KV cache compression in LVLMs, while KVmix refers to gradient-based layer importance-aware mixed-precision quantization.

2. KV cache bottlenecks and redundancy structure

In LVLMs, the LLM’s attention layers store KV pairs computed during the pre-filling stage for all input tokens, including visual tokens from a ViT+projector and text tokens. The decoder reuses this KV cache to avoid recomputation, enabling efficient generation. The inference factorization is

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$

and the cache $\mathcal{C}$ grows linearly with multimodal sequence length $T$ and with the number of layers and heads (Liu et al., 23 Oct 2025). High-resolution images, long videos, and long text therefore produce a memory bottleneck; large KV tensors also increase memory access latency and bandwidth pressure.

MixKV is motivated by the observation that multimodal KV caches differ structurally from text-only caches. Visual tokens often contain repetitive structures such as textures and patches, and the visual-text sequence is interleaved. Some heads focus on fine-grained local vision details, while others focus on global semantics. The resulting redundancy is both modality-specific and head-specific.

The empirical analysis reported for MixKV emphasizes three points. First, keys in Qwen2-VL have denser high-similarity regions than in Qwen2: average key similarity in Qwen2 peaks around $0.2$–$0.4$, while Qwen2-VL peaks around $0.6$–$0.8$, described as a $2$–$3\times$ increase. Second, within LVLMs, different heads can differ sharply in redundancy: in Qwen2-VL-7B and LLaVA-NeXT-Mistral-7B, some heads have average cosine similarity above $$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$0, while others are below $$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$1. Third, importance-only retention methods such as SnapKV and AdaKV can cover only a subset of the full information distribution, especially in high-redundancy heads, which reduces semantic coverage and robustness under tight budgets.

This characterization is specific to the LVLM setting. A plausible implication is that methods designed for text-only long-context pruning may underperform when directly transferred to multimodal caches without accounting for head-wise visual redundancy.

3. MixKV formulation: joint optimization of importance and diversity

MixKV introduces a head-wise adaptive joint optimization that mixes importance with diversity when selecting KV pairs (Liu et al., 23 Oct 2025). The method preserves high-importance pairs but compensates for redundancy by promoting diversity in heads whose keys are semantically similar.

The importance score integrates an extrinsic component derived from attention over an observation window and an intrinsic component derived by default from the value norm. The integrated importance is

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$2

The extrinsic term is the average attention over a prompt-end window, with default length $$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$3:

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$4

The intrinsic term uses VNorm in practice, normalized and scaled to match attention magnitudes:

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$5

Diversity is computed on keys, because keys govern attention patterns. MixKV normalizes keys and compares each key to the head’s global average normalized key:

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$6

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$7

Higher diversity scores correspond to more distinctive keys.

Head-wise redundancy is defined as the off-diagonal average similarity of normalized keys. With

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$8

and

$$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$9

the redundancy coefficient is

$\mathcal{C}$0

MixKV then normalizes and rescales the diversity score and forms the comprehensive selection score

$\mathcal{C}$1

This mixing has no explicit thresholds. As $\mathcal{C}$2, diversity dominates; as $\mathcal{C}$3, importance dominates. The selection step is simply top-$\mathcal{C}$4 retention under budget:

$\mathcal{C}$5

In inherited layer/head budget pipelines, the retained tensors are

$\mathcal{C}$6

The method is modality-agnostic in implementation, but because $\mathcal{C}$7 is computed per head on keys, heads with highly redundant visual semantics naturally receive higher diversity weight.

4. Integration, implementation, and empirical behavior

MixKV operates on the stored KV pairs computed during pre-filling and does not merge or aggregate values (Liu et al., 23 Oct 2025). The selected values are retained at their original positions so that subsequent queries attend only to the retained KV pairs. Integration is intentionally minimal: the baseline compression operator and budget allocation are preserved, and only the per-token scoring function is replaced by the comprehensive score $\mathcal{C}$8.

This makes MixKV compatible with several existing compression pipelines. The paper states that budgets are inherited from the baseline method, including SnapKV’s cluster budgets, AdaKV’s head-wise budgets, PyramidKV’s layer-wise pyramidal budgets, and SparseMM’s asymmetric head budgets. The default observation window length is $\mathcal{C}$9; online per-sample head weights computed from the current sample are the default and recommended option, while offline head weights computed on a dataset such as OCRBench are also possible and yield slightly smaller gains.

The computational overhead is reported as negligible. Diversity computation is linear in $T$0, redundancy uses the same normalized keys and averages, and experiments report less than $T$1 latency increase relative to baselines, with no measurable rise in peak memory. The codebase is provided at https://github.com/xuyang-liu16/MixKV.

The strongest reported gains occur under tight budgets. Under extreme compression with budget = 64, MixKV improves baseline methods by an average of $T$2 across five multimodal understanding benchmarks (Liu et al., 23 Oct 2025). On LLaVA-NeXT-Mistral-7B at budget $T$3, SnapKV improved to MixKV as follows: DocVQA $T$4, OCRBench $T$5, TextVQA $T$6, ChartQA $T$7, and TextCaps $T$8. On Qwen2-VL-7B-Instruct at the same budget, SnapKV improved to MixKV on DocVQA $T$9, OCRBench $0.2$0, TextVQA $0.2$1, ChartQA $0.2$2, and TextCaps $0.2$3.

For GUI grounding on ScreenSpot-v2 using Qwen2.5-VL-7B-Instruct, the reported gains are larger. With SnapKV, the average improves from $0.2$4 to $0.2$5 at budget $0.2$6 and from $0.2$7 to $0.2$8 at budget $0.2$9, corresponding to gains of $0.4$0 and $0.4$1. With AdaKV, the averages improve from $0.4$2 to $0.4$3 at budget $0.4$4 and from $0.4$5 to $0.4$6 at budget $0.4$7, corresponding to gains of $0.4$8 and $0.4$9.

Ablation results constrain how the method should be interpreted. Combining extrinsic attention with intrinsic VNorm outperforms combining with KNorm. Diversity-only selection degrades performance; for LLaVA-NeXT at budget $0.6$0, the diversity-only setting yields DocVQA $0.6$1 and OCRBench $0.6$2, versus the baseline’s $0.6$3 and $0.6$4. Mixing importance and diversity yields gains, and adding head-wise adaptation improves further, with online per-sample redundancy estimates slightly better than offline statistics. MixKV also extends to text-only LLMs on LongBench, where summarization and information aggregation tasks benefit more consistently than information localization tasks.

5. KVmix: the mixed-precision method sometimes informally called “MixKV”

KVmix is a different method with a different mechanism and should not be conflated with MixKV (Li et al., 18 May 2025). Rather than selecting which KV pairs to keep, KVmix quantizes the KV cache with layer-specific bit-widths determined by gradient-based importance analysis, and retains a recent full-precision subset through a dynamic long-context mechanism called Recent Pivotal Context (RPC).

The method begins from the observation that different layers and Key versus Value projections contribute unequally to loss. For layer $0.6$5, with projection matrices $0.6$6 and $0.6$7, KVmix profiles a full-precision model offline on a small set of prompts and computes importance by Frobenius norms of gradients:

$0.6$8

KV memory is expressed approximately as

$0.6$9

The practical allocation heuristic ranks layers by $0.8$0 and $0.8$1 separately and assigns higher precision to the top $0.8$2 layers, independently for K and V. The default is $0.8$3: Keys use $0.8$4 bits and Values use $0.8$5 bits, producing average precisions of $0.8$6 bits for Keys and $0.8$7 bits for Values. Increasing the high-bit fraction to $0.8$8 yields approximately $0.8$9 bits for Keys and $2$0 bits for Values.

The quantization scheme is asymmetric group-wise min-max quantization, but with different grouping for K and V. Keys are quantized per-channel across time by reshaping $2$1 to $2$2. Values are quantized per-token across channels. Quantized integers are packed into int32; for $2$3-bit quantization, KVmix uses an 11-element block with one int32, where the first ten elements use $2$4 bits and the eleventh uses $2$5 bits, described as increasing packing density by about $2$6 relative to the usual $2$7-bit packing.

RPC is recency-based. Layers assigned a high bit-width retain a larger recent full-precision fraction: for Keys, $2$8 if $2$9 and $3\times$0 otherwise; for Values, $3\times$1 if $3\times$2 and $3\times$3 otherwise. The method then retains the most recent $3\times$4 Keys and $3\times$5 Values at full precision for each layer.

The reported empirical profile is that of near-lossless quantized inference. On Llama and Mistral models, KVmix-k2.19v2.38 is reported to achieve approximately $3\times$6 reduction in dynamic peak runtime memory and up to approximately $3\times$7 throughput speedup relative to FP16, while sustaining batch size $3\times$8 with 1032 tokens/s on an RTX 4090. On LongBench across eight tasks and four models, the average accuracy loss is reported as approximately $3\times$9 versus FP16; on Wikitext-2 with Llama 2-7B-hf, perplexity is 8.71, the same as FP16; and on GSM8K with Llama 2-7B-hf, accuracy is 13.25\% versus 13.52\% for FP16.

6. Limitations, task dependence, and open directions

For MixKV, the principal limitation is that overemphasis on diversity can harm tasks requiring precise local retrieval, particularly information localization in text-only settings (Liu et al., 23 Oct 2025). The paper reports that diversity-only selection degrades performance and that head-wise adaptation only mitigates, rather than eliminates, occasional regressions. Gains can also be smaller or occasionally negative when the baseline already uses head-aware allocation, as noted for some SparseMM settings. Additional stated limitations are that experiments were conducted up to 8B-scale models, that generalization to 70B+ should be validated, and that the method does not use explicit pairwise diversity such as DPP, relying instead on a global-average proxy for efficiency.

For KVmix, the main limitations are different (Li et al., 18 May 2025). Its importance analysis is offline and depends on representative profiling prompts; heavy deployment distribution shift can make the importance ranking suboptimal. The paper also notes that very small profiling sets can be noisy, that high RPC ratios can still saturate memory in extremely long contexts, that the method has no per-head granularity, and that its specialized kernels for $$p\left(\mathbf{Y} \mid \mathbf{F}^{v}, \mathbf{F}^{t}\right)=\prod_{j=1}^L p\left(\mathbf{y}_j \mid \mathbf{F}^{v}, \mathbf{F}^{t}, \mathbf{Y}_{1:j-1}; \mathcal{C}\right),$$00 bits with int32 packing may require additional engineering for other backends or arbitrary bit-widths.

Taken together, the two methods define complementary directions in KV-cache optimization. MixKV addresses which KV positions to preserve under a pruning budget, with an explicit model of redundancy and semantic coverage. KVmix addresses how precisely different KV tensors should be stored, with layer-aware bit allocation and a recency-based full-precision residual. A plausible implication is that future work may combine head-wise redundancy-aware selection with layer-aware mixed-precision quantization, especially in multimodal long-context inference where both token retention and storage precision directly determine memory pressure and downstream accuracy.