SpectrumKV: Mixed-Precision KV Cache Transfer

• SpectrumKV is a key-value cache transfer method that reformulates KV movement as a per-token mixed-precision allocation problem rather than a binary keep/drop decision.
• It assigns each token a precision tier—FP16 for high-importance, INT8 for medium, and INT4 for low-importance—under a strict budget, using an attention-derived importance score.
• Empirical results show significant improvements in retrieval performance and transfer latency across various models while adapting dynamically to model-specific INT4 tolerance.

Searching arXiv for SpectrumKV and related KV-cache transfer/compression papers. SpectrumKV is a key-value (KV) cache transfer method for prefill–decode (PD) disaggregated LLM serving that treats network-constrained KV movement as a per-token mixed-precision allocation problem rather than a binary keep/drop decision. In this formulation, each token’s KV vectors are assigned a precision tier—FP16, INT8, or, when safe, INT4—according to estimated importance, so that low-importance tokens contribute approximate information instead of being omitted entirely. The method is introduced in “SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving” (Pengju, 7 Jun 2026).

1. Problem setting and conceptual reframing

In PD disaggregation, prompt processing is decoupled from token generation, and the KV cache produced during prefill must be transmitted over the network to the decode worker. For long contexts, that payload can dominate bandwidth and latency. Earlier PD-side KV reduction methods are described as mostly binary: selected tokens are transmitted at full precision and the rest are not transmitted. The paper positions PDTrim, StreamingLLM, and SnapKV as examples of this keep/drop framing, and argues that dropping a token destroys all of its information, which is particularly harmful for retrieval-heavy settings because a dropped “needle” token cannot be recovered without an expensive on-demand fetch (Pengju, 7 Jun 2026).

SpectrumKV reformulates this setting as a “precision spectrum.” High-importance tokens are preserved at FP16, medium-importance tokens are transmitted at INT8, and low-importance tokens are transmitted at INT4 when the model can tolerate that quantization level. The analogy drawn in the paper is to multi-tier memory systems such as RAM, SSD, and disk, with INT4 functioning as a highly compressed but still usable tier.

A central implication of this reframing is methodological rather than merely implementational. The paper’s argument is that PD KV transfer should be interpreted as an allocation problem over heterogeneous precision levels, not only as token pruning. This directly challenges the assumption that transfer reduction and token omission are equivalent objectives.

2. Precision-allocation formulation

SpectrumKV assigns each token position $j$ an importance score

$I_j = A_j e^{-\lambda \Delta_j},$

where $A_j$ is an attention-derived statistic, $\Delta_j = n-j$ is recency, and $\lambda$ is a decay rate. A small sink prefix of the earliest tokens is pinned to FP16 unconditionally, reflecting the role of attention sinks as a persistent hot set. The remaining tokens are sorted in descending order of $I_j$ and then assigned to precision tiers under a budget constraint (Pengju, 7 Jun 2026).

The transfer-cost model is normalized so that

$c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$

If token $j$ is assigned precision $q_j \in \{16,8,4\}$, the normalized transfer budget is

$b = \frac{1}{n}\sum_{j=1}^n c(q_j),$

so that $I_j = A_j e^{-\lambda \Delta_j},$0 corresponds exactly to an all-INT8 policy. Full-FP16 payload size is given as

$I_j = A_j e^{-\lambda \Delta_j},$1

where $I_j = A_j e^{-\lambda \Delta_j},$2 is the number of layers, $I_j = A_j e^{-\lambda \Delta_j},$3 is the number of KV heads, $I_j = A_j e^{-\lambda \Delta_j},$4 is head dimension, and $I_j = A_j e^{-\lambda \Delta_j},$5 bytes; the factor of 2 accounts for both keys and values.

The paper formulates the assignment as a weighted-error minimization:

$I_j = A_j e^{-\lambda \Delta_j},$6

where $I_j = A_j e^{-\lambda \Delta_j},$7 is a perturbation bound for tier $I_j = A_j e^{-\lambda \Delta_j},$8. At decode time, an attention head computes

$I_j = A_j e^{-\lambda \Delta_j},$9

If $A_j$0 is quantized to $A_j$1 with $A_j$2, then

$A_j$3

This bound shows that error scales with the attention mass carried by each tier.

For budgets in the two operating intervals described in the paper, the greedy budget-to-tier mapping is piecewise:

• If $A_j$4,

$A_j$5

• If $A_j$6,

$A_j$7

In practice, SpectrumKV first reserves $A_j$8 sink tokens at FP16 and then applies the mix to the remaining $A_j$9 tokens.

3. Three-tier policy and deployment-time adaptation

The default SpectrumKV policy is a three-tier mixed-precision scheme: FP16 for hot tokens, INT8 for warm tokens, and INT4 for cold tokens. The principal practical complication is that INT4 tolerance is model-dependent. The paper states that Qwen2.5-7B catastrophically fails under INT4 KV quantization, while Mistral-7B and Gemma-2-9B remain stable. The stated mechanism is softmax amplification when attention is strongly local (Pengju, 7 Jun 2026).

To address this, SpectrumKV introduces a one-time lightweight deployment-time probe per model. The procedure is explicit:

1. Configure a highly aggressive 3-tier greedy policy at $\Delta_j = n-j$0, so that approximately 75% of tokens would be INT4.
2. Run three needle-in-a-haystack retrieval trials at sequence length 4096.
3. If at least $\Delta_j = n-j$1 trials succeed, declare the model INT4-tolerant and enable full 3-tier operation for all budgets $\Delta_j = n-j$2. Otherwise, disable INT4 and fall back to a 2-tier FP16+INT8 policy.

Algorithmically, the pipeline is summarized as: probe once per model; compute $\Delta_j = n-j$3; pin the sink prefix to FP16; sort remaining tokens by importance; greedily allocate FP16, INT8, and optionally INT4 according to budget; quantize each token’s keys and values to its assigned precision; and transmit the mixed-precision payload.

This adaptive structure serves two roles. First, it operationalizes model-specific quantization tolerance without retraining. Second, it converts INT4 from a universal assumption into a guarded capability. A common misconception corrected by the paper is therefore that lower KV precision is uniformly usable across models.

4. Empirical behavior

The paper evaluates quality and latency on Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-it. Perplexity change relative to full FP16 is reported as

$\Delta_j = n-j$4

At a 50% normalized KV budget on WikiText-2 with sequence length 2048, PDTrim yields $\Delta_j = n-j$5, $\Delta_j = n-j$6, and $\Delta_j = n-j$7 $\Delta_j = n-j$8 on Qwen2.5-7B, Mistral-7B, and Gemma-2-9B, respectively, whereas SpectrumKV Greedy yields $\Delta_j = n-j$9, $\lambda$0, and $\lambda$1, and SpectrumKV SinkProtect yields $\lambda$2, $\lambda$3, and $\lambda$4 (Pengju, 7 Jun 2026).

The budget sweep further differentiates model behavior. At $\lambda$5, SpectrumKV Greedy in 3-tier mode is unsafe on Qwen, with perplexity blowing up, but records $\lambda$6 on Mistral and $\lambda$7 on Gemma, versus $\lambda$8 and $\lambda$9 for PDTrim. At $I_j$0, Greedy remains within $I_j$1 for Mistral and Gemma, and within $I_j$2 for Qwen when the 2-tier adaptive policy is used for Qwen at $I_j$3.

On NIAH retrieval at sequence length 4096, the adaptive policy reaches $I_j$4 on Qwen at the aggressive $I_j$5 budget versus $I_j$6 for PDTrim, and reaches $I_j$7 by $I_j$8. For Mistral, PDTrim records $I_j$9, $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$0, $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$1, and $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$2 retrieval at $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$3, while SpectrumKV Greedy records $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$4 at all four budgets and SpectrumKV Adaptive records $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$5 at all four budgets. For Gemma, PDTrim records $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$6, $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$7, $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$8, and $c(16)=1,\qquad c(8)=\tfrac12,\qquad c(4)=\tfrac14.$9, while SpectrumKV Greedy records $j$0 at all four budgets. The paper summarizes this as Mistral and Gemma preserving retrieval under the 3-tier policy.

End-to-end GPU timing of the transfer path at $j$1 shows substantial TTFT reductions versus full FP16. Reported values include Qwen2.5-7B at 2K, $j$2 ms $j$3 ms ($j$4); Qwen2.5-7B at 4K, $j$5 ms $j$6 ms ($j$7); Mistral-7B at 2K, $j$8 ms $j$9 ms ($q_j \in \{16,8,4\}$0); Mistral-7B at 4K, $q_j \in \{16,8,4\}$1 ms $q_j \in \{16,8,4\}$2 ms ($q_j \in \{16,8,4\}$3); Mistral-7B at 8K, $q_j \in \{16,8,4\}$4 ms $q_j \in \{16,8,4\}$5 ms ($q_j \in \{16,8,4\}$6); Gemma-2-9B at 2K, $q_j \in \{16,8,4\}$7 ms $q_j \in \{16,8,4\}$8 ms ($q_j \in \{16,8,4\}$9); Gemma-2-9B at 4K, $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$0 ms $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$1 ms ($b = \frac{1}{n}\sum_{j=1}^n c(q_j),$2); and Gemma-2-9B at 8K, $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$3 ms $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$4 ms ($b = \frac{1}{n}\sum_{j=1}^n c(q_j),$5).

5. Position within KV-cache research

SpectrumKV addresses PD-side KV transfer reduction through per-token mixed precision. Other contemporaneous work in the broader KV-cache literature attacks different bottlenecks. KVSlimmer, for example, studies asymmetric KV merging rather than transfer-time precision allocation. It derives a theoretical account of query/key homogeneity and value heterogeneity through the spectral energy distribution of projection weights, and introduces a gradient-free closed-form merging procedure using exact Hessian information represented through forward-pass variables (Liu et al., 1 Mar 2026).

A second related line is end-to-end KV compression by spectral denoising plus quantization. eOptShrinkQ models per-head KV blocks using a low-rank shared-context component plus a full-rank per-token residual under a spiked random matrix model, applies optimal singular-value shrinkage to extract the shared structure, and then quantizes the residual with TurboQuant. The paper reports that, on LongBench, eOptShrinkQ at approximately $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$6 bits per entry outperforms TurboQuant at $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$7 bits and that multi-needle retrieval at $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$8 bits can closely match or exceed uncompressed FP16 (Su, 6 Apr 2026).

These comparisons indicate that SpectrumKV occupies a distinct design point. It does not merge adjacent KV pairs as in KVSlimmer, nor does it first denoise and then uniformly quantize residuals as in eOptShrinkQ. Instead, it operates at PD transfer time and allocates precision non-uniformly across tokens based on importance and model-specific INT4 tolerance. This suggests a broader research trend toward structure-aware KV optimization, with token importance, spectrum, and deployment constraints treated as coupled variables rather than isolated heuristics.

6. Trade-offs, limitations, and prospective extensions

The trade-offs stated in the paper are direct. INT4 yields the greatest byte savings but requires a probe for safety. Importance ranking is crucial at moderate budgets. At extremely low budgets below $b = \frac{1}{n}\sum_{j=1}^n c(q_j),$9, the random spread of INT4 can occasionally outperform greedy assignment, which the paper presents as evidence that more refined per-head or per-layer scoring may be beneficial. Sink-protection pins early tokens to FP16 and yields small but consistent perplexity gains (Pengju, 7 Jun 2026).

The listed limitations are also specific. Full production integration is left to future work, including a vLLM-style scheduler, network contention, continuous batching, and cold-tier fetch policies. Quality experiments cover three 7B–9B models, and the INT4 tolerance boundary may shift for larger models such as 70B+. The lightweight probe tests retrieval only; a more comprehensive probe could include code generation and domain-specific prompts.

The paper identifies several extensions: per-layer or per-head precision allocation; combination with uniform quantization schemes such as KVQuant and KIVI within each tier; integration with lossless compression such as SplitZip, per-layer placement such as OrbitFlow, and KV reuse such as CacheBlend; and dynamic on-demand upgrade, where INT4 $I_j = A_j e^{-\lambda \Delta_j},$00 FP16 cold-tier fetches cost only $I_j = A_j e^{-\lambda \Delta_j},$01 bytes per element rather than $I_j = A_j e^{-\lambda \Delta_j},$02 bytes if the token had been dropped entirely.

Taken together, these trade-offs clarify the scope of SpectrumKV. It is neither a universal endorsement of INT4 nor a claim that mixed-precision transfer subsumes all other KV optimizations. Rather, it provides evidence that in PD-disaggregated serving, the KV payload can be managed more effectively when precision is allocated per token under an explicit budget and guarded by a deployment-time tolerance probe.