SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving

Abstract: Prefill-decode (PD) disaggregation decouples prompt processing from token generation, but it also turns the key-value (KV) cache into a network payload. Existing PD-side KV reduction methods are mostly binary: selected tokens are transmitted at full precision and the rest are not transmitted. This paper argues that binary selection leaves a useful design space unused. SpectrumKV assigns a precision level to each token instead: attention sinks and other high-importance tokens are protected at FP16, medium-importance tokens are sent at INT8, and low-importance tokens are sent at INT4 when the model can tolerate it. The main practical complication is that INT4 tolerance is model-dependent. Qwen2.5-7B catastrophically fails under INT4 KV quantization, while Mistral-7B and Gemma-2-9B remain stable. SpectrumKV therefore runs a lightweight deployment-time probe: three aggressive NIAH trials under a 3-tier policy. Models that pass use FP16+INT8+INT4; models that fail fall back to FP16+INT8. Across Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-it, SpectrumKV improves quality at the same transfer budget. At a 50% normalized KV budget on WikiText-2, SpectrumKV changes perplexity by +1.97%,-0.06%, and-0.44%, respectively, compared with PDTrim's +25.85%, +22.07%, and +35.63%. On NIAH retrieval at 4096 tokens, the adaptive policy reaches 52.6% on Qwen at the aggressive b=0.3 budget versus 26.3% for PDTrim, and reaches 100% by b=0.5; Mistral and Gemma preserve retrieval under the 3-tier policy. End-to-end GPU timing of the transfer path shows 50-62% TTFT reductions at b=0.5. These results suggest that PD KV transfer should be treated as a precision-allocation problem, not only as token pruning.

• The paper demonstrates that per-token mixed-precision allocation outperforms binary token pruning, preserving model quality with reduced bandwidth.
• It introduces a dynamic scoring system that assigns precision tiers (FP16, INT8, INT4) based on token importance and model tolerance, ensuring safe quantization.
• Empirical evaluations on multiple LLMs reveal improved perplexity, retrieval accuracy, and up to 62% reduction in transfer latency compared to traditional methods.

SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill–Decode Disaggregated LLM Serving

Problem Formulation and Motivation

SpectrumKV addresses the communication bottleneck of Key/Value (KV) cache transfer in prefill–decode (PD) disaggregated serving of LLMs. Under PD separation, prompt (prefill) and token generation (decode) are decoupled for resource efficiency, but the KV cache produced during prefill often has prohibitive transfer costs, especially with long contexts. Prior works mitigate this by selection-based schemes: only a subset of critical tokens is transmitted (often judged by static or attention-based importance metrics), typically at full precision, with the rest discarded. However, the binary retain/drop decision fails to utilize the design space where intermediate precision levels can offer significant bandwidth reduction without total information loss.

SpectrumKV formulates the KV transfer as a per-token precision allocation and argues—both theoretically and empirically—that this approach achieves strictly superior tradeoffs compared to hard token pruning. The core insight is that the degradation by quantizing a token is graceful and bounded, whereas dropping it deletes all potential contribution. This is formalized by error bounds showing perturbation to the output scales with attention mass on a quantized tier, and that models with highly concentrated attention—low entropy—are intrinsically more sensitive to quantization error due to softmax amplification.

SpectrumKV Policy and System Architecture

The SpectrumKV pipeline scores each token for importance (mixing attention statistics and recency), then assigns precision tiers (FP16, INT8, optionally INT4) under a normalized bandwidth budget, protecting high-importance tokens. Token scoring is parameterized and supports fallback to KV-norm when direct attention measurements are unavailable. A lightweight deployment-time probe determines whether a given model tolerates INT4 quantization for low-importance tokens; if not, the system falls back to safer 2-tier (FP16/INT8) operation.

Key system design choices are:

• Per-token tiered quantization: Transmits every token, but at a policy-selected precision. High-importance (sink, recent) tokens receive FP16; medium, INT8; lowest, INT4 if model is tolerant.
• Cold-tier fetch efficiency: In selection methods, previously dropped tokens require full-precision transfer for cold-tier fetch; in SpectrumKV, low-precision tokens are upgraded at reduced bandwidth, and information is partially available from first access.
• Pre-transfer quantization: Quantization occurs at the prefill worker, and only dequantization is required on the decode side, reducing runtime overhead and network costs.

Empirical Evaluation

Comprehensive experiments cover Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-it, evaluating perplexity (PPL), needle-in-a-haystack (NIAH) retrieval, and transfer path latency under varying budgets and context lengths.

Numerical Results

Quality at Fixed Budget: At $b = 0.5$, SpectrumKV Greedy mode (all INT8 quantization, mixed-precision assignment irrelevant) yields minimal PPL degradation (+1.97% for Qwen2.5-7B, −0.06% for Mistral-7B, −0.44% for Gemma-2-9B) compared to massive quality loss for PDTrim (+25.85%, +22.07%, +35.63%) and SWS, which hard-prune the sequence.

Budget Sweeps: For lower budgets (e.g., $b=0.3$), retrieval accuracy and PPL are dominated by model-specific INT4 tolerance. Models passing the probe (Mistral, Gemma) maintain usability (+5.41% and +2.45% PPL, 100% NIAH at $b=0.5$), while Qwen catastrophically fails without probe, validating the necessity of adaptive policy selection.

Ablation Studies: Sink pinning offers minor but consistent improvement in sink-dominant/hybrid models, while importance ranking is beneficial primarily at moderate budgets where assignment is non-trivial. Random assignment and balanced policies show that improper INT4 placement can be catastrophic.

Retrieval Preservation: Unlike binary drop methods, SpectrumKV’s low-precision tokens allow partial retrieval even at highly compressed budgets, with significant NIAH advantage versus selection baselines.

Latency: Bandwidth reduction yields 50–62% reduction in end-to-end TTFT at $b = 0.5$, with decode throughput stable or improved due to offloaded memory bandwidth.

Analytical Findings

• The softmax amplification mechanism analytically explains why INT4 is unsafe for locally dominant attention models (e.g., Qwen2.5-7B): small vector errors cause large probability shifts, unpredicted by vector-level cosine similarity.
• Layer-wise error propagation analysis formalizes how early perturbations amplify through subsequent layers, suggesting per-layer tiering is a promising extension.
• Cold-tier fetches become lightweight upgrades instead of full payload retransmits due to the always-present low-precision representation.

Theoretical and Practical Implications

SpectrumKV reframes PD disaggregated KV transfer as a bandwidth allocation problem in the precision domain rather than a selection problem, drawing analogies to memory management where “cold” items are compressed but still accessible.

The introduction of a model-specific, functional tolerance probe (rather than heuristic vector similarity) is a crucial advance, ensuring safe configuration without manual hyperparameter tuning per deployment.

Practically, SpectrumKV’s design maps directly to modern LLM serving systems, is readily compatible with existing quantization techniques (e.g., KIVI, KVQuant), and unlocks substantial cost reductions for high-throughput or latency-sensitive serving settings. The pipeline is modular and can be composed with orthogonal advances, such as per-layer placement (OrbitFlow), lossless KV compression (SplitZip), or intelligent scheduling (vLLM).

Limitations and Future Directions

• Production Deployment: The presented system is evaluated on the isolated transfer path; integration with production PD runtimes, adaptive batching, and real network environments remains for future work.
• Workload Coverage: Evaluation is on instruction-tuned 7B–9B models; further analysis is needed for much larger models and broader tasks.
• Scoring Granularity: Current scoring is aggregate per-token. Incorporating per-head, per-layer, or task-aware metrics could further increase effectiveness.
• Probe Scope: The probe can be expanded to cover more workload patterns for even finer safety guarantees.

Future development should focus on full-stack systems integration, workload diversity, adaptive budget allocation, and explorations of per-layer/per-head mixed-precision strategies.

Conclusion

SpectrumKV establishes that per-token mixed-precision KV cache transfer, governed by model-adaptive error tolerance and downstream task requirements, outperforms binary selection methods in both perplexity and retrieval at equivalent bandwidth. The protocol’s design mitigates catastrophic quantization failures through online probing and delivers bandwidth savings and latency reductions in PD LLM serving. The generality of this approach, composability with orthogonal KV compression and placement methods, and insight into model-dependent error modes make SpectrumKV a robust foundation for the next generation of scalable, communication-efficient LLM inference.

Reference: "SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill–Decode Disaggregated LLM Serving" (2606.08635)