KV-Cache Communication Overhead

• KV-Cache Communication Overhead is a key bottleneck in transformer LLM inference, arising from transferring large key-value activations across GPUs, devices, and agents.
• Recent research introduces formal models, adaptive migration, and quantization methods to minimize data transfer costs and reduce memory pressure while improving throughput.
• Advanced techniques such as on-device scheduling, anchor-based KV reuse, and cross-layer fusion enable orders-of-magnitude latency reduction in multi-agent and distributed systems.

Key-Value (KV) cache communication overhead is a central bottleneck in scaling, distributing, and privatizing transformer-based LLM inference. The KV cache comprises activations (keys and values) generated at each decoding step and is frequently exchanged across hardware and software boundaries—across GPUs, devices, machines, or agents—incurring substantial data transfer, memory pressure, and latency penalties. Addressing this overhead is critical in multi-GPU LLM serving, private inference, mixture-of-experts (MoE) scenarios, edge/cloud pipelines, and multi-agent systems. Recent research introduces formal models, adaptive migration and compression mechanisms, scheduling strategies, and application-specific optimizations, achieving orders-of-magnitude improvement in effective KV-cache communication and overall system throughput.

1. Formal Models and Sources of KV-Cache Communication Overhead

The size of the KV cache grows linearly with sequence length, head and layer count, and batch size, with a full-precision representation for a single LLaMA-13B request at 4K tokens reaching ≈3.2 GB (Qianli et al., 12 Jan 2025). For multi-GPU inference, communication overhead arises from:

• Transferring the full KV cache between devices (e.g., request migration on multi-GPU servers).
• Transferring KV cache fragments or summaries across agents or pipeline stages (multi-agent or distributed inference).
• Streaming KV chunks between edge devices and the cloud, with per-chunk transfer and computation cost (Liu et al., 23 Apr 2026).
• All-to-all network exchanges in sequence-parallel setups and MoE routing (Liu et al., 2 Aug 2025).

The end-to-end migration cost is typically modeled as the sum of transfer time and network or bus latency. For example:

$$C_\mathrm{kv} = \frac{S_\mathrm{kv}}{B_\mathrm{comm}} + L_\mathrm{comm}$$

where $S_\mathrm{kv}$ is the cache size and $B_\mathrm{comm}$ is the available bandwidth (Qianli et al., 12 Jan 2025). Alternative migration primitives, such as token-transfer plus prefill, introduce an additional compute transfer cost (Qianli et al., 12 Jan 2025). KV communication traffic is further exacerbated by redundant computation in multi-agent pipelines and dense MoE synchronization (Ye et al., 14 Oct 2025, Liu et al., 2 Aug 2025).

Other key sources include:

• Full KV replication in MoE architectures, which increases communication volume and memory by a factor equal to the number of experts (Liu et al., 2 Aug 2025).
• Inter-agent transmission of raw prefixes or recomputation of KV entries under varying context, suffering from offset-variance and redundancy (Ye et al., 14 Oct 2025, Kriuk et al., 27 Nov 2025).
• Transfer of uncompressed, high-dimensional KV in edge-cloud or mobile-server settings (Liu et al., 23 Apr 2026).

2. Adaptive Migration, Scheduling, and Overhead-Aware Transfer

Optimizing KV communication overhead requires joint consideration of communication and compute capacity, request migration cost, and system-level balance.

Multi-GPU Scheduling and Migration

Memory-efficient serving systems such as MELL employ an adaptive migration mechanism that profiles each link for a "communication boundary" $Q_\mathrm{comm}$ and compute for a "compute boundary" $Q_\mathrm{comp}$. At each epoch, KV-cache transfer ($\alpha_i=1$) and token-transfer plus prefill ($\alpha_i=0$) are jointly considered for each migrating request, forming a min-cost two-bin packing problem:

$$\min_{\alpha_i\in\{0,1\}} \sum_{i\in\mathcal{M}} [\alpha_i\,C_{\mathrm{kv},i} + (1-\alpha_i)\,C_{\mathrm{tok},i}]$$

subject to communication and compute boundary constraints (Qianli et al., 12 Jan 2025).

On-Device Scheduling

SparKV addresses edge/cloud latency bottlenecks by modeling the per-chunk transfer and compute time, and then solving a makespan minimization via mixed-integer programming:

$$\min_{\{x_i\}} \max\left\{\sum_{i} x_i \,T_{\mathrm{comm}(i)},\, \sum_{i} (1-x_i)\,T_{\mathrm{comp}(i)}\right\}$$

where $x_i$ is a binary decision indicating path assignment (Liu et al., 23 Apr 2026).

Greedy heuristics and online refinements are used in practice, with empirical TTFT reductions up to 5.1× over prior hybrid loading baselines.

MoE and Multi-Agent Routing

Expert-sharded storage and sparse routers (top-k expert selection) in PiKV reduce the fraction of KV communicated at each step compared to dense all-expert synchronization (Liu et al., 2 Aug 2025). Multi-agent systems employ alignment and interpolation to reuse overlapping KV caches under varying prefixes, reducing TTFT by up to 7.8× and enabling reuse rates above 70% (Ye et al., 14 Oct 2025).

3. Lossy KV Compression and Dimensionality Reduction

Compression of KV caches directly mitigates communication overhead by reducing the actual data volume transmitted or stored.

Blockwise Quantization and Entropy Encoding

KVComp applies controlled blockwise quantization with a specified relative error bound, followed by per-layer Huffman coding. For float16 caches, empirical compression ratios reach 4×–6× overall (up to 8× for keys and 4× for values), transforming a 40 GB/token transfer to 8 GB/token at negligible accuracy degradation (<1%) (Jiang et al., 30 Aug 2025).

Adaptive Dimensionality Compression

FDC uses per-head, per-layer SVD-based offline re-parameterization to reduce the KV dimension from $S_\mathrm{kv}$0 to $S_\mathrm{kv}$1, tailored by their actual contribution to inference. An adaptive allocation maximizes reduction under a stringent perplexity loss constraint, yielding up to 80% lower communication time and 1.97×–2.8× throughput gains (Zhang et al., 2024).

Protocol-Aware and Layer-Adaptive Quantization

Q-KVComm introduces sensitivity profiling to assign variable quantization widths per layer (e.g., 30% at 8-bit, 40% at 6-bit, 30% at 4-bit), followed by bit-packing. It achieves 5–6× compression ratios, with coherence scores above 0.77 even under severe bit-width reduction (Kriuk et al., 27 Nov 2025).

4. Specialized Cache Management and Eviction Strategies

Workloads with special structure or privacy requirements require distinct strategies.

MPC-Friendly KV Eviction

MPCache integrates a two-stage approach: one-time static pruning and online dynamic selection. By combining prefill-based attention importance and online query-aware cluster-based token selection with cryptographically efficient approximations, MPCache reduces per-step KV size by up to 8.4× and communication by 3.39–8.37×, all while keeping round-trip MPC latency and client privacy practical (Zeng et al., 12 Jan 2025).

Chain-of-Thought–Oriented KV Caching

Crystal-KV exploits the answer-first principle to identify and retain only those KV entries that directly impact final answer quality. An attention-aware LRFU policy, paired with adaptive layer/head budgeting, achieves memory reductions of 90% and up to 12× throughput improvement without answer accuracy loss (even exceeding FullKV on coding/math chains) (Wang et al., 5 Jan 2026).

Cross-Layer Fusion and Sharing

FusedKV and FusedKV-Lite reconstruct upper-layer KV caches from lower-layer caches through learnable or pointer-based fusion. Both methods achieve a 50% reduction in cache memory and substantially lower I/O, with negligible or beneficial perplexity impact. FusedKV incurs 1.5× I/O but yields the lowest PPL; FusedKV-Lite matches baseline I/O and is recommended for I/O-bound scenarios (Lin et al., 3 Dec 2025).

5. Distributed and Multi-Agent Communication Protocols

In scalable and agentic architectures, KV-cache communication is often paired with more nuanced protocol layers.

Anchor Pool–Based Offset Alignment

KVCOMM introduces an anchor-pool mechanism for online cross-context KV reuse in multi-agent settings, matching and interpolating cache offsets using semantic similarity and empirical anchor statistics. Compared to full prefill, TTFT is improved by up to 7.8× with 70-88% reuse rates and sub-3% accuracy loss (Ye et al., 14 Oct 2025).

Adaptive Cross-Architecture Compression and Integration

Q-KVComm extends beyond compression: it layers adaptive quantization, hybrid fact extraction, and calibration transforms to bridge model heterogeneity, delivering a new multi-agent communication paradigm that is representation based instead of text based (Kriuk et al., 27 Nov 2025).

Modal Decomposition in Agent Specialization

LRAgent decomposes agent KV caches into a base (shared) and adapter (low-rank) part, allowing nearly full cache sharing and Flash-LoRA-Attention kernel application. This achieves memory reductions approaching the FullShared ideal, with negligible overhead and near-baseline accuracy across agentic QA tasks (Jeon et al., 1 Feb 2026).

6. Empirical Impact and Best Practices

The impact of optimized KV-cache communication is substantiated by systematic empirical measurements.

Method/System	Compression/Overhead Gain	Latency/Throughput Impact	Quality Loss	Reference
MELL (multi-GPU serving)	GPUs: –31%; Utilization: +43%	Migration freq: ≤1.5 req/s; Penalty <10%	None	(Qianli et al., 12 Jan 2025)
MPCache (MPC LLM)	3.39–8.37× comm. reduction	1.8–2.01× latency ↓	<5% F1/ROUGE loss	(Zeng et al., 12 Jan 2025)
FusedKV-Lite	50% memory, equal I/O	TTFT ↓~50%	≤0.05 PPL diff	(Lin et al., 3 Dec 2025)
KVComp	4–6× mem, 80% comm. ↓	Throughput >2× cuBLAS	<1% accuracy drop	(Jiang et al., 30 Aug 2025)
Crystal-KV (CoT)	90% mem savings	Throughput 7.6×, latency ↓1.24×	none/↑ (CoT)	(Wang et al., 5 Jan 2026)
SparKV (on-device)	TTFT ↓1.3–5.1×, energy ↓1.5–3.3×	Robust under bandwidth fluctuation	None	(Liu et al., 23 Apr 2026)
PiKV (MoE)	40–50% comm., 3.9× mem ↓	2.1×–2.4× latency ↓, 2.3–3.1× throughput	<1.5%	(Liu et al., 2 Aug 2025)
Q-KVComm (multi-agent)	5–6× comm. ↓, 80–90% bandwidth ↓	Coherence: ≥0.77	≤5% (4b), ≤2% (8b)	(Kriuk et al., 27 Nov 2025)

System-level best practices include profiling bandwidth and compute headroom, co-locating chatty GPU pairs, batching request migrations, and enforcing global migration budgets (Qianli et al., 12 Jan 2025). In cross-device scenarios, migratory adaptivity and error-aware protocol selection are essential for maximizing throughput and minimizing tail latency.

7. Limitations, Trade-offs, and Open Directions

Despite substantial progress, reducing KV-cache communication overhead remains contingent on context, architecture, and workload. Aggressive compression may induce accuracy loss, requiring task-specific calibration (e.g., answer-critical entries in CoT (Wang et al., 5 Jan 2026)). Hierarchical scheduling and adaptivity (e.g., greedy refinements, anchor pools) present robust real-time solutions, but require further exploration for extremely long contexts or non-standard positional encodings. In MoE and agentic settings, sharing mechanisms must consider read locality, heterogeneity, and role specialization to avoid accuracy and latency trade-offs (Liu et al., 2 Aug 2025, Jeon et al., 1 Feb 2026). The field is evolving, with new paradigms such as representation-based multi-agent protocols and cross-layer synthetic fusion representing active areas of investigation (Kriuk et al., 27 Nov 2025, Lin et al., 3 Dec 2025).