KV Cache Offloading & Memory Optimization

Updated 2 December 2025

KV cache offloading with optimized memory management is a technique that distributes per-token key-value data across GPU, CPU, and storage to enable scalable LLM inference.
It leverages hierarchical allocation, dynamic offloading, and compression methods to reduce latency and improve throughput while meeting strict service level objectives.
Empirical evaluations show significant gains in time-to-first-token, memory efficiency, and throughput across diverse architectures and serving frameworks.

Key-value (KV) cache offloading with optimized memory management is a critical paradigm for scaling LLM inference under strict GPU, CPU, or storage capacity constraints. The linear growth of the KV cache—comprising per-token, per-layer key and value tensors for autoregressive attention—poses acute challenges in low-latency, high-throughput serving, especially as context lengths reach tens or even millions of tokens. Modern systems implement a rich hierarchy of offloading, compression, scheduling, and resource coordination to address both memory pressure and data movement overhead. This article systematically presents model architectures, offloading algorithms, mathematical formulations, and empirical results from recent research, highlighting layer-wise, block-wise, and hardware-adaptive solutions.

1. Memory Hierarchy, Data Structures, and Layer-wise Partitioning

LLM KV caches consist of keys and values for each transformer layer and attention head, with per-request GPU-resident blocks and host or secondary-storage regions for spilled entries. Layer-wise resource management, as pioneered in "LayerKV" (Xiong et al., 1 Oct 2024), divides the per-request cache into $L$ logical layer blocks. Each $(r,\ell)$ pair is tracked in a BlockTable indicating its location (GPU or host), memory offset, and block identity. The aggregate GPU and host memory footprints are rigorously expressed as: $M_\text{gpu,total} = \sum_{r=1}^R x_r S_r 2dH b,\qquad M_\text{host,total} = \sum_{r=1}^R (L-x_r) S_r 2dH b$ where $x_r$ is the number of layers per request $r$ held on GPU. Dynamic block allocation and fine-grained release rely on global counters and a per-request, per-layer block mapping.

Beyond simple partitioning, systems such as vTensor (Xu et al., 22 Jul 2024) implement virtual-memory–backed tensor abstractions. vTensor decouples CUDA-visible tensor handles from their physical backing via a manager orchestrating dynamic chunk allocation, extension, and reclamation—thus enabling elastic, fragmentation-resistant allocation for the ever-growing KV cache footprint.

2. Optimized Offloading: Strategies for GPU→CPU and Heterogeneous Memory

Offloading strategies span several dimensions: the offload destination (CPU pinned/pageable memory, disk/NVMe, CXL-PNM), granularity (token, block, layer), and reloading patterns (synchronous, overlapped). LayerKV (Xiong et al., 1 Oct 2024) offloads lower-sensitivity layers to host, holding only a minimal prefix on GPU to limit the service’s Time To First Token (TTFT) and SLO violation rate. TailorKV (Yao et al., 26 May 2025) partitions layers as "quantization-friendly" (on-GPU, aggressively quantized) or "sparsity-friendly" (CPU-offloaded, only dominant tokens asynchronously reloaded).

MIRAGE (Li et al., 15 Jul 2025) introduces parameter remapping: static model parameter pages (immutable weights) are dynamically evicted from GPU HBM to CPU memory, repurposing the freed pages as extra KV cache capacity. This unidirectional "lending" exploits the relative stability of model parameters and uniformity of compute pipelines, avoiding costly bidirectional KV swap traffic.

For disk-bound inference (e.g., on-device, mobile platforms), KVSwap (Zhang et al., 14 Nov 2025) offloads entire caches to secondary storage, maintains a low-rank in-RAM K cache, and applies a prediction-based group prefetch mechanism to amortize I/O and maximize overlap with computation. PCIe transfers are staged and controlled by policies that dynamically trade off buffer reuse against group granularity and I/O bandwidth.

Processing-Near-Memory (PNM) and Compute Express Link (CXL) architectures (Kim et al., 31 Oct 2025) further offload cache management and selection to near-memory accelerators, enabling full 1M-token contexts and multi-socket scaling.

3. Compression and Hybrid Cache Reduction

Lossy and lossless compression is central to amortizing the memory and I/O cost of cache offloading:

KVComp (Jiang et al., 30 Aug 2025) implements a two-stage blockwise pipeline: "channel-wise" quantization and high-throughput Huffman encoding on keys/values, fusing decompression directly into the mat-vec kernel of attention. This strategy delivers 47–83% memory reduction with $<3\%$ accuracy loss, sometimes accelerating attention relative to uncompressed cuBLAS due to reduced global memory bandwidth consumption.
ZSMerge (Liu et al., 13 Mar 2025) applies zero-shot, head-level importance scoring, dynamic budget allocation, and compensated residual merging to compress cache entries, achieving 20:1 memory reduction for LLaMA2-7B with no retraining.
KVTC (Staniszewski et al., 3 Nov 2025) introduces PCA-based transform coding, adaptive quantization, and byte-oriented entropy compression (DEFLATE/nvCOMP) to yield 16–20 $\times$ compression with negligible loss across models and tasks, further accelerating cross-node cache movement.
TailorKV (Yao et al., 26 May 2025) combines 1-bit GEMV for "dense/quantization-friendly" layers with dynamic token importance scoring and sparse relayout for "sparsity-friendly" layers, performing cache reloading only on demand and fully overlapping PCIe transfers.

Hybrid memory management thus often fuses (i) predictive/importance-aware token selection, (ii) block- or channel-wise quantization, and (iii) asynchronous fetch or prefetch orchestration.

4. SLO-Aware, Device-Adaptive Scheduling and Multi-GPU Coordination

Modern serving systems employ SLO-aware schedulers and online algorithms to maximize utilization under memory constraints:

LayerKV (Xiong et al., 1 Oct 2024) schedules per-request allocations to minimize queuing delays and TTFT, ensuring that no layer’s offload cost exceeds the token prefill time of subsequent requests.
MELL (Qianli et al., 12 Jan 2025) introduces adaptive request migration across GPUs to balance load: requests are migrated either as raw KV-caches (communication-heavy) or as prompts for recomputation (compute-heavy), following a hybrid bin-packing cost model. Epoch-wise batched scheduling and migration ensure a 4/3-competitive occupancy rate and hard bounds on migration frequency.
PiKV (Liu et al., 2 Aug 2025)—for Mixture-of-Experts architectures—shards expert-specific caches, adaptively schedules retention and eviction, and routes only top- $k$ queries to each expert, lowering communication and memory costs via both topology and cache-aware penalties.

5. Asynchronous Overlap and Hardware-Aware Runtime Design

Overlap of computation and I/O is essential in minimizing end-to-end latency:

KVPR (Jiang et al., 26 Nov 2024) models the per-layer wall-clock time as

$t^i(l) = t_\text{act}^i(l) + \max\left(t_\text{recomp}^i(l),\,t_\text{kv\_stream}^i(l)\right)$

and uses a runtime LP to pick the activation/payload split that minimizes total decode time. This typically keeps the GPU busy nearly 99% of the time, drastically reducing PCIe bottlenecks.

CLO (Yi et al., 18 Nov 2025) introduces a head-wise approximate on-GPU cache and speculative prefetching, with a zero-copy transfer engine based on GPU BAR space and a GPU-centric synchronization protocol that eliminates host-launch stalls. This allows fully saturating PCIe links and hiding data transfer latency, achieving 9–67% throughput improvement.
InfiniGen (Lee et al., 28 Jun 2024) employs a runtime rehearsal mechanism: a minimal, SVD-skewed dot-product reveals the handful of crucial tokens per layer, and only those KV entries are transferred to the device, reducing per-iteration PCIe traffic by 20–50 $\times$ .

6. Budget Allocation, Importance-Aware Eviction, and Episodic Compression

Memory efficiency is maximized by adapting KV retention to token and layer importance:

EpiCache (Kim et al., 22 Sep 2025) introduces block-wise prefill and episodic clustering for multi-turn conversational history, bounding peak cache to $\le M + M_\text{block}$ and assigning per-layer budgets via measured sensitivity to key eviction:

$M_\ell = M\,\frac{S_\ell^\alpha}{\sum_{j=1}^L S_j^\alpha}$

where $S_\ell$ is cosine dissimilarity sensitivity.

ZSMerge (Liu et al., 13 Mar 2025) maintains per-token, per-head importance scores via exponential smoothing and partitions the fixed budget into context, recency, and residual (merged) zones, supporting stratified offload to DRAM or NVMe without retraining.

This importance-aware stratification complements both system and kernel-level memory optimization.

7. Empirical Performance, Scalability, and System Integration

Comprehensive evaluations reveal that state-of-the-art KV offloading with optimized memory management attains the following:

Up to 69 $\times$ improvement in TTFT and 28.7% reduction in SLO violation rates at large context lengths (LayerKV (Xiong et al., 1 Oct 2024)).
47–83% GPU memory reduction at $<3\%$ accuracy loss (KVComp (Jiang et al., 30 Aug 2025)), with throughput in compressed+fused-attention regimes sometimes exceeding that of uncompressed kernels.
1.8 $\times$ –3.2 $\times$ end-to-end throughput improvement and 1.7 $\times$ –2.4 $\times$ lower latency in PiKV for MoE (Switch, GLaM, PaLM, Mixtral) (Liu et al., 2 Aug 2025).
Orders-of-magnitude context scaling: PNM/CXL memory architectures (1M-tokens, 405B-parameters) achieving $21.9\times$ throughput, $60\times$ energy reduction, $7.3\times$ cost reduction at million-token scale (Kim et al., 31 Oct 2025).
Compressors (KVTC (Staniszewski et al., 3 Nov 2025), ZSMerge (Liu et al., 13 Mar 2025)) attain 16–20 $\times$ footprint reduction at $<1$ point accuracy loss, with negligible latency penalty over cacheless recompute.

These gains are realized while maintaining compatibility with leading serving frameworks (vLLM, Triton, Hugging Face, FasterTransformer) and integrating into both single-node and distributed infrastructures.

In summary, KV cache offloading with optimized memory management merges layer/block-level allocation and partitioning, hybrid/compression-based cache reduction, device-adaptive runtime scheduling, and asynchronous orchestration across heterogeneous memory/storage hierarchies. The resulting systems—by dynamically balancing data movement, compute overlap, and memory utilization—enable efficient, scalable inference for large and long-context LLMs without compromising generation quality or latency guarantees (Xiong et al., 1 Oct 2024, Jiang et al., 30 Aug 2025, Yao et al., 26 May 2025, Kim et al., 31 Oct 2025, Zhang et al., 14 Nov 2025, Jiang et al., 26 Nov 2024, Li et al., 15 Jul 2025, Qianli et al., 12 Jan 2025, Liu et al., 2 Aug 2025, Liu et al., 13 Mar 2025, Staniszewski et al., 3 Nov 2025, Yi et al., 18 Nov 2025, Kim et al., 22 Sep 2025, Lee et al., 28 Jun 2024, Xu et al., 22 Jul 2024).