MorphKV: Optimizing KV Cache for LLMs

• MorphKV is a family of techniques that manage key-value caches in large language models, reducing memory and bandwidth demands during long-context inference.
• It uses a constant-sized, correlation-aware cache algorithm that retains recent tokens and contextually relevant past tokens to maintain output fidelity.
• Additionally, its pressure-aware resizing mechanism dynamically adjusts cache blocks under load, ensuring efficient resource utilization and low latency.

MorphKV refers to a family of inference-time techniques and runtime mechanisms designed to optimize Key-Value (KV) cache management for LLMs deployed in real-world environments. The term encompasses two primary approaches: (1) a constant-sized, correlation-aware cache-morphing algorithm for extended LLM responses; and (2) an elastic, pressure-sensing KV cache resizing system for adaptive serving under bursty or memory-constrained loads. Both address the fundamental challenge of handling the linear growth in memory and bandwidth demand imposed by traditional KV caching in autoregressive Transformers, particularly during long-context or long-response inference.

1. Motivation and Problem Scope

Autoregressive Transformer-based LLMs use a KV cache during inference, where each new token adds its respective key and value vectors to the cache, enabling subsequent tokens to attend back arbitrarily along the generated context. This leads to three principal bottlenecks:

• Excessive GPU memory consumption, often surpassing on-chip high-bandwidth memory (HBM) limits as sequence lengths grow.
• Elevated off-chip bandwidth requirements, as large caches must be accessed at every decode step, exacerbating hardware utilization stress.
• Latency amplification due to cache size growth, manifesting acutely in long-response or throughput-critical tasks.

Prior reduction approaches (fixed-window retention, oracle/hard attention-based pruning, learned token selection, or prefill-only pruning) incur trade-offs between memory, accuracy, and scalability, such as discarding crucial context or introducing selection bias, and fail to simultaneously guarantee bounded memory and high-fidelity attention (Ghadia et al., 2 Mar 2025).

2. MorphKV: Constant-Sized Correlation-Aware Cache Algorithm

The MorphKV algorithm enforces a hard cap on the KV cache size at each decoding step through a dynamic "morphing" process. It employs two principal heuristics:

• Local Coherence ($\mathcal H_1$): persistently maintains the most recent $R$ tokens, ensuring that local attention structure required for syntactic and semantic continuity is preserved.
• Distant Relevance ($\mathcal H_2$): among all non-recent (older) tokens, retains only the top $C$ tokens that exhibit strongest attention-based correlation with the recent window, thus surfacing contextually salient but distant events.

The core procedure involves:

1. Identifying the recent window of $R$ tokens.
2. Aggregating (sum or max fusion) their per-head attention distributions over past tokens to form a global significance score vector $F_i$ at time step $i$.
3. Selecting the $C$ highest-ranking non-recent tokens from $F_i$ and combining these with the $R$ most recent tokens to construct the next-step cache $G_{i+1}$, yielding a constant cache size $C+R$.
4. Updating this process at every decoding timestep, thus adaptively tracking both local and relevant long-range dependencies without requiring a priori knowledge of which tokens will be attended.

Mathematically, MorphKV approximates the combinatorially optimal subset selection problem for minimal attention output loss, using greedy heuristics. The theoretical guarantee is that, with suitable $C$ and $R$, the $\ell_2$ norm difference in attention outputs $\|O_i - O_i'\|_2$ can be bounded by $\epsilon$, provided $F_i$ accurately identifies influential past states (Ghadia et al., 2 Mar 2025).

3. Pressure-Aware KV Cache Resizing

A distinct but complementary application of MorphKV (as described in MorphServe) targets dynamic serving scenarios with fluctuating memory pressure. Here, MorphKV acts as a runtime resizing policy for blocks of KV cache ("KVC blocks") on each GPU worker:

• Memory and queue pressure are continually monitored: the ratio $P = M_\mathrm{KV} / M_\mathrm{GPU,\,total}$ and queue delay are smoothed over short time windows.
• Elastic expansion/contraction: When $P$ exceeds the high-water threshold ($\theta_\mathrm{high} \approx 0.85$) or queue delays rise, MorphKV triggers model-layer swapping (to free memory) and allocates as many new KVC blocks as allowed, growing the KV cache beyond static full-precision limits if necessary. Conversely, when pressure falls below a low-water mark ($\theta_\mathrm{low} \approx 0.60$), surplus KVC blocks are deallocated incrementally to avoid thrashing (Su et al., 24 May 2025).

Expansion and contraction uses simple, tunable policies:

• When $P > \theta_\mathrm{high}$:

$$\Delta C^+ = \min\left(\Delta C_\mathrm{max},\, \lceil\alpha (P - \theta_\mathrm{high}) C_\mathrm{curr}\rceil\right)$$

and adjust cache accordingly.

• When $P < \theta_\mathrm{low}$:

$$\Delta C^- = \min\left(C_\mathrm{curr} - C_\mathrm{min},\, \lfloor\beta (\theta_\mathrm{low} - P) C_\mathrm{curr}\rfloor\right)$$

Block attach/detach operations are performed asynchronously on a dedicated CUDA stream to ensure non-blocking execution relative to decode/prefill streams, minimizing runtime impact.

4. Empirical Benchmarks and Performance Outcomes

Correlation-Aware MorphKV Algorithm (Ghadia et al., 2 Mar 2025)

• Memory savings: On LongWriter benchmark, MorphKV used only 0.25× the full cache; on LongGenBench 0.55×; on LongBench 0.15×, representing up to 52.9% reduction relative to baseline methods such as SnapKV and H₂O.
• Accuracy improvements: Delivered up to 18.2 percentage points higher performance on long-response tasks over competing state-of-the-art, and matched/exceeded long-context accuracy of SnapKV while using approximately 50% less memory.
• Robustness to response length: Exhibited only 10% performance drop at 4× longer outputs, compared to 15–18% for prior approaches.

Pressure-Aware KV Cache Resizing (Su et al., 24 May 2025)

• Service-level objective (SLO) compliance: MorphServe’s joint LayerSwapper and MorphKV policies reduced average SLO violations by 92.45% versus full-precision, and improved P95 Time-to-First-Token (TTFT) latency by 2.2×–3.9× (and up to 19.5× in performance mode).
• Dynamic expansion: MorphKV allowed the KV cache to expand by up to 32.97% beyond traditional static allocation thresholds during periods of peak load, eliminating request preemption and recomputation.
• Queue delay mitigation: Achieved up to 3.8× reduction in prefill queueing delays, directly reducing TTFT variance.
• Resource utilization: Improved average GPU memory utilization by 29.29% and raised output accuracy by 3.58% compared to statically-quantized baselines.

Tables below summarize benchmark results and algorithmic strategies (values from referenced papers):

Approach	Memory Peak (LongWriter)	Relative Accuracy (LongWriter)
MorphKV	0.25×	+4.5% over H₂O
H₂O	~1×	Baseline
SnapKV	~4×	Matched/exceeded in most cases

Policy	Expansion Trigger	Max Observed Gain
MorphKV Resizing (MorphServe)	P > 0.85, delay > 100 ms	32.97% KVC growth, 92.45% SLO reduction

5. Implementation and Compatibility

Both forms of MorphKV operate as inference-time modules and are orthogonal to underlying model architectures. Key implementation aspects include:

• Cache operations: MorphKV’s constant-sized algorithm is a drop-in replacement, requiring only the tracking and ranking of attention scores from recent tokens; fusion is vectorizable and incurs negligible per-step overhead.
• Runtime resizing: Block-level KVC resizing is layered atop PagedAttention primitives as used in vLLM and SwiftLLM, integrating with modern attention kernels (e.g., FlashAttention v2), Grouped Query Attention (GQA), and Multi-Head Latent Attention (MLA), all without recompilation or kernel modification. Block attach/detach leverages CUDA memory registration, and MorphKV maintains a dedicated CUDA stream for resizing activities.
• Compatibility with other strategies: MorphKV’s resizing is agnostic to per-block eviction or lossy compression schemes (such as SnapKV or pyramid-KV); such methods may be used in conjunction with block orchestration (Su et al., 24 May 2025).

6. Practical Implications and Limitations

Predictable and bounded GPU memory consumption is a primary benefit: with a constant cache size, allocation can avoid HBM over-subscription and reduce per-token latency jitter. In the resizing context, elastic resource management enables efficient load balancing and minimizes service interruptions. The major overhead (tracking $R$ attention profiles; block (un)mapping) is minor, with empirical per-token overhead remaining under 1%.

A limitation is the dependency on attention-based heuristics to identify distant relevance; theoretical analysis does not yield a closed-form recovery bound, but empirical evidence supports the practical utility of the approximations. Aggressive cache expansion during high pressure can risk proximity to OOM if not balanced with timely layer swapping or proper conservative tuning parameters.

7. Future Directions

Research has identified promising directions for further development:

• Adaptive, per-layer tuning of the $C, R$ parameters to better match layer-specific context requirements.
• Tighter theoretical characterization of attention-output approximation error under the MorphKV selection heuristics.
• Development of learned fusion functions that incorporate token embeddings alongside attention scores, potentially refining the identification of influential distant context with minimal bias.
• Integration and co-design with next-generation attention and scheduling frameworks for even higher throughput and lower tail latency (Ghadia et al., 2 Mar 2025).