---
title: KV-Cache Fusion in Transformers
url: https://www.emergentmind.com/topics/kv-cache-fusion
type: topic
---

# KV-Cache Fusion in Transformers

KV-Cache Fusion refers to a broad family of methods designed to reduce the memory and computational footprint of key-value (KV) caches in Transformer-based language models by exploiting similarities or redundancies within or across cache entries. These approaches aim to mitigate the rapidly growing cache costs that arise from the persistence of KV tensors across layers, tokens, requests, and retrieval-augmented input structures—especially critical in long-context or highly concurrent inference scenarios. KV-Cache Fusion encompasses clustering and block-level fusion, layer-wise cache sharing, adaptive merging within sequences, low-rank tensor decomposition, and fusion strategies tailored for retrieval-augmented generation (RAG).

## 1. Theoretical Foundation and Motivation

Transformer LLMs, during inference, accumulate for each active session a KV-cache recording keys and values for every previous position and for every layer. The total memory required scales linearly with the number of concurrent requests ($B$), layers ($L$), maximum sequence length ($S$), and head dimension ($d$): $M = 2 \cdot B \cdot L \cdot S \cdot H \cdot d$. For long inputs and high concurrency, this memory demand dominates GPU resources and severely limits throughput.

KV-Cache Fusion responds to this bottleneck by identifying highly similar or redundant cache blocks—whether across requests, chunks, layers, or tokens—and combining them into shared, compressed, or fused representations. The principal goals are to reduce the number of unique cache entries, decrease memory and I/O costs, and maintain decoding quality and latency under various serving loads [2601.03067][2410.18517][2407.08454][2512.03870].

## 2. Block-Level and Joint Encoding Fusion

**Joint encoding of KV-cache blocks** ("Fast Fusion") leverages similarities across requests (Batch Fast Fusion, BFF) or within input chunks (Chunks Fast Fusion, CFF) by fusing blocks whose normalized direction cosine similarity exceeds a threshold $u$. Blocks assigned to the same cluster are replaced by a shared unit direction and associated scalar norms. The standard block table (pointer-based lookup) is retained, and no tensor kernel rewrites are required; attention computations alias to shared blocks via pointer redirection.

The compression/quality tradeoff is analyzed via a Gaussian-kernel model of similarity scores and Poisson rate estimation:
- A higher fusion threshold $u$ lowers distortion but reduces the compression ratio.
- Bound on attention-drift is $\|s' - s\|_1 \le 2\epsilon + O(\epsilon^2)$, with $\epsilon = \max_t \|k_t\|\|q\|/\sqrt{d} \cdot \sqrt{2(1-u)}$.
- Empirical results on Llama3.1-8B and Qwen2.5-72B show up to $4.38\times$ cache compression with $<0.5\%$ accuracy loss and $\approx 40\%$ throughput improvement under high concurrency [2601.03067].

## 3. Adaptive Token-Level Fusion

**Adaptive merging within a sequence** (KVMerger) addresses intra-sequence redundancy by greedily clustering adjacent tokens (within each layer) whose key vectors exhibit pairwise cosine similarity above a threshold $\epsilon$. For each "merging set," a Gaussian kernel weighing scheme computes a fused representative (pivoted at the token with maximal attention) via
$$
k_p^* = \sum_{i \in \mathcal{S}} w_i k_i \,,\qquad v_p^* = \sum_{i \in \mathcal{S}} w_i v_i
$$
with merge weights $w_i \propto \exp(-\|k_p - k_i\|_2^2 / 2\sigma^2)$. This method enforces data-independent, persistent sparsity and achieves compression by discarding non-pivotal tokens. It delivers substantial memory savings (down to 35% of original cache) with negligible accuracy loss, outperforming token eviction-based techniques especially on long contexts [2407.08454].

## 4. Layer-Wise and Cross-Layer KV-Cache Fusion

**Layer-wise sharing and fusion** targets redundancy across layers. In KVSharer, certain layers are selected (by maximizing Euclidean dissimilarity in their mean KV vectors) to reuse the KV cache from others. This can halve or further reduce cache memory with minimal degradation—achieving up to $1.65\times$ generation acceleration and maintaining perplexity within $<3\%$ of baseline on various LLMs. This method composes naturally with within-layer token pruning and quantization strategies [2410.18517].

**Cross-layer fusion with data-driven weights** (FusedKV) further refines this paradigm by constructing the top-layer KV cache as a learnable, feature-wise linear combination of post-RoPE keys from the bottom- and mid-level layers. Empirical analysis shows keys favor mid-levels while values are best reconstructed from early layers. FusedKV and its I/O-optimized variant (FusedKV-Lite, which reuses only one source per K/V) attain a $2\times$ memory reduction and can slightly improve perplexity over the baseline Transformer; throughput penalties are minor or amortized on memory-bound pipelines [2512.03870].

## 5. Dimension-Reduction and Low-Rank Fusion

**Low-rank projection approaches** (Palu) compress the KV-cache along the hidden dimension by decomposing the key and value projection matrices via SVD (joint, group, or per-head) and caching only the down-projected latents ($h$) per token. Keys and values are reconstructed on-demand for attention scoring. Rank allocation is guided by Fisher information, and Walsh-Hadamard transforms mitigate quantization outliers. This method can deliver $50$--$90\%$ compression, up to $2.91\times$ end-to-end speedup, and near-baseline perplexity—especially when paired with moderate head grouping [2407.21118].

## 6. Fusion for Retrieval-Augmented Generation

**Fusion strategies for RAG systems** address the subtleties of cache sharing with retrieved document chunks.
- Naive chunk-level caching misses all cross-chunk attention, producing a substantial drop in output quality.
- Prefix–Scale–Recompute (PSR) [*Editor's term*] fuses prior methods: (i) absorbs chunk head sinks during prefill, (ii) rescales softmax temperature and attention to cached blocks, and (iii) performs selective recomputation of tokens with high $\delta K$ scores. This tripartite fusion closes most of the accuracy gap to full prefill and saves over $70\%$ compute [2603.20218].
- FusionRAG optimizes offline chunk fusion (embedding cross-chunk context into each pre-cached chunk via top-$n$ neighbor fusion) and online, question-guided recomputation of the most critical tokens. With as little as $15\%$ recomputation, it recovers near-oracle generation quality and achieves $2.66\times$–$9.39\times$ TTFT reductions [2601.12904].

## 7. Implementation, Integration, and Performance

A recurrent theme is the avoidance of kernel rewrites and preservation of standard dense tensor layouts. Fused blocks are integrated via indirection tables or pointer manipulation, keeping lookup/update at $O(1)$ and fully compatible with typical GPU-accelerated attention kernels. Overheads from merging/fusion steps are amortized, typically remaining below $1\%$ of decode time for batch sizes in realistic server settings [2601.03067]. Adaptive thresholding allows dynamic tradeoff between memory and quality, and most fusion schemes are compatible with quantization or token-level sparsity.

Empirically, across methods and benchmarks, KV-Cache Fusion delivers memory reductions from $1.5\times$ to $4.4\times$ and throughput gains from $1.3\times$ to $4\times$, with $<1\%$–$3\%$ accuracy or perplexity penalty. In RAG-specific tasks, quality recovery can approach $99\%$ of full-attention performance at one-tenth of the compute cost [2601.03067][2410.18517][2407.08454][2512.03870][2603.20218][2601.12904].

---

**Representative KV-Cache Fusion techniques and properties**

| Technique            | Fusion Scope            | Typical Compression | Accuracy Impact      |
|----------------------|------------------------|---------------------|---------------------|
| Joint block encoding | Blocks across req./chunks | Up to $4.4\times$   | $<0.5\%$ F1 drop    |
| Adaptive merging     | Tokens within sequence | $2\times$–$2.8\times$ | Negligible          |
| Layer-wise sharing   | Whole Transformer layers | $2\times$           | $<3\%$ PPL drop     |
| Low-rank projection  | Hidden dim (per head/group) | $2\times$–$10\times$ | $<1\%$–$7\%$ acc. drop |
| RAG FusionRAG/PSR    | Cross-chunk fusion, selective recompute | $2\times$–$10\times$ (TTFT speedup) | $<5\%$ F1 gap vs full prefill |

## References

- "Joint Encoding of KV-Cache Blocks for Scalable LLM Serving" [2601.03067]
- "KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing" [2410.18517]
- "Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks" [2407.08454]
- "Palu: Compressing KV-Cache with Low-Rank Projection" [2407.21118]
- "Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers" [2512.03870]
- "An experimental study of KV cache reuse strategies in chunk-level caching systems" [2603.20218]
- "From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation" [2601.12904]

Source: https://www.emergentmind.com/topics/kv-cache-fusion