---
title: Cross-layer KV Cache Sharing
url: https://www.emergentmind.com/topics/cross-layer-kv-cache-sharing
type: topic
---

# Cross-layer KV Cache Sharing

Cross-layer Key-Value (KV) cache sharing is a methodological class for reducing the key and value memory overhead in multi-layer transformer decoders by reusing, merging, or compressing the KV caches across different layers, rather than treating each layer’s cache as fully independent. This paradigm addresses both the O(L·T·D) scaling of memory usage—where L is the number of layers, T the sequence length, and D the hidden dimension—and the practical deployment bottlenecks of large language model (LLM) inference, especially for long-context or batch-parallel generation. Research in this area spans purely architectural innovations, data-driven one-shot merging strategies, orthogonal hardware-layer system integration, and composable hybrid techniques incorporating quantization, pruning, and low-rank subspace recovery.

## 1. Formal Definition and Core Motivations

In a standard L-layer transformer decoder, the inference-time KV cache comprises two tensors per layer:
\[ K_\ell(t) = W_K h_{\ell-1}(t) \in \mathbb{R}^{B \times H \times d_k}, \quad V_\ell(t) = W_V h_{\ell-1}(t) \in \mathbb{R}^{B \times H \times d_k} \]
accumulating across time into \( K_\ell \in \mathbb{R}^{B \times H \times S \times d_k} \), with S the unrolled sequence. For all L layers and both keys and values, the total storage (in “elements”) is:
\[ \text{Mem}_{kv} = 2 B L S D, ~\text{with}~ D = H d_k \]
Cross-layer KV cache sharing schemes seek to realize \( \text{Mem}^{\text{shared}}_{kv} < \text{Mem}_{kv} \) by replacing or compressing layer-wise caches, motivated by the substantial (often >80%) fraction of on-device memory that K/V storage demands during LLM decoding and by empirical observations of redundancy or recoverability in K/V representations across the depth of modern decoders [2410.18517][2412.19442].

## 2. Methodological Taxonomy of Cross-Layer KV Sharing

Current approaches are categorized by the nature of sharing, adaptation strategy, and combinations with intra-layer methods:

- **Direct reuse or pointer sharing:** Later layers use the KV cache of earlier “anchor” layers without recomputation [2410.15252][2405.12981].
    - E.g., Cross-Layer Attention (CLA): partition layers into groups of size c, anchor computes KV, followers reuse it. KV cache memory shrinks by c-fold [2405.12981].
- **Similarity/Distance-gated reuse:** Sharing is applied only if adjacent layers’ caches are sufficiently (dis)similar according to a specific metric (e.g., L1/L2/cosine distance of per-head values) [2410.18517][2512.06727].
- **Low-rank subspace consolidation:** SVD is used to merge KV caches of a group of G layers into a single low-rank basis plus per-layer coefficients [2503.18893][2508.16134].
- **Fusion and asymmetric mapping:** Upper-layer KV caches are constructed as learnable or rule-based fusions of specific lower-layer (e.g., middle/bottom) caches, motivated by empirical signal propagation asymmetry [2512.03870].
- **Index or block sharing (system-level):** Sparse or paged caches re-use index tables or blocks between adjacent layers/steps, with dynamic filtering [2501.06807][2507.21433].
- **Quantization and compression synergy:** Cross-layer sharing is coupled with low-bit quantization and/or autoencoder-based per-layer compression, often in a plug-and-play manner [2510.11236][2410.15252][2512.06727].

## 3. Key Algorithms and Theoretical Analysis

Representative workflows center on three axes: similarity computation, merge/reuse decision, and post-processing/reconstruction.

### 3.1 Proxy similarity metrics
- Cosine-similarity or Lp distance between average/flattened KV vectors: \( D(i,j) = 1 - \langle \bar k_i, \bar k_j \rangle / (\| \bar k_i \|_2 \| \bar k_j \|_2) \), or \( E(i,j) = \| \bar k_i - \bar k_j \|_2 \) [2410.18517][2512.06727].
- Per-head distances for fine-grained selective sharing: \( s^K_{\ell,i} = \frac{1}{T \cdot D} \sum_{t,j} |K_\ell[i,t,j] - K_{\ell+1}[i,t,j]| \) [2512.06727].

### 3.2 Merge strategy & scheduling
- Greedy selection of sharing pairs by highest dissimilarity (counterintuitive result: dissimilar caches preserve performance better) [2410.18517].
- Layer-wise SVD and threshold-based grouping, with optional adaptive budget allocation via cosine similarity of latent keys [2503.18893][2508.16134].
- For asymmetric fusion, explicit constraints (e.g., only value caches from bottom, keys from bottom/middle) and preservation of positional information (e.g., RoPE-space fusion with symmetric weighting) [2512.03870].

### 3.3 Memory savings and compute model
- Relative compression, e.g., sharing C out of L layers yields 
  \[ \Delta \text{Mem} = 1 - \frac{L-C}{L} = \frac{C}{L} \]
  for direct reuse [2410.18517].
- For groupwise low-rank sharing: 
  \[ \text{Bits}_\text{tot} = 2 \cdot (L/G) \cdot n \cdot B, \]
  where B is the bit-width, G is group size, and n is tensor length [2510.11236].

### 3.4 Example pseudocode: selective cross-layer reuse [2410.18517]
```python
for each input x, for each decoding step t:
    for layer ℓ = 1 ... L:
        if (i←ℓ) in sharing_strategy:
            K_ℓ(t), V_ℓ(t) = K_i(t), V_i(t)  # copy from layer i
        else:
            K_ℓ(t), V_ℓ(t) = compute as usual
        append to running cache
```
This structure is modified in various ways for dynamic fusion, low-rank SVD, or selective block reuse.

## 4. Empirical Results: Compression, Speed, and Accuracy

- **Direct cross-layer reuse (CLA, YOCO):** 
    - 50% cache memory reduction with <0.1–2% PPL loss [2405.12981][2410.15252][2512.03870].
    - Full stack throughput speedup up to 1.65×, especially in long-sequence regimes [2410.18517][2405.12981].
- **Dissimilarity-based strategy (KVSharer):**
    - 25% layers shared: 28% memory savings, 1.65× generation acceleration, ≤10% PPL increase [2410.18517].
    - Scaling to 30–35% as sequence length grows (e.g., 72% memory of baseline at 2048 tokens).
- **Fusion (FusedKV/FusedKV-Lite):** 
    - 50% memory reduction, lowest perplexity of all 50%-memory methods, outperforms both vanilla CLA/YOCO and GQA [2512.03870].
- **Low-rank SVD consolidation (xKV, CommonKV):** 
    - xKV: up to 6.8× higher compression (relative to competitive inter-layer baselines), 2.7% higher accuracy [2503.18893].
    - CommonKV: up to 98% effective compression with negligible PPL loss when combined with quantization/eviction [2508.16134].
- **Similarity-guided per-head reuse (KV-CAR):** 
    - 6.6–12.5% savings at <0.5 PPL increase with careful head selection; up to 47.8% reduction when combined with per-layer autoencoder [2512.06727].
- **System-level/Index-based reuse (MemShare, MPCache):** 
    - Up to 85% throughput improvement (MemShare) [2507.21433].
    - 1.9× faster, 5.9× lower communication than full-cache in MPC inference (MPCache) [2501.06807].

## 5. Architectural and Systems Integration

- **Architectural considerations:** Most methods exploit the redundancy present in middle/deep layers, favoring sharing in those regions. Early layers often require distinct caches due to greater contextual variability (practices: share only for ℓ > L/3, or use dynamic gating) [2412.19442][2410.14442].
- **Hybridization and composability:** Orthogonal stacking of cross-layer sharing with intra-layer pruning (token selection), quantization (int4/int2), and autoencoding is common. Techniques are often “plug-and-play,” requiring no model retraining (e.g., xKV, KVSharer, CLLA) [2410.18517][2503.18893][2510.11236].
- **System-level orchestration:** Enterprise LLM serving benefits from cross-query and cross-session cache sharing, enabling prefill offloading, tiered memory orchestration, and even agent-level prefix alignment (LMCache, KVCOMM) [2510.09665][2510.12872].
- **Scheduling and dynamic selection:** Dynamic attention similarity estimation and online cost models (e.g., recompute/load tradeoff in Krul) enable optimal partitioning of which layers to share vs. recompute at restore time [2507.08045].

## 6. Trade-offs, Limitations, and Best Practices

| Challenge              | Manifestation                                         | Solution/Best Practice                                 |
|------------------------|------------------------------------------------------|--------------------------------------------------------|
| Accuracy trade-off     | Over-sharing leads to degraded PPL/accuracy          | Adaptive similarity thresholds; hybrid with quant/AE    |
| Layer/Head heterogeneity| Early layers/important heads are less redundant      | Share only deeper layers or non-critical heads          |
| Added compute overhead | SVD, similarity, or head alignment adds FLOPs        | Use static mapping, efficient pre-pass calibration      |
| System bottlenecks     | CPU/GPU sync, data movement can become limiting      | Batched movement, asynchronous scheduling, pointer tables|
| Hardware adaptability  | Integration with paged attention, tensor parallelism | Minimal code change, modular connectors                |

Empirical studies consistently find that moderate sharing rates (25–50% layers, or subset of heads) enable near-lossless (<2% PPL) compression and 1.3–2× throughput gains [2410.18517][2512.03870][2512.06727]. More aggressive ratios require accompanying low-rank or quantization methods—e.g., CLLA achieves ≈2% total cache size with lossless performance via the combination of cross-layer sharing, low-rank latent compression, and int4 quantization [2410.15252].

## 7. Outlook, Extensions, and Open Problems

The cross-layer KV cache sharing paradigm is rapidly extending into areas such as:

- **Fine-grained, semantic token-based sharing:** LSH-based token matching for cache pointer redirection (across prompts and/or layers) [2509.24832].
- **Dynamic, workload-aware hybridization:** Per-conversation or per-task optimization of sharing patterns and compression parameters, e.g., Krul’s token-wise attention pattern similarity [2507.08045].
- **Fusion with system-level pipelining and orchestration:** Multi-agent KV sharing, prefill offloading, and distributed cache management for LLM serving at scale [2510.09665][2510.12872].
- **Unified theoretical foundations:** A general theory of recoverability, redundancy, and information preservation under cross-layer re-use remains open.

Common open questions include: precise characterization of which layers/heads are best merged; universal standards for calibration and validation; and boundary conditions under continued scaling or more exotic architectures (e.g., Mixture-of-Experts). Nevertheless, cross-layer KV cache sharing is a foundational ingredient for the next generation of efficient LLM inference [2412.19442].

Source: https://www.emergentmind.com/topics/cross-layer-kv-cache-sharing