---
title: 'SemShareKV: Efficient Semantic KV Sharing'
url: https://www.emergentmind.com/topics/semsharekv
type: topic
---

# SemShareKV: Efficient Semantic KV Sharing

Searching arXiv for the cited papers to ground the article in current preprints.
SemShareKV denotes semantic key–value cache sharing for large language model inference, and more specifically the framework introduced in "SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching" [2509.24832]. It addresses a regime in which prompts are semantically similar but lexically different, so exact-prefix caching cannot exploit most of the latent reuse. The central idea is to reuse a reference prompt’s KV cache for a target prompt through token-level fuzzy matching over embeddings, while preserving positional information with Rotary Position Embedding (RoPE) and correcting mismatches through selective recomputation and retention. In the broader serving literature, the same problem class also appears in semantic-aware multi-tenant KV reuse systems such as KVShare [2503.16525], and in the distributed KV-management taxonomy it falls naturally into shared-store or hybrid-tier designs [2607.02574].

## 1. Problem setting and motivation

SemShareKV arises from the fact that long-context inference has shifted the KV cache from a minor runtime artifact into a dominant systems object. In a decoder-only transformer, each attention layer stores keys and values for every past token, and the KV footprint for one request is

$$
\text{KV bytes} = 2 \times L \times H_{\text{kv}} \times D \times S \times b,
$$

with the factor \(2\) accounting for keys and values. During decode, KV-read traffic scales as

$$
\sum_{t=0}^{T-1} (S+t) = TS + \frac{T(T-1)}{2},
$$

so long contexts create both memory-capacity pressure and bandwidth pressure [2607.02574].

The practical motivation is strongest when workloads contain semantically similar but lexically divergent prompts. The SemShareKV paper names multi-document summarization and conversational agents as prototypical cases, while the KVShare paper emphasizes repetitive template-like requests in healthcare, education, finance, and customer support. In such settings, many requests “say the same thing differently,” and recomputing nearly identical KV states for each request inflates prefill cost and Time to First Token (TTFT) [2509.24832][2503.16525].

This problem is distinct from merely serving longer prompts. KVShare explicitly frames the issue around cross-request reuse under multi-tenant scenarios, including cases where common text is not literally a prefix and where new tokens perturb attention during decode. SemShareKV sharpens that setting further: the reference and target prompts may be paraphrased, reordered, or only partially overlapping, so token identity is an insufficient reuse criterion [2503.16525][2509.24832].

## 2. Position within the KV-cache literature

SemShareKV occupies a specific niche between exact-match reuse and generic KV compression. Exact-prefix systems such as the prefix cache in vLLM and SGLang work when a new request starts with exactly the same prefix as a previous request, but they do not reuse common text that appears in the middle or end of a prompt. RAG-oriented segment reuse systems such as CacheBlend and Epic reuse block KV and repair cross-attention and positional effects with selective recomputation, but they focus on explicit user-uploaded context rather than free-form prompt similarity [2503.16525].

A second neighboring line is semantic response caching, exemplified by GPTCache and GPT Semantic Cache. These systems cache responses for semantically similar queries and return the cached answer when similarity exceeds a threshold. Their advantage is obvious latency reduction, but the limitation is equally clear: they do not reuse KV; they bypass the model. The KVShare evaluation highlights that this can collapse task fidelity on translation and summarization, because outputs must remain tightly coupled to the exact input rather than to coarse semantic similarity alone [2503.16525].

A third neighboring line is intra-prompt compression and retention. SemShareKV explicitly situates itself against token selection and eviction methods such as H2O, PyramidKV, SnapKV, Quest, and Scissorhands, as well as low-rank and quantization methods such as ShadowKV, PQCache, and Eigen Attention. These methods reduce KV for one prompt, whereas SemShareKV is fundamentally a cross-prompt reuse method [2509.24832].

Cross-layer KV sharing is related but orthogonal. "A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference" formalizes a mapping \(kv(i)\) that lets non-KV layers reuse another layer’s K/V, and "Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers" introduces FusedKV and FusedKV-Lite as asymmetric cross-layer reconstruction schemes. These methods exploit redundancy across layers inside one forward pass; SemShareKV exploits redundancy across semantically similar prompts across requests [2410.14442][2512.03870].

## 3. Core SemShareKV methodology

SemShareKV maintains three caches per prompt: an **E cache** of token embeddings, a **K cache**, and a **V cache**. Operationally, prompts and E caches are stored on CPU, an LSH index is built over token embeddings, and a new target prompt is matched to the most similar reference prompt above an empirical similarity threshold of approximately \(0.8\). On GPU, the reference KV cache is loaded, token-level matching aligns the reference and target token sequences, the reference KV is reordered accordingly, and the transformer then recomputes or evicts tokens layer by layer rather than recomputing the entire prompt from scratch [2509.24832].

The key technical move is token-level fuzzy matching by Locality-Sensitive Hashing on token embeddings rather than on raw text. For each target token, SemShareKV retrieves the nearest token in the reference prompt and defines a mapping

$$
\text{match}(i)=j.
$$

The reused KV cache is then formed by placing the reference token’s \(K/V\) at the target position \(i\). Because plain embeddings contain no positional information, the framework injects RoPE into embeddings before hashing. The RoPE operation for a 2D pair \((x_{2k},x_{2k+1})\) is

$$
\text{RoPE}(\mathbf{x}) =
\begin{bmatrix}
\cos(\theta_k) & -\sin(\theta_k) \\
\sin(\theta_k) & \cos(\theta_k)
\end{bmatrix}
\begin{bmatrix}
x_{2k} \\
x_{2k+1}
\end{bmatrix},
\qquad
\theta_k = 10000^{-2k/d}.
$$

The paper reports that without RoPE, many target tokens collapse onto the first few reference tokens, creating an attention-sink effect; with RoPE, early query tokens tend to match early query tokens and later context tokens map to later context tokens [2509.24832].

RoPE is also critical at the KV-storage level. SemShareKV evaluates three storage strategies and finds that **storing KV with RoPE already applied** yields the lowest deviation between reused and fully recomputed KV. Utility is measured by

$$
\sigma_K = \lVert K^{\text{reused}} - K^{\text{recomputed}} \rVert_2,\quad
\sigma_V = \lVert V^{\text{reused}} - V^{\text{recomputed}} \rVert_2,\quad
\sigma_{KV} = \sigma_K + \sigma_V.
$$

This design choice ties the positional structure used for token matching to the positional structure embedded in the stored cache [2509.24832].

The framework does not rely on raw reuse alone. In **Layer 1**, SemShareKV recomputes all tokens and uses the recomputed values as ground truth. In higher layers it selectively recomputes mismatched tokens and selectively evicts low-attention tokens. The policy is based on three observations reported by the paper: high-deviation tokens are stable across layers, deeper layers attend to fewer tokens, and deeper layers contain more redundant information. Tokens are partitioned into **Hot** and **Cold** sets using a dynamic ratio \(r_{\text{dynamic}}\) derived from **Attention Recovery**, defined as the minimal number of tokens needed to account for a target fraction such as \(55\%\) of total attention mass. The number of recomputed tokens is

$$
\text{Recomputed} = \omega_c \cdot c + \omega_h \cdot h,
$$

with \(\omega_c = 0.1\) and \(\omega_h = 0.5\) in the reported experiments, and the first-layer retention ratio is

$$
\text{Retained} = \max(0.8, r_{\text{dynamic}}).
$$

The intended effect is to preserve high-attention or high-deviation tokens while allowing aggressive reuse and eviction elsewhere [2509.24832].

## 4. System realizations and serving semantics

The broader serving realization of SemShareKV is exemplified by KVShare, which explicitly targets cross-request and multi-tenant KV reuse. Its pipeline consists of semantic similarity search over a vector database of past prompts, a **KV Editor** that aligns the new request with a retrieved similar request, **PartialAttention** for selective recomputation, and KV storage organized by a **DELTA Tree**. The system retrieves top-\(N\) similar requests using text embeddings from GTE or mGTE, computes edit operations between the stored and incoming prompts, reuses KV for unchanged tokens, inserts placeholder tensors for inserted or replaced tokens, and recomputes only the necessary subset during prefill and decode [2503.16525].

This edit-based design is a systems-level answer to the central semantic-alignment problem. KVShare’s walkthrough uses the contrast between prompts about **Jon Snow** and **Lin Daiyu** to show why naïve cache transplantation fails: the prompts are semantically similar as tasks but differ on entity names, book titles, and numeric constraints. The KV Editor treats the transformation from one prompt to another as a sequence-edit problem with **insert**, **delete**, and **replace** operations. The DELTA Tree then stores requests as nodes and edit deltas as edges so that the cheapest reuse path can be selected when many related prompts coexist [2503.16525].

KVShare also extends semantic reuse into scheduler design. Its abstract attributes its serving gains to two mechanisms: a **Dual-Stage High Deviation algorithm (DHD)** that conditionally selects a small portion of KV cache to be recomputed during both prefill and decode phases, and a **cache-aware scheduler** that prioritizes requests based on KV cache hit rates and orchestrates continuous batching. The same abstract reports that this multi-tenant design shares KV cache across requests without sacrificing model accuracy [2503.16525].

At the distributed-systems level, the 2026 KV-management survey places SemShareKV-style systems most naturally in the **shared-store** or **hybrid-tier** archetypes. Along the survey’s axes, semantic KV sharing aligns with **A2: Shared store** or **A3: Shared memory**, **B2: Cross-session** with **B3: Opportunistic** retention, and ownership models centered on either a coordinator or a distributed directory. The same survey emphasizes that shared prefixes should generally be treated as immutable, read-only KV objects, while each request appends a private suffix; this is the semantics most consistent with semantic KV reuse across users and tenants [2607.02574].

## 5. Empirical results

The SemShareKV evaluation centers on long-context text tasks and compares against **Full KV**, **SnapKV**, and **H2O**. On **MultiNews**, the paper reports **6.25× faster** TTFT than Full KV and **2.23× faster** TTFT than SnapKV. At **5k tokens**, it reports **42% reduction** in GPU KV memory. For quality, the main table reports the following ROUGE-L values. On **Mistral-7B**, MultiNews improves from **22.10** with Full KV to **23.15** with SemShareKV, and SAMSum improves from **18.79** to **21.22**. On **LLaMA-3.1-8B**, MultiNews changes from **22.49** to **23.18**, and SAMSum from **16.69** to **18.61**. The paper therefore characterizes generation quality as comparable to or better than Full KV on most evaluated datasets [2509.24832].

The ablations are important because they test whether fuzzy token matching carries real semantic signal. On SAMSum, **Ablation-Zero** drops ROUGE-L to **14.63** and **Ablation-Random** to **5.38**, versus **21.22** for SemShareKV. On MultiNews, the corresponding scores are **17.71**, **12.67**, and **23.15**. The same study also degrades the similarity between reference and target prompts by removing or replacing sentences; it reports that performance declines gradually, that reuse still works reasonably well even with **50% sentences removed**, and that degradation becomes more pronounced when similarity drops below approximately **0.8**, which motivates the empirical cutoff [2509.24832].

KVShare evaluates a somewhat different axis: multi-user prompt similarity and token-level hit rates. On **WildChat-Similar**, it reports that at least **60%** of requests have **\(\ge 90\%\)** of their tokens hit in the KV cache. On **ShareGPT-Similar**, the corresponding figure is **> 50%**, and on **LMSYSChat-Similar** it is **> 20%**. Its downstream evaluation uses **WMT19** and **SAMSum**. For **Qwen2.5-7B**, WMT19 BLEU changes from **0.3801** under Full Compute to **0.3576** under KVShare, while SAMSum ROUGE-L changes from **0.2234** to **0.2933**. For **Llama3-8B**, WMT19 changes from **0.3484** to **0.3574**, and SAMSum from **0.2322** to **0.2358**. In the same table, **GPTCache** collapses on both tasks, with WMT19 BLEU as low as **0.0399** and **0.0096** and SAMSum ROUGE-L as low as **0.1516** and **0.0544**, underscoring the difference between semantic KV sharing and semantic answer reuse. The KVShare abstract further reports **up to 9.39x** TTFT reduction, **1.2x** throughput increase relative to full KV recompute, and **20.38%** accuracy improvement over SOTA methods [2503.16525].

## 6. Limitations, misconceptions, and future directions

The most immediate limitation is workload dependence. SemShareKV reports that for prompts shorter than approximately **700 tokens**, LSH and cache-reordering overhead is not amortized and speedups become limited. It also notes sensitivity to LSH parameters, the global similarity threshold, and recompute/retain ratios such as \(\omega_c\), \(\omega_h\), \(\alpha_{\text{recomp}}\), and \(\alpha_{\text{retain}}\). The evaluation is concentrated on summarization and long-context text workloads, and the paper explicitly notes that it does not yet integrate **FlashAttention** [2509.24832].

KVShare identifies a related but more systems-oriented set of limits. It benefits most when a request stream contains a significant pool of similar prompts; in heterogeneous traffic, hit rates necessarily fall. Its walkthrough also marks the embedding model and similarity thresholds as critical design parameters, notes that DELTA Tree growth raises engineering questions around eviction and compaction, and describes multimodal support as conceptual rather than deeply evaluated. The same discussion flags open interactions with speculative decoding, quantization, and early exit [2503.16525].

Two recurrent misconceptions are addressed directly by the literature. First, semantic KV sharing is **not** semantic response caching: the former reuses internal state and still re-runs the model on the exact prompt, whereas the latter may return a stale or semantically mismatched answer. Second, semantic cross-request sharing is **not** the same as cross-layer KV sharing: the former transfers reuse across prompts, the latter across layers within a model. The distinction matters because the relevant failure modes differ—semantic alignment across prompts in one case, representational adequacy across depths in the other [2503.16525][2410.14442].

The distributed-systems literature adds further unresolved issues. The 2026 survey identifies missing KV-specific measurements for remote KV access patterns, metadata-path cost, completion and synchronization overhead, reuse-distance distributions, prefetchability slack, P99 tail attribution, and public trace availability. It also singles out robust shared-cache semantics, tenant isolation, tiered eviction, speculative decoding, MoE serving, per-session KV semantics, and durable KV as open problems. A plausible implication is that future SemShareKV systems will be evaluated not only by ROUGE-L, BLEU, TTFT, and memory reduction, but also by explicit ownership semantics and distributed-memory behavior under multi-tenant load [2607.02574].

Source: https://www.emergentmind.com/topics/semsharekv