SemShareKV: Efficient Semantic KV Sharing

• SemShareKV is a semantic key–value caching method designed to improve inference efficiency by reusing token-level caches across semantically similar prompts.
• It employs token-level fuzzy matching via Locality-Sensitive Hashing combined with Rotary Position Embedding to align and reorder cached states accurately.
• Empirical results demonstrate significant reductions in Time to First Token and memory usage for long-context tasks, improving overall model serving performance.

Searching arXiv for the cited papers to ground the article in current preprints. SemShareKV denotes semantic key–value cache sharing for LLM inference, and more specifically the framework introduced in "SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching" (Zhao et al., 29 Sep 2025). It addresses a regime in which prompts are semantically similar but lexically different, so exact-prefix caching cannot exploit most of the latent reuse. The central idea is to reuse a reference prompt’s KV cache for a target prompt through token-level fuzzy matching over embeddings, while preserving positional information with Rotary Position Embedding (RoPE) and correcting mismatches through selective recomputation and retention. In the broader serving literature, the same problem class also appears in semantic-aware multi-tenant KV reuse systems such as KVShare (Yang et al., 17 Mar 2025), and in the distributed KV-management taxonomy it falls naturally into shared-store or hybrid-tier designs (Li et al., 30 Jun 2026).

1. Problem setting and motivation

SemShareKV arises from the fact that long-context inference has shifted the KV cache from a minor runtime artifact into a dominant systems object. In a decoder-only transformer, each attention layer stores keys and values for every past token, and the KV footprint for one request is

$$\text{KV bytes} = 2 \times L \times H_{\text{kv}} \times D \times S \times b,$$

with the factor $2$ accounting for keys and values. During decode, KV-read traffic scales as

$\sum_{t=0}^{T-1} (S+t) = TS + \frac{T(T-1)}{2},$

so long contexts create both memory-capacity pressure and bandwidth pressure (Li et al., 30 Jun 2026).

The practical motivation is strongest when workloads contain semantically similar but lexically divergent prompts. The SemShareKV paper names multi-document summarization and conversational agents as prototypical cases, while the KVShare paper emphasizes repetitive template-like requests in healthcare, education, finance, and customer support. In such settings, many requests “say the same thing differently,” and recomputing nearly identical KV states for each request inflates prefill cost and Time to First Token (TTFT) (Zhao et al., 29 Sep 2025, Yang et al., 17 Mar 2025).

This problem is distinct from merely serving longer prompts. KVShare explicitly frames the issue around cross-request reuse under multi-tenant scenarios, including cases where common text is not literally a prefix and where new tokens perturb attention during decode. SemShareKV sharpens that setting further: the reference and target prompts may be paraphrased, reordered, or only partially overlapping, so token identity is an insufficient reuse criterion (Yang et al., 17 Mar 2025, Zhao et al., 29 Sep 2025).

2. Position within the KV-cache literature

SemShareKV occupies a specific niche between exact-match reuse and generic KV compression. Exact-prefix systems such as the prefix cache in vLLM and SGLang work when a new request starts with exactly the same prefix as a previous request, but they do not reuse common text that appears in the middle or end of a prompt. RAG-oriented segment reuse systems such as CacheBlend and Epic reuse block KV and repair cross-attention and positional effects with selective recomputation, but they focus on explicit user-uploaded context rather than free-form prompt similarity (Yang et al., 17 Mar 2025).

A second neighboring line is semantic response caching, exemplified by GPTCache and GPT Semantic Cache. These systems cache responses for semantically similar queries and return the cached answer when similarity exceeds a threshold. Their advantage is obvious latency reduction, but the limitation is equally clear: they do not reuse KV; they bypass the model. The KVShare evaluation highlights that this can collapse task fidelity on translation and summarization, because outputs must remain tightly coupled to the exact input rather than to coarse semantic similarity alone (Yang et al., 17 Mar 2025).

A third neighboring line is intra-prompt compression and retention. SemShareKV explicitly situates itself against token selection and eviction methods such as H2O, PyramidKV, SnapKV, Quest, and Scissorhands, as well as low-rank and quantization methods such as ShadowKV, PQCache, and Eigen Attention. These methods reduce KV for one prompt, whereas SemShareKV is fundamentally a cross-prompt reuse method (Zhao et al., 29 Sep 2025).

Cross-layer KV sharing is related but orthogonal. "A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference" formalizes a mapping $kv(i)$ that lets non-KV layers reuse another layer’s K/V, and "Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers" introduces FusedKV and FusedKV-Lite as asymmetric cross-layer reconstruction schemes. These methods exploit redundancy across layers inside one forward pass; SemShareKV exploits redundancy across semantically similar prompts across requests (Wu et al., 2024, Lin et al., 3 Dec 2025).

3. Core SemShareKV methodology

SemShareKV maintains three caches per prompt: an E cache of token embeddings, a K cache, and a V cache. Operationally, prompts and E caches are stored on CPU, an LSH index is built over token embeddings, and a new target prompt is matched to the most similar reference prompt above an empirical similarity threshold of approximately $0.8$. On GPU, the reference KV cache is loaded, token-level matching aligns the reference and target token sequences, the reference KV is reordered accordingly, and the transformer then recomputes or evicts tokens layer by layer rather than recomputing the entire prompt from scratch (Zhao et al., 29 Sep 2025).

The key technical move is token-level fuzzy matching by Locality-Sensitive Hashing on token embeddings rather than on raw text. For each target token, SemShareKV retrieves the nearest token in the reference prompt and defines a mapping

$\text{match}(i)=j.$

The reused KV cache is then formed by placing the reference token’s $K/V$ at the target position $i$. Because plain embeddings contain no positional information, the framework injects RoPE into embeddings before hashing. The RoPE operation for a 2D pair $(x_{2k},x_{2k+1})$ is

$$\text{RoPE}(\mathbf{x}) = \begin{bmatrix} \cos(\theta_k) & -\sin(\theta_k) \ \sin(\theta_k) & \cos(\theta_k) \end{bmatrix} \begin{bmatrix} x_{2k} \ x_{2k+1} \end{bmatrix}, \qquad \theta_k = 10000^{-2k/d}.$$

The paper reports that without RoPE, many target tokens collapse onto the first few reference tokens, creating an attention-sink effect; with RoPE, early query tokens tend to match early query tokens and later context tokens map to later context tokens (Zhao et al., 29 Sep 2025).

RoPE is also critical at the KV-storage level. SemShareKV evaluates three storage strategies and finds that storing KV with RoPE already applied yields the lowest deviation between reused and fully recomputed KV. Utility is measured by

$2$0

This design choice ties the positional structure used for token matching to the positional structure embedded in the stored cache (Zhao et al., 29 Sep 2025).

The framework does not rely on raw reuse alone. In Layer 1, SemShareKV recomputes all tokens and uses the recomputed values as ground truth. In higher layers it selectively recomputes mismatched tokens and selectively evicts low-attention tokens. The policy is based on three observations reported by the paper: high-deviation tokens are stable across layers, deeper layers attend to fewer tokens, and deeper layers contain more redundant information. Tokens are partitioned into Hot and Cold sets using a dynamic ratio $2$1 derived from Attention Recovery, defined as the minimal number of tokens needed to account for a target fraction such as $2$2 of total attention mass. The number of recomputed tokens is

$2$3

with $2$4 and $2$5 in the reported experiments, and the first-layer retention ratio is

$2$6

The intended effect is to preserve high-attention or high-deviation tokens while allowing aggressive reuse and eviction elsewhere (Zhao et al., 29 Sep 2025).

4. System realizations and serving semantics

The broader serving realization of SemShareKV is exemplified by KVShare, which explicitly targets cross-request and multi-tenant KV reuse. Its pipeline consists of semantic similarity search over a vector database of past prompts, a KV Editor that aligns the new request with a retrieved similar request, PartialAttention for selective recomputation, and KV storage organized by a DELTA Tree. The system retrieves top-$2$7 similar requests using text embeddings from GTE or mGTE, computes edit operations between the stored and incoming prompts, reuses KV for unchanged tokens, inserts placeholder tensors for inserted or replaced tokens, and recomputes only the necessary subset during prefill and decode (Yang et al., 17 Mar 2025).

This edit-based design is a systems-level answer to the central semantic-alignment problem. KVShare’s walkthrough uses the contrast between prompts about Jon Snow and Lin Daiyu to show why naïve cache transplantation fails: the prompts are semantically similar as tasks but differ on entity names, book titles, and numeric constraints. The KV Editor treats the transformation from one prompt to another as a sequence-edit problem with insert, delete, and replace operations. The DELTA Tree then stores requests as nodes and edit deltas as edges so that the cheapest reuse path can be selected when many related prompts coexist (Yang et al., 17 Mar 2025).

KVShare also extends semantic reuse into scheduler design. Its abstract attributes its serving gains to two mechanisms: a Dual-Stage High Deviation algorithm (DHD) that conditionally selects a small portion of KV cache to be recomputed during both prefill and decode phases, and a cache-aware scheduler that prioritizes requests based on KV cache hit rates and orchestrates continuous batching. The same abstract reports that this multi-tenant design shares KV cache across requests without sacrificing model accuracy (Yang et al., 17 Mar 2025).

At the distributed-systems level, the 2026 KV-management survey places SemShareKV-style systems most naturally in the shared-store or hybrid-tier archetypes. Along the survey’s axes, semantic KV sharing aligns with A2: Shared store or A3: Shared memory, B2: Cross-session with B3: Opportunistic retention, and ownership models centered on either a coordinator or a distributed directory. The same survey emphasizes that shared prefixes should generally be treated as immutable, read-only KV objects, while each request appends a private suffix; this is the semantics most consistent with semantic KV reuse across users and tenants (Li et al., 30 Jun 2026).

5. Empirical results

The SemShareKV evaluation centers on long-context text tasks and compares against Full KV, SnapKV, and H2O. On MultiNews, the paper reports 6.25× faster TTFT than Full KV and 2.23× faster TTFT than SnapKV. At 5k tokens, it reports 42% reduction in GPU KV memory. For quality, the main table reports the following ROUGE-L values. On Mistral-7B, MultiNews improves from 22.10 with Full KV to 23.15 with SemShareKV, and SAMSum improves from 18.79 to 21.22. On LLaMA-3.1-8B, MultiNews changes from 22.49 to 23.18, and SAMSum from 16.69 to 18.61. The paper therefore characterizes generation quality as comparable to or better than Full KV on most evaluated datasets (Zhao et al., 29 Sep 2025).

The ablations are important because they test whether fuzzy token matching carries real semantic signal. On SAMSum, Ablation-Zero drops ROUGE-L to 14.63 and Ablation-Random to 5.38, versus 21.22 for SemShareKV. On MultiNews, the corresponding scores are 17.71, 12.67, and 23.15. The same study also degrades the similarity between reference and target prompts by removing or replacing sentences; it reports that performance declines gradually, that reuse still works reasonably well even with 50% sentences removed, and that degradation becomes more pronounced when similarity drops below approximately 0.8, which motivates the empirical cutoff (Zhao et al., 29 Sep 2025).

KVShare evaluates a somewhat different axis: multi-user prompt similarity and token-level hit rates. On WildChat-Similar, it reports that at least 60% of requests have $2$8 of their tokens hit in the KV cache. On ShareGPT-Similar, the corresponding figure is > 50%, and on LMSYSChat-Similar it is > 20%. Its downstream evaluation uses WMT19 and SAMSum. For Qwen2.5-7B, WMT19 BLEU changes from 0.3801 under Full Compute to 0.3576 under KVShare, while SAMSum ROUGE-L changes from 0.2234 to 0.2933. For Llama3-8B, WMT19 changes from 0.3484 to 0.3574, and SAMSum from 0.2322 to 0.2358. In the same table, GPTCache collapses on both tasks, with WMT19 BLEU as low as 0.0399 and 0.0096 and SAMSum ROUGE-L as low as 0.1516 and 0.0544, underscoring the difference between semantic KV sharing and semantic answer reuse. The KVShare abstract further reports up to 9.39x TTFT reduction, 1.2x throughput increase relative to full KV recompute, and 20.38% accuracy improvement over SOTA methods (Yang et al., 17 Mar 2025).

6. Limitations, misconceptions, and future directions

The most immediate limitation is workload dependence. SemShareKV reports that for prompts shorter than approximately 700 tokens, LSH and cache-reordering overhead is not amortized and speedups become limited. It also notes sensitivity to LSH parameters, the global similarity threshold, and recompute/retain ratios such as $2$9, $\sum_{t=0}^{T-1} (S+t) = TS + \frac{T(T-1)}{2},$0, $\sum_{t=0}^{T-1} (S+t) = TS + \frac{T(T-1)}{2},$1, and $\sum_{t=0}^{T-1} (S+t) = TS + \frac{T(T-1)}{2},$2. The evaluation is concentrated on summarization and long-context text workloads, and the paper explicitly notes that it does not yet integrate FlashAttention (Zhao et al., 29 Sep 2025).

KVShare identifies a related but more systems-oriented set of limits. It benefits most when a request stream contains a significant pool of similar prompts; in heterogeneous traffic, hit rates necessarily fall. Its walkthrough also marks the embedding model and similarity thresholds as critical design parameters, notes that DELTA Tree growth raises engineering questions around eviction and compaction, and describes multimodal support as conceptual rather than deeply evaluated. The same discussion flags open interactions with speculative decoding, quantization, and early exit (Yang et al., 17 Mar 2025).

Two recurrent misconceptions are addressed directly by the literature. First, semantic KV sharing is not semantic response caching: the former reuses internal state and still re-runs the model on the exact prompt, whereas the latter may return a stale or semantically mismatched answer. Second, semantic cross-request sharing is not the same as cross-layer KV sharing: the former transfers reuse across prompts, the latter across layers within a model. The distinction matters because the relevant failure modes differ—semantic alignment across prompts in one case, representational adequacy across depths in the other (Yang et al., 17 Mar 2025, Wu et al., 2024).

The distributed-systems literature adds further unresolved issues. The 2026 survey identifies missing KV-specific measurements for remote KV access patterns, metadata-path cost, completion and synchronization overhead, reuse-distance distributions, prefetchability slack, P99 tail attribution, and public trace availability. It also singles out robust shared-cache semantics, tenant isolation, tiered eviction, speculative decoding, MoE serving, per-session KV semantics, and durable KV as open problems. A plausible implication is that future SemShareKV systems will be evaluated not only by ROUGE-L, BLEU, TTFT, and memory reduction, but also by explicit ownership semantics and distributed-memory behavior under multi-tenant load (Li et al., 30 Jun 2026).