- The paper presents a resolution-adaptive KV cache (SeKV) that preserves all context information using hierarchical semantic memory to address GPU memory bottlenecks in long-context LLM inference.
- It uses entropy-guided semantic segmentation with dual caching—GPU-resident anchors and CPU-based low-rank SVD—for selective reconstruction of token-level details.
- Empirical evaluations show SeKV achieves up to +5.9% improvement in retrieval tasks and reduces GPU memory usage by 53.3% at 128K context compared to full KV caching.
SeKV: Resolution-Adaptive Hierarchical Semantic Memory for Efficient Long-Context LLM Inference
Motivation and Problem Statement
The practical deployment of long-context LLMs is limited not by context window length, but by the GPU memory bottleneck induced by the Key-Value (KV) cache. The cache grows linearly with sequence length and must be retained throughout autoregressive decoding, making full KV caching infeasible for extended contexts. Standard approaches to KV compression, such as token eviction, quantization, or static semantic chunking, either permanently discard nonlocal information or irreversibly merge fine-grained evidence, resulting in irrecoverable performance loss on retrieval, reasoning, or multi-turn tasks. SeKV is introduced as a response to the demand for memory-scalable, adaptively recoverable context representations that preserve access to all relevant information across large inputs.
(Figure 1)
Figure 1: Legacy token eviction removes semantically critical tokens, leading to retrieval failures and hallucination, while SeKV retains all information in semantic spans and dynamically zooms in on query-relevant regions.
SeKV Architecture and Methodology
SeKV reframes the KV cache as a hierarchical, resolution-adaptive memory system organized around semantic spans. The cache is decomposed into two representations: a lightweight summary vector for each entropy-guided span (retained on GPU) for fast routing, and a CPU-resident low-rank SVD basis for accurate token-level reconstruction activated on demand. High-surprisal tokens at semantic boundaries are always preserved as anchors to guarantee precise evidence retrieval. During decoding, a trained zoom-in controller selects spans requiring full-resolution expansion, triggering asynchronous SVD base transfer from CPU to GPU. This allows the system to reconstruct token-level KV states for only the most query-relevant parts of the context.
(Figure 2)
Figure 2: SeKV overview—entropy-guided segmentation, anchor/summary on GPU, SVD bases on CPU, and a two-stage adaptive routing and reconstruction pipeline.
SeKV is trained with distillation from a full KV teacher, using a composite loss: (1) output distribution alignment, (2) direct supervision of the zoom-in policy for retrieving spans with high attention mass, (3) low-rank reconstruction loss to set a per-span rank budget, and (4) an explicit memory budget term ensuring overall efficiency. The base LLM remains frozen.
Semantic Span Segmentation and Representation
Segmentation boundaries are detected by computing token-level surprisal during prefill, with thresholds based on statistical deviation ($\alpha$-scaled standard deviation above the mean). Within each span, keys and values are projected to routing space using learned per-head/layer projections and summarized via surprisal-weighted pooling. The low-rank SVD factorization is also supervised by a parameterized soft gating function for adaptive fidelity.
Hierarchical Memory and Inference
The SeKV memory hierarchy consists of (a) GPU-resident, full-fidelity anchors/summary vectors for rapid attention routing, and (b) CPU-hosted, low-rank SVD factors for reconstructing token-level detail as needed. At each generation step, the zoom-in mechanism (a learned per-head thresholded sigmoid gate) determines which spans should trigger reconstruction, with fine-grained attention computed only over these distributed reconstructions and the always-on anchors. This dual pipeline ensures that the majority of context remains highly compressed, with expensive token-level operations restricted to crucial evidence regions.
Empirical Evaluation
SeKV is evaluated on LongBench, RULER, InfiniteBench, NIAH, and many-shot GSM8K reasoning. Across all four core benchmarks and five LLM backbones (including Llama-3.1-8B, Qwen2.5-14B, and Mistral-7B), SeKV strictly outperforms prior compression methods under matched memory budgets.
Key results:
- SeKV delivers an average +5.9% gain over the strongest prior semantic compression baseline (SentenceKV) at 10% GPU-resident KV budget.
- On memory-constrained, retrieval-sensitive tasks such as NIAH (Needle-in-a-Haystack), SeKV’s retrieval success is both quantitatively and qualitatively more stable across all positions and context lengths.
(Figure 3)
Figure 3: In NIAH benchmarks, SeKV maintains stable, high-fidelity retrieval across needle depth and context length, avoiding the degradation seen for token-eviction and static compression.
- The memory/throughput efficiency is highly competitive. SeKV achieves a 53.3% reduction in GPU memory use compared to full KV caching at 128K context, with nearly flat GPU memory scaling as context increases, in marked contrast to the steep growth for Full KV.
(Figure 4)
Figure 4: SeKV maintains nearly-flat GPU memory usage as context length grows, distinguishing itself from Full KV's linear growth profile.
- Latency and throughput of SeKV are improved over Full KV; it comes close to the fastest eviction-based methods but without the severe accuracy tradeoffs.
- On many-shot arithmetic reasoning (GSM8K), SeKV preserves both demonstration structure and sparse retrieval, yielding up to +2.3 accuracy over SentenceKV and within 2.4 points of Full KV reference.
Adaptivity and Specialization Analysis
Zoom-in analysis reveals that span expansions are highly selective, concentrated in specific mid/late transformer layers and a subset of heads. This supports the use of per-head/layer thresholds and validates the architectural focus on adaptive rather than static memory expansion.
(Figure 5)
Figure 5: Layer/head-wise analysis shows selective zoom-in activation, with most heads rarely reconstructing token-level detail.
(Figure 6)
Figure 6: Across models, zoom-in activity consistently clusters in retrieval-oriented heads/layers and tasks, validating SeKV's adaptive resolution control.
Theoretical and Practical Implications
SeKV’s core innovation is in the decoupling of evidence preservation from expensive full-resolution caching. The approach systematically avoids irreversible information loss: unlike token-eviction strategies, all information remains recoverable with cost proportional only to the query's need. Compared to static semantic grouping, SeKV addresses the “late evidence” problem by enabling on-demand detail recovery, crucial for tasks where peripheral input becomes relevant only upon extended generation or mid-conversation topical shifts.
On the theoretical side, SeKV’s success highlights the crucial role of query-adaptive memory access in long-context LLMs, motivating future directions in memory hierarchies, dynamic compression policies, and architectural extensions for unified training of both routing and core modeling. Its design is inherently compatible with frozen pre-trained backbones, requiring only a lightweight routing adaptation, supporting broader deployment in resource-constrained, upstream-agnostic, or multi-backbone scenarios. The span segmentation and low-rank policies are likely extensible to non-text domains (e.g., code, structured data) where high-entropy boundaries and semantic coherence are non-aligned with tokenization.
Future developments may explore tighter integration with hardware-aware transfer/compression, further joint optimization of routing and model weights, application to multimodal LLMs, and advanced recovery of extremely long-range dependencies.
Conclusion
SeKV provides a resolution-adaptive, semantic-aware cached memory for LLM inference that achieves state-of-the-art compression/accuracy trade-offs for long-context processing. By balancing efficient summary-based routing with selective token-level span reconstruction, it closes the gap between memory efficiency and retrieval fidelity—enabling practical deployment of LLMs at context windows far beyond the limits of naive caching. The work establishes dynamic span-based compression as the preferred approach in regimes where access to rare but highly distributed evidence is essential for downstream reasoning and retrieval performance.
Reference: "SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference" [2606.31145]
