DynSplit-KV: Dynamic KVCache Compression for LLMs

• The paper introduces DynSplit-KV, a method that dynamically splits semantic units in KVCache to maintain full-attention accuracy while drastically reducing memory and computational costs.
• It leverages dynamic importance-aware delimiter selection and a variable-to-fixed mapping strategy to efficiently compress variable-length semantic blocks for batched attention.
• Empirical evaluations reveal significant improvements, with faster inference speeds and reduced memory requirements compared to full-KV approaches, enabling practical long-context LLM deployments.

DynSplit-KV is a KVCache compression algorithm for transformer-based LLMs in long-context inference settings. It addresses the critical bottlenecks of exponential KVCache growth and corresponding peaks in inference latency by introducing dynamic semantic splitting for cache compression and an efficient mapping of variable-length blocks into fixed-length representations. By leveraging model attention dependencies to inform splitting, it matches full-attention accuracy with drastically reduced memory and computational requirements (Ye et al., 3 Feb 2026).

1. Motivation and Problem Background

Transformer LLMs rely on Key-Value caches (KVCache) to prevent redundant recomputation of hidden states at each decoding token. For a context of length $L$ tokens and dimensionality $d$, the memory requirement for the cache scales as

$\mathrm{Mem}_\text{KV} \approx O(L \times d)$

which becomes impractical at large context lengths (e.g., 80 GB for Llama2-13B at $L=128K$, exceeding typical single-GPU memory). Inference latency also grows due to the need to access all cached entries, making naive full-KV approaches inefficient. Prior compression strategies partition the context at rigid boundaries—either fixed-interval or on pre-defined delimiters such as punctuation. However, this “rigid” splitting results in severe accuracy degradation (ranging from 5.5% to 55.1% across benchmarks), as it is misaligned with semantic units that models actually attend to.

DynSplit-KV’s distinctive insight is that semantic units—the natural boundaries at which context can be split with minimal loss—are highly scenario- and domain-dependent. Dynamic strategies indexing into model attention are therefore required for optimal KVCache compression (Ye et al., 3 Feb 2026).

2. Dynamic Importance-Aware Delimiter Selection (DD-Select)

The splitting stage in DynSplit-KV employs a dynamic delimiter selection algorithm, DD-Select, which scores candidate boundaries to minimize downstream prediction degradation.

Token-level importance scoring considers a set $\mathcal{D}$ of potential delimiter positions (e.g., punctuation, newlines). For a delimiter $i \in \mathcal{D}$, regions are defined as follows:

• Retained: $\mathcal{O}_i = \{ k : i-R+1 \leq k \leq i \}$ (last $R$ tokens preceding $i$)
• Dropped: $\mathcal{D}_i = \{ k : 0 \leq k < i-R+1 \}$
• Future window: $d$0

An attention-derived score for each candidate is: $d$1 where $d$2 controls the penalty for long-range dependency.

Selection algorithm: For each chunk, a range around the ideal chunk length is considered and the delimiter maximizing a weighted sum of semantic score and proximity to the ideal position is chosen (see DD-Select pseudocode in (Ye et al., 3 Feb 2026)).

3. Uniform Mapping Strategy (V2F): Compression and Efficient Attention

The dynamic delimiter scheme produces variable-length semantic blocks $d$3. This variable-length format is incompatible with highly optimized, batched KV attention implementations.

To address this, DynSplit-KV introduces a V2F mapping (variable-to-fixed), compressing each block $d$4 into

$d$5

$d$6

Attention scores are then computed at the block level; only the top-$d$7 blocks are selected, and their scores are broadcast to all contained tokens. Ultimately, a global top-$d$8 token selection is performed prior to attention, so only the most relevant compressed representations are retained.

Complexity analysis:

• Block compression is $d$9
• Per-step block scoring and selection is $\mathrm{Mem}_\text{KV} \approx O(L \times d)$0 (with $\mathrm{Mem}_\text{KV} \approx O(L \times d)$1), yielding practical speedups over $\mathrm{Mem}_\text{KV} \approx O(L \times d)$2 full attention (Ye et al., 3 Feb 2026).

4. Integrated Inference Pipeline and System Details

During inference, DynSplit-KV operates as follows:

1. Prefill Stage: Full attention is run initially to compute attention maps across context.
2. DD-Select: Dynamic splitting produces variable-length semantic blocks.
3. V2F Mapping and Selection: Each block is compressed; block and token selection identifies blocks to retain.
4. FlashAttention: Only the selected tokens/blocks are attended to in the next decoding step using FlashAttention or similar.

For autoregressive decoding, only the most recent chunk boundaries are recalculated on-the-fly. The mechanism for merging old/new KV entries leverages the same V2F approach, ensuring memory efficiency and cache coherence (Ye et al., 3 Feb 2026).

5. Theoretical Complexity and Resource Reduction

The resource utilization of DynSplit-KV is sublinear in the sequence length,

$\mathrm{Mem}_\text{KV} \approx O(L \times d)$3

with $\mathrm{Mem}_\text{KV} \approx O(L \times d)$4 (for block size $\mathrm{Mem}_\text{KV} \approx O(L \times d)$5), compared to the baseline $\mathrm{Mem}_\text{KV} \approx O(L \times d)$6 memory. Inference throughput improves by a factor proportional to the average block size $\mathrm{Mem}_\text{KV} \approx O(L \times d)$7, with per-step time for update and attention dominated by compressed block-level operations.

In practical LLM workloads, these memory and time savings significantly outperform both rigid-splitting and uncompressed approaches. The V2F mapping ensures the overhead from variable block lengths is amortized via fixed-length representations, reducing inference complexity and system-level peak memory consumption (Ye et al., 3 Feb 2026).

6. Empirical Evaluation and Ablation Studies

Evaluation on the LongBench suite and custom retrieval tasks demonstrates:

• Accuracy: DynSplit-KV achieves average LongBench score of 42.6 with only 10% of KV cache, matching full attention and outperforming Quest/SentenceKV baselines by 5–10 points.
• Inference speed: 2.16× speedup over full FlashAttention on 32K input/4K output tasks.
• Peak GPU memory: Reduces memory from 80 GB (full-KV) to 30 GB at batch size 15 and context length 32K (a 2.6× reduction).
• Robust retrieval: In synthetic long-context retrieval, achieves 100% passkey recovery at 64-token cache budget (significantly outperforming alternatives, which need 256–512 tokens).
• Ablation: Disabling semantic weighting or length control yields 20–25% accuracy decrease; using alternative pooling for compression (e.g., mean) does not reduce the gains, confirming superiority of semantically aligned dynamic splitting rather than just block compression (Ye et al., 3 Feb 2026).

Method	LongBench (avg)	Speedup vs. FlashAttn	Peak GPU Memory (32K len, batch 15)
Full-KV	42.7	1×	80 GB
Quest	37.5	–	–
SentenceKV	31.6	–	–
DynSplit-KV	42.6	2.16×	30 GB

7. Broader Impact and Comparative Analysis

DynSplit-KV’s semantic-aware, attention-guided splitting distinguishes it from prior KVCache compression methods that employ inflexible chunking or delimiter logic. Rigid approaches introduce misalignment with underlying attention dependencies, compromising model accuracy especially for heterogeneous tasks and domains. In contrast, DynSplit-KV’s adaptive, importance-aware strategy enables consistent high-accuracy inference while reducing hardware requirements and operational costs.

A plausible implication is that variants of DynSplit-KV may enable practical ultra-long-context LLM deployments on single-GPU nodes and facilitate low-latency, memory-efficient production LLM systems for a broad range of applications—especially where context windows exceed tens of thousands of tokens and strict fidelity to salient context is crucial (Ye et al., 3 Feb 2026).