Block-Level KV Caching

• Block-Level KV caching is a memory management technique that segments transformer key-value caches into fixed blocks to alleviate memory and throughput bottlenecks.
• It enables advanced cache compression, fusion, and quantization methods that achieve significant memory savings and performance improvements.
• Block-wise eviction and system integration techniques preserve positional fidelity and reduce latency, making it effective for long-context inference.

Block-level KV caching refers to a family of memory management and compression methodologies in large-scale transformer architectures that operate by partitioning the key–value (KV) cache into fixed-length blocks. These approaches address severe memory and throughput bottlenecks arising from the linear growth of KV caches, particularly in systems supporting long context, high concurrency, persistent prefix reuse, or multi-modal/block-wise generation. Block-level caching underpins high-performance inference, scalable resource partitioning, and advanced cache compression techniques across language, vision, and video-generation models.

1. Core Abstraction: Definition and Layout of KV-Cache Blocks

In transformer-based inference, each session maintains a KV cache with per-token, per-layer, and per-head hidden state projections, which are attended for efficient autoregressive generation. Rather than treating the entire sequence as a single large tensor, block-level schemes segment the cache into contiguous, fixed-size blocks or "pages." Formally, for batch size $B$, number of blocks per session $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$, tokens per block $t$, number of attention heads $h$, and head dimension $d$, the cache layout is $(B,\,p,\,t,\,h,\,d)$ (Kampeas et al., 6 Jan 2026, Chitty-Venkata et al., 4 Sep 2025).

This chunked organization reduces memory fragmentation (paging), enables hardware-aligned matrix–matrix operations, and establishes the unit over which compression, eviction, and sharing algorithms operate. PagedAttention in vLLM and similar systems implement such granular partitioning on GPU, with each physical or logical page mapped to a fixed number of cache tokens (Chitty-Venkata et al., 4 Sep 2025, Wang et al., 18 Dec 2025).

2. Block-Level Compression, Fusion, and Sharing Mechanisms

Block-level partitioning enables a wide spectrum of cache compression and sharing methods. The principal mechanism is joint encoding or fusion: nearly-parallel key and value blocks across requests or input segments (e.g., shared documents, code chunks) are identified and collapsed into a single normalized "direction" (unit vector) with per-block norms for each slot. This is algorithmically realized as a tree-structured pairwise fusion over all blocks:

• For unfolded $k \in \mathbb{R}^{n\times r}$ (keys) and $v \in \mathbb{R}^{n\times r}$ (values), with $r = t\cdot h\cdot d$, a similarity threshold $\text{thr}$ is set. Block pairs with $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$0 similarity $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$1 are merged and recursively fused, updating a block table to point slots at the shared representation (Kampeas et al., 6 Jan 2026). The complexity is $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$2 in the number of blocks.

Block-level alignment is critical for scalable position-independent caching: canonical padding and segment recomputation at block boundaries permit sharing segment-aligned blocks across requests and arbitrary positions, dramatically reducing high bandwidth memory (HBM) duplication (Wang et al., 18 Dec 2025). This approach underpins systems such as MEPIC, which recomputes only the first block of a chunk while reusing the remainder, made possible by chunk-aligned block padding and RoPE fusion in the attention kernel.

3. Block-Wise Eviction, Caching, and Positional Fidelity

Memory savings are actualized by block-wise eviction—removal of entire blocks/pages according to adaptive strategies. This is especially important for stateful inference, where cache length $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$3 unavoidably grows with conversation history or document context (Poudel, 23 Oct 2025).

Block removal preserves critical contiguousness: eliminating non-contiguous tokens (as in naive token-level eviction) destroys the integrity of positional encoding schemes, particularly rotary embeddings (RoPE), causing severe output degradation. Block-wise eviction strategies score and remove entire blocks:

• Block scoring can use sums or means of per-token attention weights, value/key L2-norm ratios, or segmental semantic weights (Chitty-Venkata et al., 4 Sep 2025Poudel, 23 Oct 2025Chen et al., 26 Oct 2025).
• Joint primacy–recency windows (e.g., preserve the first $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$4 blocks and last $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$5 blocks) maintain both gist and recency (Poudel, 23 Oct 2025).
• Semantic segmentation (as in SABlock) aligns compression with linguistic structure, adaptively searching for the largest block sizes per semantic unit while guaranteeing a minimum fidelity ratio per segment (Chen et al., 26 Oct 2025).

By enforcing block-level granularity and positional contiguity, these strategies preserve RoPE signals and maintain high-quality outputs even with aggressive cache pruning.

4. Block-Level Quantization, Compression, and Pareto Optimization

Block partitioning enables fine-grained, metadata-efficient quantization and transform-coding. Blocks of $p=\lceil\text{seq\_len} / \text{block\_size}\rceil$6 tokens ($p=\lceil\text{seq\_len} / \text{block\_size}\rceil$7) are quantized independently with scalar scale and zero-point, balancing quantization error and metadata overhead between per-token ($p=\lceil\text{seq\_len} / \text{block\_size}\rceil$8) and per-tensor ($p=\lceil\text{seq\_len} / \text{block\_size}\rceil$9) extremes (Gokhale et al., 1 Dec 2025):

• Each $t$0 block of keys/values is quantized (round-to-nearest) with per-block $t$1, $t$2, and the quantized tensor is stored.
• Mixed-precision schemes (distinct bitwidths for K/V) and K-smoothing further improve error–memory trade-off.
• Transform-based compression (e.g., KVTC) applies PCA-based feature decorrelation per block, then optimal scalar quantization and DEFLATE entropy coding for up to $t$3 size reduction with negligible accuracy degradation (Staniszewski et al., 3 Nov 2025).

KV Pareto demonstrates block granularity (typically $t$4–$t$5) supports $t$6 memory reduction with $t$7 accuracy loss; per-block quantization is empirically Pareto-optimal versus per-token or per-tensor schemes (Gokhale et al., 1 Dec 2025).

5. Implementation, Scheduling, and Systems Integration

Block-level caching is foundational for paging-based scheduling and scalable system integration:

• PagedAttention and similar memory managers build a block table mapping logical indices to physical device pages, enabling sparse allocation, sharing, and zero-fragmentation eviction (Chitty-Venkata et al., 4 Sep 2025, Wang et al., 18 Dec 2025).
• LayerKV and similar approaches introduce block-per-layer allocation and offloading to control instantaneous GPU pressure and queueing latency. SLO-aware schedulers optimize which blocks (layers) to keep in GPU, subject to system objectives (Xiong et al., 2024).
• For persistent prefix cache offloading, ContiguousKV introduces the "ContiguousChunk" abstraction, aligning block granularity with storage/I/O to eliminate read amplification and enable coordinated multi-layer asynchronous prefetching (Zou et al., 20 Jan 2026).
• In high-throughput serving, joint encoding and block sharing maximize concurrency while ensuring memory efficiency and minimal recomputation overhead (Kampeas et al., 6 Jan 2026, Wang et al., 18 Dec 2025).

Table: Fundamental Block-Level KV Caching Operations

Operation	Block-Level Mechanism	Key Reference
Paging/Allocation	Fixed-size block/page partition	(Chitty-Venkata et al., 4 Sep 2025)
Compression/Fusion	Block vector fusion/joint encoding	(Kampeas et al., 6 Jan 2026)
Eviction	Block scoring & page removal	(Poudel, 23 Oct 2025)
Quantization	Per-block (group-wise) quantization	(Gokhale et al., 1 Dec 2025)
Alignment/Reuse	Canonical chunk/block mapping	(Wang et al., 18 Dec 2025)
Offloading/I/O	Chunk-aligned caching/prefetching	(Zou et al., 20 Jan 2026)

6. Empirical Impact and Practical Considerations

Block-level KV caching strategies have demonstrated substantial gains in memory efficiency, throughput, and output quality across diverse LLMs, benchmarks, and workloads:

• Joint encoding of blocks achieves $t$8 compression with negligible loss and $t$940% throughput boost over baseline cache structures (Kampeas et al., 6 Jan 2026).
• Structured block-wise eviction yields up to $h$0 cache memory savings, with accuracy within $h$1 of full-cache for long-context summarization (Chitty-Venkata et al., 4 Sep 2025).
• Semantic segmentation and adaptive block sizing (e.g., SABlock) produce $h$2 memory reduction and $h$3 decoding speedup at 128k context length with nearly full baseline accuracy (Chen et al., 26 Oct 2025).
• Cache offloading and chunk-aligned I/O (ContiguousKV) enable $h$4 end-to-end latency reduction for persistent prefix cache loading (Zou et al., 20 Jan 2026).
• Block-level quantization, fused with AWQ model weight quantization, achieves optimal memory–accuracy frontiers for edge deployments, supporting up to $h$5k context inference (Gokhale et al., 1 Dec 2025).

Best practices include enforcing contiguous block boundaries, adaptively tuning block size to semantic structure, exploiting block reusability via chunk alignment, and co-optimizing quantization and paging at the block level for practical, high-throughput LLM serving.

7. Limitations, Extensions, and Theoretical Guarantees

Block-level approaches introduce tunable hyperparameters (block size, quantization bits, fusion thresholds) whose optimal settings are model- and workload-dependent. Limitations include:

• Coarse block granularity can dilute fine-grained token importance, though adaptive segmentation alleviates this risk (Chen et al., 26 Oct 2025).
• Stale block duplication is possible if alignment logic fails; robust chunk/block mapping is essential (Wang et al., 18 Dec 2025).
• Theoretical results guarantee minimum fidelity ratios per segment for adaptive block selection. For instance, SABlock ensures each segment retains at least a $h$6-fraction ($h$7) of the original attention-weighted information content (Chen et al., 26 Oct 2025).
• Hybrid block-wise/token-wise and per-layer block allocation are active areas of further system optimization, especially for extreme context length or tiered offloading scenarios (Xiong et al., 2024).

A plausible implication is that block-level KV caching, when coupled with semantic, structural, and quantization-aware policies, sets the foundation for scalable, memory-efficient, and high-quality inference in modern multi-modal and long-context transformer systems.