Chunk Level Caching (CLC)

• Chunk Level Caching (CLC) is a strategy that caches intermediate data segments, balancing granularity between whole-object and token-level caching.
• It is applied in various domains such as video delivery, distributed databases, and long-context LLM inference to reduce redundant computation.
• Designs leverage metadata and context-aware mechanisms to manage trade-offs between cache storage overhead, semantic completeness, and recomputation cost.

Chunk Level Caching (CLC) denotes a family of caching and reuse strategies in which the operational unit is a chunk rather than an entire object or an individual token. The literature applies this idea in several settings: partial video-object caching guided by audience retention, path-coordinated chunk placement in content-centric networking, sub-query fragmentation in distributed data caching, key-value reuse and compression in long-context LLM inference, retrieval-augmented generation with position-independent chunk reuse, and chunkwise reuse in autoregressive video or world-model generation (Maggi et al., 2015, Li et al., 2017, Venkata et al., 2019, Liu et al., 1 Feb 2025, Hu et al., 13 Jun 2025, Ouyang et al., 28 Sep 2025, Agarwal et al., 5 Feb 2025, Ma et al., 11 Feb 2026). Taken together, these works treat chunking as a middle granularity that can reduce redundant traffic or computation while preserving more structure than token-level or byte-level policies.

1. Scope and definitions

The meaning of a “chunk” is system-dependent. In video-on-demand and content delivery, a chunk is a contiguous temporal segment of a video; in CCN it is a named content object; in distributed query caching it is an independently executable sub-query; in long-context LLM inference it may be a fixed-size token window, a semantic text segment, or a boundary-aware span; in autoregressive video generation it is a latent video segment processed sequentially (Maggi et al., 2015, Li et al., 2017, Venkata et al., 2019, Liu et al., 1 Feb 2025, Ma et al., 11 Feb 2026).

Domain	Chunk definition	Representative papers
Video delivery and CCN	Consecutive video segment or named content chunk	(Maggi et al., 2015, Li et al., 2017)
Distributed query caching	Independently executable sub-query	(Venkata et al., 2019)
Long-context LLM inference	Fixed-size window, semantic chunk, or boundary-aware segment	(Liu et al., 1 Feb 2025, Hu et al., 13 Jun 2025, Li et al., 9 Mar 2026, Ouyang et al., 28 Sep 2025)
RAG / position-independent KV reuse	Retrieved text chunk with reusable chunk-cache	(Agarwal et al., 5 Feb 2025, Wang et al., 18 Dec 2025, Cestola et al., 3 Mar 2026)
Autoregressive video and world models	Latent video chunk or generation chunk	(Ma et al., 11 Feb 2026, Xu et al., 3 May 2026, Zeng et al., 22 Apr 2026)

Several papers make the same contrast explicitly: conventional systems often cache either whole files or token-level states, whereas CLC stores, selects, moves, or recomputes only parts of the object. In the video-caching literature, this appears as partial caching and chunk-LRU rather than whole-file LRU (Maggi et al., 2015). In LLM inference, ChunkKV states that it treats “semantic chunks—rather than isolated tokens—as basic compression units,” while Chelsea is described as “exactly a form of chunk-level KV caching” because it compresses the cache at the granularity of chunks into cluster centroids (Liu et al., 1 Feb 2025, Hu et al., 13 Jun 2025). ChunkLLM likewise defines chunk boundaries and then keeps KV only for selected key chunks (Ouyang et al., 28 Sep 2025).

Taken together, these works suggest three recurring properties of CLC. First, chunking introduces a control surface between fully monolithic caching and per-token or per-byte decisions. Second, chunk definitions may be fixed, learned, structure-aware, or workload-aware. Third, chunk granularity is usually coupled to an explicit trade-off between semantic completeness, metadata overhead, and retrieval or recomputation cost.

2. Early formulations in networked content and distributed data systems

A foundational use of CLC appears in video delivery systems that exploit audience retention rate. For a video $i$, the retention function is defined as

$$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$

and the effective popularity of a chunk beginning at position $x_{k-1}$ is $p_i R_i(x_{k-1})$, combining video popularity $p_i$ with within-video watch probability (Maggi et al., 2015). With full information, the optimal partial allocation is characterized by

$$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$

which becomes prefix caching under monotone abandonment. The practical policy introduced there, chunk-LRU, divides each video into $N+1$ consecutive chunks, never caches the tail chunk $[\nu,1]$, and extends Che’s approximation to derive chunk hit probabilities

$h_{k,i} = 1 - e^{-p_i R_i(x_{k-1}) t_C}.$

The paper proves that refining chunk granularity improves performance and reports that even $N=20$ chunks remains close to the infinitesimal-chunk optimum (Maggi et al., 2015).

In content-centric networking, CLS defines a more explicit path-coordinated chunk policy. Each CCN router caches named chunks, but CLS enforces that there is at most one copy of a chunk on the path between a server and a leaf router. A hit pulls that copy one level toward the leaf router; an eviction pushes it one level toward the server; and a trail tuple

$$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$0

records where later requests should search (Li et al., 2017). The paper reports that, in a three-level testbed with Zipf parameter $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$1 and cache size $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$2, the hit ratio was $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$3 for CLS, compared with $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$4 for LCD and $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$5 for CCN. It also reports about a $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$6 reduction in hit distance relative to CCN and LCD (Li et al., 2017). Here CLC is not only a storage policy but also a routing policy.

A related but semantically richer formulation appears in distributed database caching through sub-query fragmentation. There, a chunk is an independently executable sub-query

$$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$7

and cached query objects carry metadata including cache location, volume, complexity, per-location timestamps and frequencies, and associated queries: $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$8 The system uses Query Evaluation Trees whose leaves are chunk-like sub-queries, then promotes recurring or co-occurring fragments into cacheable objects and relocates them across a community cache network according to usage and locality (Venkata et al., 2019). This formulation broadens CLC from media delivery to query-aware distributed data systems.

3. KV-cache CLC in long-context LLM inference

Long-context LLM work uses CLC primarily for KV cache compression, retrieval, and selection. ChunkKV is the clearest semantic-preserving formulation. It divides the key sequence into fixed-size contiguous token windows of size $$R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],$$9, computes attention from an observe window of recent queries to all keys, aggregates token scores inside each chunk, and keeps the top-$x_{k-1}$0 chunks under a KV budget $x_{k-1}$1 (Liu et al., 1 Feb 2025). The key claim is that a kept chunk preserves all of its tokens’ K/V states, avoiding fragmented context. The paper reports that KV cache can consume up to $x_{k-1}$2 of total memory during inference, and that ChunkKV’s layer-wise index reuse improves throughput by $x_{k-1}$3 while outperforming state-of-the-art methods by up to $x_{k-1}$4 in precision at the same compression ratio (Liu et al., 1 Feb 2025).

Chelsea uses chunking differently: it performs online clustering and merging of KV entries inside chunks. The sequence is divided into chunks, Chunked Soft Matching partitions each chunk into alternating sets, matches similar keys using cosine similarity, prunes low-similarity edges, and merges cluster members into weighted centroids (Hu et al., 13 Jun 2025). The paper’s implementation-level motivation is that full similarity search is $x_{k-1}$5, whereas local chunking reduces similarity computation to $x_{k-1}$6 when chunk size $x_{k-1}$7 is small. It reports up to 80% reduction in KV cache memory usage, up to 3.19$x_{k-1}$8 decoding speedup, and up to 2.72$x_{k-1}$9 end-to-end latency reduction while maintaining comparable model performance (Hu et al., 13 Jun 2025). Chelsea therefore treats a chunk as a local region within which redundancy can be safely collapsed.

LycheeCluster is a retrieval-oriented CLC design. It builds structure-aware variable-length chunks aligned to natural delimiters, computes a representative chunk key

$p_i R_i(x_{k-1})$0

and organizes these chunk embeddings into a hierarchical index with centroids and radii (Li et al., 9 Mar 2026). Retrieval then prunes whole branches by the upper bound

$p_i R_i(x_{k-1})$1

The paper reports that replacing Quest’s fixed-size pages with boundary-aware chunks yielded a 10.6% average accuracy improvement and 15% on JSON tasks in a pilot study, that fixed-size chunking caused a 3.03% drop on LongBench V2 structured-data tasks, and that end-to-end inference speedup reaches 3.6$p_i R_i(x_{k-1})$2 (Li et al., 9 Mar 2026). In this formulation, the chunk is a retrieval unit whose semantic coherence matters as much as its cache cost.

ChunkLLM turns chunk selection into a lightweight trainable module. A Chunk Adapter predicts whether a token is the last token of a chunk, and a QK Adapter computes chunk-level attention scores from compressed queries and boundary-token keys. Chunk selection is triggered only when the current token is detected as a chunk boundary, a property the paper calls Intra-Chunk Attention Consistency (Ouyang et al., 28 Sep 2025). On long-context benchmarks it reports 48.58% key-value cache retention, 98.64% of the performance on long-context benchmarks, and a maximum 4.48$p_i R_i(x_{k-1})$3 speedup at 120K tokens (Ouyang et al., 28 Sep 2025).

Taken together, these LLM papers separate CLC into at least three operational modes: chunk-preserving compression (ChunkKV), chunk-internal merging (Chelsea), and chunk-level retrieval or selection (LycheeCluster and ChunkLLM). That division is explicit in the mechanisms even when the term itself is used differently.

4. Position-independent chunk reuse in retrieval-augmented generation

RAG workloads expose a stricter version of CLC: the same retrieved text chunk may reappear at different positions and with different neighboring chunks. Cache-Craft formulates this as chunk-cache reuse. For a chunk $p_i R_i(x_{k-1})$4 that originally appeared after prefix $p_i R_i(x_{k-1})$5, the chunk-cache is

$p_i R_i(x_{k-1})$6

The paper argues that naive reuse fails because cached K/V depend on both positional encoding and contextualization by preceding chunks (Agarwal et al., 5 Feb 2025). To quantify that dependence, it defines Chunk Context Impact

$p_i R_i(x_{k-1})$7

an adjusted prefix-overlap score $p_i R_i(x_{k-1})$8, and then the Cache Fix Overhead

$p_i R_i(x_{k-1})$9

Only the most contextualized tokens in a reused chunk are recomputed. Cache-Craft reports 51% less redundant computation than state-of-the-art prefix caching and 75% less than full recomputation; with continuous batching on a production workload it reports 1.6X throughput speedup and 2X end-to-end latency reduction over prefix caching while maintaining quality for both LLaMA-3-8B and LLaMA-3-70B (Agarwal et al., 5 Feb 2025).

MEPIC addresses a different bottleneck: even when position-independent caching is correct enough, chunk KV often remains request-specific in HBM because positional encoding and selective recomputation make physically identical sharing impossible. MEPIC aligns chunk KV to paged storage, shifts recomputation from token-level to block-level so that only the first block is request-specific, stores KV without positional encoding, and applies RoPE inside a fused attention kernel at runtime (Wang et al., 18 Dec 2025). The remaining blocks become fully shareable across positions, requests, and batches. The paper reports up to 2x lower HBM usage than state-of-the-art PIC at comparable latency and accuracy, and up to 5x for long prompts (Wang et al., 18 Dec 2025).

The experimental study of KV cache reuse strategies turns these mechanisms into a broader diagnosis. It evaluates naïve reuse, Cacheblend, EPIC, Link0, Cacheclip, APE, and fine-tuned BlockAttention-style designs, and concludes that existing CLC approaches have fundamental limitations because chunk caches computed in isolation miss cross-chunk dependencies (Cestola et al., 3 Mar 2026). The same paper also argues that the methods are complementary, then proposes PSR (Prefix–Scale–Recompute), which combines fake prefix tokens for sink mitigation, APE-style attention scaling, and Cacheblend-style selective recomputation. It reports up to 5% absolute F1 improvement over the best single technique (Cestola et al., 3 Mar 2026).

Taken together, these RAG and PIC papers identify three persistent difficulties of CLC: position dependence, cross-chunk attention loss, and memory layout divergence. Their solutions differ—selective repair, runtime positional fusion, or method composition—but the constraint is the same.

5. Chunk-level caching in autoregressive video and world models

In autoregressive video generation, CLC usually means reusing chunk computations across denoising steps or across generation chunks rather than storing document-like chunks. FlowCache is the explicit chunkwise formulation. A video is partitioned into chunks $p_i$0, and each chunk maintains an independent accumulator based on the relative latent change

$p_i$1

The chunkwise policy recomputes or reuses each chunk independently according to the cumulative threshold rule $p_i$2 (Ma et al., 11 Feb 2026). FlowCache complements this with a joint importance–redundancy optimized KV cache compression mechanism for clean chunks. It reports 2.38 times speedup on MAGI-1 and 6.7 times on SkyReels-V2, with VBench changes of 0.87 increase and 0.79 decrease respectively (Ma et al., 11 Feb 2026).

MotionCache presents CLC partly as the baseline it aims to supersede. Existing methods such as FlowCache use a binary per chunk decision: the whole chunk is either recomputed or skipped. MotionCache argues that this is too coarse because motion is heterogeneous within a chunk; it formalizes cache error as

$p_i$3

and then uses frame differences as a proxy for residual instability (Xu et al., 3 May 2026). The resulting method keeps chunk-level decisions in an early warm-up phase but switches to motion-aware token-level reuse later. It reports speedups of 6.28$p_i$4 on SkyReels-V2 and 1.64$p_i$5 on MAGI-1, while VBench drops are 1% and 0.01% respectively (Xu et al., 3 May 2026). This usefully clarifies that in video diffusion literature, “chunk-level caching” often denotes an all-or-nothing skip policy over whole latent chunks.

X-Cache moves the reuse axis again: it caches block residuals across consecutive autoregressive chunks in few-step world models. For block $p_i$6 at denoising step $p_i$7, it stores

$p_i$8

and reuses it at the next chunk via

$p_i$9

A dual-metric gate over structure-aware and action-aware fingerprints decides whether to reuse or recompute, while KV update chunks are always fully recomputed so that approximate latents do not contaminate the rolling KV cache (Zeng et al., 22 Apr 2026). The paper reports 71% block skip rate and 2.6x wall-clock speedup with minimum degradation (Zeng et al., 22 Apr 2026).

Taken together, these video and world-model papers show that CLC can refer to chunkwise denoising reuse, chunkwise residual reuse, or chunkwise KV compression. The commonality is the independent state maintained per chunk and the use of chunk-specific policies rather than a single global schedule.

6. Design principles, limitations, and adjacent directions

Several design principles recur across the literature. One is the importance of granularity selection. In chunk-LRU, finer chunking improves performance and even $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$0 chunks is close to optimal (Maggi et al., 2015). In Chelsea, performance is relatively insensitive for chunk sizes $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$1, but chunking is essential for efficiency (Hu et al., 13 Jun 2025). LycheeCluster reports that structure-aware chunking outperforms fixed-size pages, and ChunkLLM reports that learned semantic boundaries outperform fixed segmentation on Needle-in-a-Haystack-style retrieval (Li et al., 9 Mar 2026, Ouyang et al., 28 Sep 2025). MotionCache, by contrast, argues that whole-chunk reuse becomes suboptimal when motion varies sharply within the chunk (Xu et al., 3 May 2026). Taken together, these works suggest that chunk size is not only a systems parameter but also a semantic assumption.

A second principle is that CLC almost always requires auxiliary metadata. Examples include the trail tuple $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$2 in CCN, the retention-weighted popularity $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$3 in video delivery, cluster degrees in Chelsea, representative chunk keys in LycheeCluster, per-layer chunk votes in ChunkLLM, $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$4, $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$5, and $$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$6 in Cache-Craft, and canonical padded chunk hashes plus reference-counted block ownership in MEPIC (Li et al., 2017, Maggi et al., 2015, Hu et al., 13 Jun 2025, Li et al., 9 Mar 2026, Ouyang et al., 28 Sep 2025, Agarwal et al., 5 Feb 2025, Wang et al., 18 Dec 2025). CLC is therefore rarely “just” a storage format; it is typically an indexed policy with chunk-local state.

A third principle is that cross-chunk dependence is the central accuracy bottleneck in neural CLC. Cache-Craft addresses it by selective recomputation of the most contextualized tokens (Agarwal et al., 5 Feb 2025). MEPIC addresses it partly by making the remaining blocks canonical and by fusing positional encoding at runtime (Wang et al., 18 Dec 2025). The 2026 experimental study argues that isolated chunk prefills impose a fundamental limitation, and PSR’s improvement comes from combining sink mitigation, attention scaling, and selective recomputation rather than relying on a single fix (Cestola et al., 3 Mar 2026). This suggests that CLC for neural sequence models is most accurate when it incorporates some mechanism to repair or emulate missing global context.

Several adjacent lines of work extend chunk-level control without being conventional chunk caches. ChunkWise LoRA dynamically partitions sequences into variable-length chunks and couples them to policy-driven KV-cache strategies such as per-chunk quantization, windowing, and sparsification; it reports up to 34% lower latency and 38% memory reduction compared with baseline LoRA (Thakkar et al., 28 Jan 2026). BlockFFN introduces chunk-level sparsity (CLS) in MoE-style FFNs, reporting over 80% TLS and 70% 8-token CLS, with kernels achieving up to 3.67$$Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},$$7 speedup on end-side devices (Song et al., 11 Jul 2025). A plausible implication is that chunk-level control is becoming a general systems abstraction, even when the cached object is no longer a classic KV block.

Across domains, CLC therefore denotes not one algorithm but a recurring systems idea: represent reuse, compression, movement, or recomputation at chunk granularity; choose chunks so that they align with locality, semantics, or dynamics; and supplement them with enough metadata or repair mechanisms to keep performance close to that of full computation.