---
title: Chunk Level Caching (CLC)
url: https://www.emergentmind.com/topics/chunk-level-caching-clc
type: topic
---

# Chunk Level Caching (CLC)

Chunk Level Caching (CLC) denotes a family of caching and reuse strategies in which the operational unit is a **chunk** rather than an entire object or an individual token. The literature applies this idea in several settings: partial video-object caching guided by audience retention, path-coordinated chunk placement in content-centric networking, sub-query fragmentation in distributed data caching, key-value reuse and compression in long-context large language model inference, retrieval-augmented generation with position-independent chunk reuse, and chunkwise reuse in autoregressive video or world-model generation [1512.03274][1701.02524][1910.04991][2502.00299][2506.11418][2510.02361][2502.15734][2602.10825]. Taken together, these works treat chunking as a middle granularity that can reduce redundant traffic or computation while preserving more structure than token-level or byte-level policies.

## 1. Scope and definitions

The meaning of a “chunk” is system-dependent. In video-on-demand and content delivery, a chunk is a contiguous temporal segment of a video; in CCN it is a named content object; in distributed query caching it is an independently executable sub-query; in long-context LLM inference it may be a fixed-size token window, a semantic text segment, or a boundary-aware span; in autoregressive video generation it is a latent video segment processed sequentially [1512.03274][1701.02524][1910.04991][2502.00299][2602.10825].

| Domain | Chunk definition | Representative papers |
|---|---|---|
| Video delivery and CCN | Consecutive video segment or named content chunk | [1512.03274], [1701.02524] |
| Distributed query caching | Independently executable sub-query | [1910.04991] |
| Long-context LLM inference | Fixed-size window, semantic chunk, or boundary-aware segment | [2502.00299], [2506.11418], [2603.08453], [2510.02361] |
| RAG / position-independent KV reuse | Retrieved text chunk with reusable chunk-cache | [2502.15734], [2512.16822], [2603.20218] |
| Autoregressive video and world models | Latent video chunk or generation chunk | [2602.10825], [2605.01725], [2604.20289] |

Several papers make the same contrast explicitly: conventional systems often cache either whole files or token-level states, whereas CLC stores, selects, moves, or recomputes only parts of the object. In the video-caching literature, this appears as **partial caching** and **chunk-LRU** rather than whole-file LRU [1512.03274]. In LLM inference, ChunkKV states that it treats “semantic chunks—rather than isolated tokens—as basic compression units,” while Chelsea is described as “exactly a form of chunk-level KV caching” because it compresses the cache at the granularity of chunks into cluster centroids [2502.00299][2506.11418]. ChunkLLM likewise defines chunk boundaries and then keeps KV only for selected **key chunks** [2510.02361].

Taken together, these works suggest three recurring properties of CLC. First, chunking introduces a **control surface** between fully monolithic caching and per-token or per-byte decisions. Second, chunk definitions may be **fixed**, **learned**, **structure-aware**, or **workload-aware**. Third, chunk granularity is usually coupled to an explicit trade-off between semantic completeness, metadata overhead, and retrieval or recomputation cost.

## 2. Early formulations in networked content and distributed data systems

A foundational use of CLC appears in video delivery systems that exploit **audience retention rate**. For a video \(i\), the retention function is defined as
\[
R_i(\tau) = \Pr(\tau \in W_i), \qquad \tau \in [0,1],
\]
and the effective popularity of a chunk beginning at position \(x_{k-1}\) is \(p_i R_i(x_{k-1})\), combining video popularity \(p_i\) with within-video watch probability [1512.03274]. With full information, the optimal partial allocation is characterized by
\[
Y_i^*(\mu) = \{\tau \in [0,1] : p_i R_i(\tau) \ge \mu\},
\]
which becomes prefix caching under monotone abandonment. The practical policy introduced there, **chunk-LRU**, divides each video into \(N+1\) consecutive chunks, never caches the tail chunk \([\nu,1]\), and extends Che’s approximation to derive chunk hit probabilities
\[
h_{k,i} = 1 - e^{-p_i R_i(x_{k-1}) t_C}.
\]
The paper proves that refining chunk granularity improves performance and reports that even \(N=20\) chunks remains close to the infinitesimal-chunk optimum [1512.03274].

In content-centric networking, CLS defines a more explicit path-coordinated chunk policy. Each CCN router caches named chunks, but CLS enforces that there is **at most one copy of a chunk on the path between a server and a leaf router**. A hit pulls that copy one level toward the leaf router; an eviction pushes it one level toward the server; and a trail tuple
\[
(ID,\; in,\; out,\; h)
\]
records where later requests should search [1701.02524]. The paper reports that, in a three-level testbed with Zipf parameter \(0.9\) and cache size \(20\%\), the hit ratio was \(15.40\%\) for CLS, compared with \(14.43\%\) for LCD and \(14.28\%\) for CCN. It also reports about a \(12\%\) reduction in hit distance relative to CCN and LCD [1701.02524]. Here CLC is not only a storage policy but also a routing policy.

A related but semantically richer formulation appears in distributed database caching through **sub-query fragmentation**. There, a chunk is an independently executable sub-query
\[
q_i = \langle q_i^R,\; q_i^A,\; q_i^P,\; q_i^C \rangle,
\]
and cached query objects carry metadata including cache location, volume, complexity, per-location timestamps and frequencies, and associated queries:
\[
\langle \Diamond S,\; CLoc,\; V,\; C,\; \{T_{uLoc}\},\; \{F_{uLoc}\},\; \{D_i\}\rangle.
\]
The system uses Query Evaluation Trees whose leaves are chunk-like sub-queries, then promotes recurring or co-occurring fragments into cacheable objects and relocates them across a community cache network according to usage and locality [1910.04991]. This formulation broadens CLC from media delivery to query-aware distributed data systems.

## 3. KV-cache CLC in long-context LLM inference

Long-context LLM work uses CLC primarily for **KV cache compression, retrieval, and selection**. ChunkKV is the clearest semantic-preserving formulation. It divides the key sequence into fixed-size contiguous token windows of size \(c\), computes attention from an observe window of recent queries to all keys, aggregates token scores inside each chunk, and keeps the top-\(k\) chunks under a KV budget \(L_{\text{max}}\) [2502.00299]. The key claim is that a kept chunk preserves all of its tokens’ K/V states, avoiding fragmented context. The paper reports that KV cache can consume up to \(70\%\) of total memory during inference, and that ChunkKV’s layer-wise index reuse improves throughput by \(26.5\%\) while outperforming state-of-the-art methods by up to \(8.7\%\) in precision at the same compression ratio [2502.00299].

Chelsea uses chunking differently: it performs **online clustering and merging** of KV entries inside chunks. The sequence is divided into chunks, Chunked Soft Matching partitions each chunk into alternating sets, matches similar keys using cosine similarity, prunes low-similarity edges, and merges cluster members into weighted centroids [2506.11418]. The paper’s implementation-level motivation is that full similarity search is \(O(n^2)\), whereas local chunking reduces similarity computation to \(O(nc)\) when chunk size \(c\) is small. It reports up to **80% reduction in KV cache memory usage**, up to **3.19\(\times\)** decoding speedup, and up to **2.72\(\times\)** end-to-end latency reduction while maintaining comparable model performance [2506.11418]. Chelsea therefore treats a chunk as a local region within which redundancy can be safely collapsed.

LycheeCluster is a retrieval-oriented CLC design. It builds **structure-aware variable-length chunks** aligned to natural delimiters, computes a representative chunk key
\[
\bar{k}_j = \frac{1}{|s_j|} \sum_{i \in s_j} k_i,
\qquad
\bar{k}_j \leftarrow \frac{\bar{k}_j}{\|\bar{k}_j\|_2},
\]
and organizes these chunk embeddings into a hierarchical index with centroids and radii [2603.08453]. Retrieval then prunes whole branches by the upper bound
\[
q_t^\top v \le q_t^\top \mu_u + \|q_t\|_2 \cdot r_u.
\]
The paper reports that replacing Quest’s fixed-size pages with boundary-aware chunks yielded a **10.6% average accuracy** improvement and **15% on JSON** tasks in a pilot study, that fixed-size chunking caused a **3.03% drop** on LongBench V2 structured-data tasks, and that end-to-end inference speedup reaches **3.6\(\times\)** [2603.08453]. In this formulation, the chunk is a retrieval unit whose semantic coherence matters as much as its cache cost.

ChunkLLM turns chunk selection into a lightweight trainable module. A **Chunk Adapter** predicts whether a token is the last token of a chunk, and a **QK Adapter** computes chunk-level attention scores from compressed queries and boundary-token keys. Chunk selection is triggered only when the current token is detected as a chunk boundary, a property the paper calls **Intra-Chunk Attention Consistency** [2510.02361]. On long-context benchmarks it reports **48.58% key-value cache retention**, **98.64% of the performance** on long-context benchmarks, and a maximum **4.48\(\times\)** speedup at 120K tokens [2510.02361].

Taken together, these LLM papers separate CLC into at least three operational modes: **chunk-preserving compression** (ChunkKV), **chunk-internal merging** (Chelsea), and **chunk-level retrieval or selection** (LycheeCluster and ChunkLLM). That division is explicit in the mechanisms even when the term itself is used differently.

## 4. Position-independent chunk reuse in retrieval-augmented generation

RAG workloads expose a stricter version of CLC: the same retrieved text chunk may reappear at different positions and with different neighboring chunks. Cache-Craft formulates this as **chunk-cache** reuse. For a chunk \(C\) that originally appeared after prefix \(C_1:\dots:C_{i-1}\), the chunk-cache is
\[
\mathscr{C}\big(C \mid C_1 : \dots : C_{i-1}\big) := \{(K_C^l, V_C^l) \mid l \in [L]\}.
\]
The paper argues that naive reuse fails because cached K/V depend on both positional encoding and contextualization by preceding chunks [2502.15734]. To quantify that dependence, it defines **Chunk Context Impact**
\[
CCI(C_i) = \frac{1}{1 + e^{-\frac{\bar a}{\bar b}}},
\]
an adjusted prefix-overlap score \(\beta'\), and then the **Cache Fix Overhead**
\[
CFO(C_i \mid S_{\text{new}}) = \alpha \cdot CCI(C_i)\cdot (1 - \beta'(C_i \mid S_{\text{new}})).
\]
Only the most contextualized tokens in a reused chunk are recomputed. Cache-Craft reports **51%** less redundant computation than state-of-the-art prefix caching and **75%** less than full recomputation; with continuous batching on a production workload it reports **1.6X** throughput speedup and **2X** end-to-end latency reduction over prefix caching while maintaining quality for both LLaMA-3-8B and LLaMA-3-70B [2502.15734].

MEPIC addresses a different bottleneck: even when position-independent caching is correct enough, chunk KV often remains **request-specific in HBM** because positional encoding and selective recomputation make physically identical sharing impossible. MEPIC aligns chunk KV to paged storage, shifts recomputation from token-level to **block-level** so that only the first block is request-specific, stores KV without positional encoding, and applies RoPE inside a fused attention kernel at runtime [2512.16822]. The remaining blocks become fully shareable across positions, requests, and batches. The paper reports up to **2x** lower HBM usage than state-of-the-art PIC at comparable latency and accuracy, and up to **5x** for long prompts [2512.16822].

The experimental study of KV cache reuse strategies turns these mechanisms into a broader diagnosis. It evaluates naïve reuse, Cacheblend, EPIC, Link0, Cacheclip, APE, and fine-tuned BlockAttention-style designs, and concludes that existing CLC approaches have **fundamental limitations** because chunk caches computed in isolation miss cross-chunk dependencies [2603.20218]. The same paper also argues that the methods are **complementary**, then proposes **PSR (Prefix–Scale–Recompute)**, which combines fake prefix tokens for sink mitigation, APE-style attention scaling, and Cacheblend-style selective recomputation. It reports up to **5% absolute F1** improvement over the best single technique [2603.20218].

Taken together, these RAG and PIC papers identify three persistent difficulties of CLC: **position dependence**, **cross-chunk attention loss**, and **memory layout divergence**. Their solutions differ—selective repair, runtime positional fusion, or method composition—but the constraint is the same.

## 5. Chunk-level caching in autoregressive video and world models

In autoregressive video generation, CLC usually means **reusing chunk computations across denoising steps or across generation chunks** rather than storing document-like chunks. FlowCache is the explicit chunkwise formulation. A video is partitioned into chunks \(X_i\), and each chunk maintains an independent accumulator based on the relative latent change
\[
L1_{\text{rel}}(X,t,i) = \frac{\|X^i_{t-1} - X^i_t\|_1}{\|X^i_t\|_1}.
\]
The chunkwise policy recomputes or reuses each chunk independently according to the cumulative threshold rule \(f(X,t,i)\) [2602.10825]. FlowCache complements this with a joint importance–redundancy optimized KV cache compression mechanism for clean chunks. It reports **2.38 times** speedup on MAGI-1 and **6.7 times** on SkyReels-V2, with VBench changes of **0.87 increase** and **0.79 decrease** respectively [2602.10825].

MotionCache presents CLC partly as the baseline it aims to supersede. Existing methods such as FlowCache use a **binary per chunk** decision: the whole chunk is either recomputed or skipped. MotionCache argues that this is too coarse because motion is heterogeneous within a chunk; it formalizes cache error as
\[
\epsilon_{t-1}^i = \Delta t \cdot \| \mathcal{R}_{t-1}^i - \mathcal{R}_{t}^i \|_2,
\]
and then uses frame differences as a proxy for residual instability [2605.01725]. The resulting method keeps chunk-level decisions in an early warm-up phase but switches to motion-aware token-level reuse later. It reports speedups of **6.28\(\times\)** on SkyReels-V2 and **1.64\(\times\)** on MAGI-1, while VBench drops are **1%** and **0.01%** respectively [2605.01725]. This usefully clarifies that in video diffusion literature, “chunk-level caching” often denotes an all-or-nothing skip policy over whole latent chunks.

X-Cache moves the reuse axis again: it caches **block residuals across consecutive autoregressive chunks** in few-step world models. For block \(b\) at denoising step \(t\), it stores
\[
\hat{\mathbf r}_{t,b} \leftarrow \mathbf r^{(n)}_{t,b}
\]
and reuses it at the next chunk via
\[
\tilde{\mathbf x}^{(n+1)}_{t,b} = \mathbf x^{(n+1)}_{t,b-1} + \hat{\mathbf r}_{t,b}.
\]
A dual-metric gate over structure-aware and action-aware fingerprints decides whether to reuse or recompute, while **KV update chunks** are always fully recomputed so that approximate latents do not contaminate the rolling KV cache [2604.20289]. The paper reports **71% block skip rate** and **2.6x wall-clock speedup** with minimum degradation [2604.20289].

Taken together, these video and world-model papers show that CLC can refer to **chunkwise denoising reuse**, **chunkwise residual reuse**, or **chunkwise KV compression**. The commonality is the independent state maintained per chunk and the use of chunk-specific policies rather than a single global schedule.

## 6. Design principles, limitations, and adjacent directions

Several design principles recur across the literature. One is the importance of **granularity selection**. In chunk-LRU, finer chunking improves performance and even \(N=20\) chunks is close to optimal [1512.03274]. In Chelsea, performance is relatively insensitive for chunk sizes \(16,64,256\), but chunking is essential for efficiency [2506.11418]. LycheeCluster reports that structure-aware chunking outperforms fixed-size pages, and ChunkLLM reports that learned semantic boundaries outperform fixed segmentation on Needle-in-a-Haystack-style retrieval [2603.08453][2510.02361]. MotionCache, by contrast, argues that whole-chunk reuse becomes suboptimal when motion varies sharply within the chunk [2605.01725]. Taken together, these works suggest that chunk size is not only a systems parameter but also a semantic assumption.

A second principle is that CLC almost always requires **auxiliary metadata**. Examples include the trail tuple \((ID,in,out,h)\) in CCN, the retention-weighted popularity \(p_iR_i(\tau)\) in video delivery, cluster degrees in Chelsea, representative chunk keys in LycheeCluster, per-layer chunk votes in ChunkLLM, \(CCI\), \(\beta'\), and \(CFO\) in Cache-Craft, and canonical padded chunk hashes plus reference-counted block ownership in MEPIC [1701.02524][1512.03274][2506.11418][2603.08453][2510.02361][2502.15734][2512.16822]. CLC is therefore rarely “just” a storage format; it is typically an indexed policy with chunk-local state.

A third principle is that **cross-chunk dependence** is the central accuracy bottleneck in neural CLC. Cache-Craft addresses it by selective recomputation of the most contextualized tokens [2502.15734]. MEPIC addresses it partly by making the remaining blocks canonical and by fusing positional encoding at runtime [2512.16822]. The 2026 experimental study argues that isolated chunk prefills impose a fundamental limitation, and PSR’s improvement comes from combining sink mitigation, attention scaling, and selective recomputation rather than relying on a single fix [2603.20218]. This suggests that CLC for neural sequence models is most accurate when it incorporates some mechanism to repair or emulate missing global context.

Several adjacent lines of work extend chunk-level control without being conventional chunk caches. ChunkWise LoRA dynamically partitions sequences into variable-length chunks and couples them to **policy-driven KV-cache strategies** such as per-chunk quantization, windowing, and sparsification; it reports up to **34% lower latency** and **38% memory reduction** compared with baseline LoRA [2601.21109]. BlockFFN introduces **chunk-level sparsity (CLS)** in MoE-style FFNs, reporting over **80% TLS** and **70% 8-token CLS**, with kernels achieving up to **3.67\(\times\)** speedup on end-side devices [2507.08771]. A plausible implication is that chunk-level control is becoming a general systems abstraction, even when the cached object is no longer a classic KV block.

Across domains, CLC therefore denotes not one algorithm but a recurring systems idea: represent reuse, compression, movement, or recomputation at chunk granularity; choose chunks so that they align with locality, semantics, or dynamics; and supplement them with enough metadata or repair mechanisms to keep performance close to that of full computation.

Source: https://www.emergentmind.com/topics/chunk-level-caching-clc