Re-Prefill Phase in LLM Serving Pipelines

• Re-Prefill Phase is the process of retrieving shared prefix KV-cache segments from slower memory (e.g., SSD, CPU DRAM) to reconstruct the active cache state for LLM pipelines.
• It minimizes I/O bottlenecks by aligning token groups into ContiguousChunks and overlapping asynchronous prefetching with computation, thereby reducing read amplification and idle cycles.
• Empirical evaluations demonstrate up to 6.16× speedup and a 16.3× reduction in I/O volume while maintaining output quality within ±1% compared to full-cache methods.

A re-prefill phase is a critical component of LLM serving pipelines that exploit shared prompt prefixes across conversational, multi-turn, or retrieval-augmented workloads. This phase involves loading pre-computed key–value (KV) cache segments corresponding to a long, potentially shared prefix from secondary storage (e.g., CPU DRAM or SSD) and then integrating new or unique suffix tokens, culminating in the generation of the first output token. The re-prefill phase is a key enabler for bounded GPU and DRAM utilization in scenarios involving long user or document histories, chat systems, or repeated retrieval tasks. The fundamental systems challenge lies in bridging the divergent granularities and dependencies that arise when algorithmic KV-pruning operates at fine scales, while I/O layers stage data in coarse blocks, and in overlapping cache fetches with computation to maximally utilize available hardware.

1. Architectural Role and System Requirements

The re-prefill phase is invoked whenever a serving system must reconstruct the active KV-cache state for a persistent, long prefix that is not currently resident on the fast path (GPU/DRAM), either because of memory pressure, multi-turn sharing, or cross-session KV reuse (Zou et al., 20 Jan 2026). Operationally, the phase performs two actions: (1) it loads the existing prefix KV-cache from a slower memory tier (such as SSD or off-GPU RAM) based on an importance-pruned selection, and (2) it computes the new cache entries for any non-shared suffix (such as new prompt turns or appended tokens) and performs a cross-attention step over the aggregate to generate the first token output. This phase is central for scalable serving in multi-turn dialogue, search pipelines, and retrieval-augmented generation and is especially pressing in settings where active cache sizes can exceed tens of thousands of tokens.

2. I/O and Compute Bottlenecks in Traditional Re-Prefill

Two principal bottlenecks afflict mainstream re-prefill implementations:

A. Read Amplification Due to Granularity Mismatch: Semantic-aware KV-pruning typically selects individual tokens or small groups (e.g., c=16 tokens) as “important” for cache restoration. However, existing systems (e.g., IMPRESS) load KV data in large, fixed-size blocks (B=64 tokens typical), resulting in a read-amplification factor of $A = B/k$ when the number of required tokens $k \ll B$. This leads to read amplification routinely exceeding $10\times$ to $50\times$, severely under-utilizing I/O bandwidth and prolonging the time-to-first-token (TTFT) (Zou et al., 20 Jan 2026).

B. Sequential Dependency and Idle Bubbles: Standard serving stacks execute prefix KV cache loading for each layer strictly before proceeding with query computation and scoring. This per-layer serializing creates idle “bubbles,” with either the GPU stalled waiting for cache loads or the storage device underutilized during computation. Even attempts at overlapping compute and I/O in previous systems are insufficient to saturate both resources effectively (Zou et al., 20 Jan 2026).

3. ContiguousKV—Granularity Alignment and Asynchronous Prefetch

ContiguousKV offers a unified solution that eliminates both inefficiencies via three mechanisms:

A. ContiguousChunk Granularity: The n-token prefix is partitioned into $N = \lceil n / c \rceil$ contiguous “chunks” of exactly $c$ tokens (e.g., $c=16$). All operations—pruning, eviction, I/O—operate uniformly at the ContiguousChunk level. Let chunk $j$ hold tokens $t_{(j-1)c+1},\dots,t_{jc}$, and let $c \cdot S$ (with $k \ll B$0KB/token in Qwen2.5-7B) denote chunk size in storage. This alignment collapses the read amplification to $k \ll B$1 and eliminates over-fetch relative to coarse block-based systems (Zou et al., 20 Jan 2026).

B. Two-Level Asynchronous Prefetch: Across transformer layers, the set of selected ContiguousChunks varies only slowly. By aggregating computation over periods of $k \ll B$2 layers (e.g., $k \ll B$3), with each period $k \ll B$4 sharing a chunk index set $k \ll B$5, and overlapping prefetches both within a period (intra-period) and speculatively across consecutive periods (inter-period), the system pre-loads relevant chunks ahead of demand. With 52–64% overlap in chunk indices between adjacent periods, this pipelining achieves near-continuous hardware utilization and largely hides I/O stall (Zou et al., 20 Jan 2026).

C. Attention-Guided Cache Management: Within limited GPU/DRAM, which chunks to retain is decided by a semantic score derived from cumulative token attention mass and chunk access frequency:

• $k \ll B$6 (token $k \ll B$7 attention mass)
• $k \ll B$8 (chunk $k \ll B$9 attention sum)
• Update: $10\times$0 and $10\times$1 every access
• Cache score per chunk: $10\times$2 Min-heaps on both the GPU and CPU evict the lowest-$10\times$3 chunks, thus attention (semantic importance) dictates residency under memory pressure (Zou et al., 20 Jan 2026).

4. Quantitative Performance Evaluation and Comparative Results

Comprehensive benchmarks on Qwen2.5-7B demonstrate:

• Speedup: ContiguousKV reduces TTFT by $10\times$4 versus IMPRESS, $10\times$5 versus AS+H2O+LFU, and $10\times$6 versus AS+LRU, for a 5% memory budget.
• Read Reduction: Only ~6% as many tokens need to be loaded from SSD compared to IMPRESS, a $10\times$7 reduction in I/O volume.
• Tail Latency: P95 TTFT drops by up to $10\times$8 s on the RTE benchmark.
• Accuracy: Output quality is preserved (within $10\times$9 of full-cache AS+LRU) and ContiguousKV outperforms IMPRESS by 3–8% at low resource budgets.
• Memory Overhead: Out of a 10 GB GPU + 24 GB CPU budget, only 0.2 GB/0.4 GB is allocated for prefetch buffers, with the remainder managed by attention-based heaps (Zou et al., 20 Jan 2026).

5. Theoretical and Practical Insights for Engineering Re-Prefill

Design lessons include:

• Aligning I/O block size to semantically meaningful units (token groups) is essential to minimizing read amplification. The ContiguousChunk abstraction generalizes to arbitrary group sizes.
• Inter-layer and inter-period chunk reuse can be exploited aggressively for non-blocking prefetching; future work could dynamically adapt period size or predict future chunk importance.
• Attention-based eviction policies can be extended beyond the per-request scope, e.g., for cross-session or federated cache management.
• The ContiguousKV design is orthogonal to other advances such as KV quantization or cluster-scale SSD caching, and is readily extensible to novel storage tiers with finer addressability.

6. Broader Impact and Future Directions

ContiguousKV transforms the re-prefill phase from an I/O-bound bottleneck to a pipelined, semantically directed process without loss of output quality. It addresses fundamental scalability challenges in emerging long-context, multi-turn, and shared-prefix serving workloads. Open directions include combining granularity-aligned I/O with learned or adaptive chunk predictors and integrating with distributed or federated cache architectures. The outlined approach—in unifying data management granularity, decoupling I/O and computation via period-preemptive pipelining, and using intrinsic model signals for cache residence—provides a template for future systems seeking to achieve both efficiency and accuracy in large-scale, memory-aware LLM deployment (Zou et al., 20 Jan 2026).