Direct Cache Restoration Techniques

• Direct cache restoration is a set of mechanisms that selectively recovers evicted or displaced data and state from caches.
• It enhances performance in both hardware and LLM serving by balancing I/O and computation via techniques such as KV cache reload and intermediate activation restoration.
• Empirical evaluations demonstrate benefits like improved IPC and reduced latency, highlighting efficient use of reuse-distance metrics and scheduling strategies.

Direct cache restoration denotes a family of mechanisms that reinstate cache-resident data, restore equivalent state required by a cache consumer, or selectively reconstruct the subset of evicted state worth preserving after displacement, offloading, inactivity, or failure. In hardware cache hierarchies, it appears as selective copy-back of clean victim lines from an upper-level cache into a lower-level cache in exclusive or non-inclusive organizations (Wang et al., 2021). In stateful large-language-model serving, it includes direct KV-cache reload, restoration from intermediate activations, compressed-cache-assisted reconstruction, persistent quantized KV reuse, and filesystem-bypassing NVMe retrieval paths (Gao et al., 2024, Nian et al., 28 Apr 2026, Shkolnikov, 17 Feb 2026, Jeong et al., 29 Apr 2026). Related work extends the idea beyond literal cache copying to recovery of consistent persistent state after volatile-cache loss and to deliberate software rewarming after post-GPU shared-cache displacement (Zeng et al., 2022, Alpay et al., 25 Jun 2026).

1. Terminological scope and restoration targets

The literature does not use “direct cache restoration” in a single narrow sense. In CPU cache hierarchies, the restoration target is a displaced cache line, typically a clean victim from a private cache, and the question is whether that line should be copied back into the immediately lower-level cache. In this setting, the hierarchy policy is decisive: in an inclusive hierarchy, a clean line displaced from an upper level need not be restored because the lower level already contains it, whereas exclusive and non-inclusive hierarchies permit either discard or copy-back without violating correctness (Wang et al., 2021).

In LLM systems, the restored object is usually the transformer’s contextual state, especially the KV cache. Here the restoration target may be the full KV cache, a compressed or shared form of it, or an equivalent intermediate representation such as per-layer hidden states. HCache restores KV state from intermediate activations rather than from original tokens; CacheFlow treats restoration as a compute/I/O scheduling problem across tokens, layers, and GPUs; Krul restores state from compressed/shared KV caches plus recomputation; DUAL-BLADE makes KV retrieval behave more like a direct storage access path; and persistent Q4 KV cache systems restore quantized cache blocks directly into the attention path (Gao et al., 2024, Nian et al., 28 Apr 2026, Wen et al., 10 Jul 2025, Jeong et al., 29 Apr 2026, Shkolnikov, 17 Feb 2026).

A broader systems meaning appears in failure-recovery and phase-boundary studies. ReplayCache does not restore the volatile cache itself; instead, after a power outage, it re-executes the potentially unpersisted stores to restore the consistent NVM state from which interrupted execution can safely resume (Zeng et al., 2022). On Apple M4 Pro, a completed GPU command leaves residual shared-cache displacement, and one additional CPU traversal largely restores CPU-side performance by rewarming the cache (Alpay et al., 25 Jun 2026). This suggests that “restoration” may refer either to repopulating a cache structure or to re-establishing the effective working set needed by the next phase.

2. Selective direct restoration in CPU cache hierarchies

In the microarchitectural setting, the clearest formulation of direct cache restoration is the copy-back prediction policy, CBP, proposed for exclusive or non-inclusive hierarchies with STT-MRAM LLC (Wang et al., 2021). The motivating observation is that neither of the two obvious policies is satisfactory. Copying back all clean lines pollutes the lower-level cache with lines that will never be reused again, while copying back none loses lines that would soon be referenced again. The proposal is therefore selective restoration: only clean victim lines judged likely to be reused are copied back.

The paper defines reuse distance as the number of cache misses across all cache lines between two consecutive accesses to that same line, and in its motivation study treats lines with reuse distance greater than 1000 as dead (Wang et al., 2021). CBP combines three signals to compute a per-line priority: whether the line was prefetched, its hit frequency, and its reuse distance. It maintains a private reuse distance rd for each valid line and a set-level average reuse distance RD, both in 4-bit saturating counters. On every miss, rd of every valid line in the set is incremented; on a hit, the hit line’s rd is accumulated into RDsum, reset to zero, and after eight hits the set average is updated as RD ← RDsum / 8. The line with the highest priority becomes the victim. Dirty victims are copied back as usual; clean victims are copied back only if priority is below 9 (Wang et al., 2021).

The implemented environment is a two-level exclusive hierarchy in gem5 with a 32KB 8-way L1 ICache, a 32KB 8-way L1 DCache, a 1MB 16-way STT-MRAM L2/LLC, LRU for L1 ICache, CBP for L1 DCache, stride prefetching, 8GB DDR3-1600 DRAM, and an in-order 64-bit CPU core (Wang et al., 2021). The hardware overhead is reported as about 7 bits per cache line for rd plus a small number of bits per set for RD, counters, and related state, for a total area cost of roughly 1.3%. Across nine SPEC CPU 2017 benchmarks with high MPKI, CBP improves IPC by an average of 2.5%, reaches up to 12.8% higher IPC on 521.wrf_r, and copies back 2.9% fewer cache lines on average than the LRU baseline that copies back all clean lines. One benchmark, 548.exchange2_r, underperforms because copy-backs are rare in both schemes, so the restoration decision has little opportunity to matter (Wang et al., 2021).

The significance of this work is conceptual as well as empirical. It defines direct restoration not as unconditional write-back, but as a locality-sensitive decision over clean victims. The lower-level cache acts as a reservoir for potentially reusable lines, and reuse-distance-guided restoration attempts to preserve locality while avoiding lower-level pollution.

3. Direct restoration in LLM serving and persistent inference state

Stateful LLM serving has made direct cache restoration a first-order systems concern because only a limited number of contexts can remain resident in GPU memory. HCache addresses this by restoring LLM state from intermediate activations rather than from original tokens or from a full offloaded KV cache (Gao et al., 2024). For token $i$ at layer $L$, it uses the relations $K_L^i = W_L^k \cdot H_L^i$ and $V_L^i = W_L^v \cdot H_L^i$, stores hidden states, transmits them to GPU, and reconstructs KV cache with GEMM-style projections. The reported cost model gives hidden-state transmission as half the size of KV transmission and hidden-state-based recomputation as at least 6× faster than token recomputation. Evaluations report TTFT reductions of up to 1.93× versus KV offload and up to 5.73× versus token recomputation, while consuming 1.92–2.40× less storage space (Gao et al., 2024).

CacheFlow reframes KV restoration as a 3D-parallel scheduling problem rather than a per-request choice between recomputation and loading (Nian et al., 28 Apr 2026). Its token dimension divides the cached prefix into chunks of size $C$, commonly 512 tokens, and uses a forward compute pointer and a backward I/O pointer that meet in the middle. A similar two-pointer scheme is applied across layers, and distributed deployments exploit a GPU dimension through a boundary activation abstraction that allows each stage to restore its local KV cache concurrently. The paper gives an optimal split for the 2D token/layer formulation with harmonic-mean restoration time and reports TTFT reductions of 10%–62% over existing advances, with end-to-end improvements of 1.1×–1.7× overall and stronger gains on long-context workloads (Nian et al., 28 Apr 2026).

Persistent quantized KV restoration appears most explicitly in the multi-agent edge setting (Shkolnikov, 17 Feb 2026). There, each agent’s KV cache is quantized to Q4, stored to disk in safetensors, and later restored directly into the fused quantized attention kernel without first dequantizing into FP16. The system uses a block pool with fixed 256-token blocks isolated by agent ID, a model-agnostic ModelCacheSpec, a BatchQuantizedKVCache for concurrent inference, and cross-phase context injection that accumulates attention state across phases. On Apple M4 Pro, the paper reports that cache restoration reduces TTFT by up to 136x for Gemma 3 12B, 76x for DeepSeek-Coder-V2-Lite 16B, and 111x for Llama 3.1 8B, depending on context length, while Q4 fits 4x more agent contexts into fixed device memory than FP16 (Shkolnikov, 17 Feb 2026).

These systems share a common objective: replace repeated prefix replay with reuse of already computed state. They differ, however, in what is stored and restored. HCache stores hidden states, CacheFlow schedules mixed recomputation and I/O, and persistent Q4 systems restore quantized KV blocks directly.

4. Hybrid and storage-centric restoration paths

Not all restoration systems are direct full-cache reload schemes. Krul is explicit on this point: it is not a pure direct cache restoration method in the strict sense because it does not simply reload a preserved full KV cache (Wen et al., 10 Jul 2025). Instead, it stores a compressed version of the historical KV cache on CPU memory, selects a conversation-specific compression map $\mathcal{M}$ based on attention similarity across layer pairs, and later restores state by loading compressed/shared KV caches and recomputing missing caches in a bubble-free recomputation-loading pipeline. Its preemptive compression strategy selector identifies I-E, I-P, and I-R layers and targets compression at I-R layers; its token-wise heterogeneous attention similarity estimator splits similarity estimation between prefilling and decoding; and its scheduler chooses a recomputation ratio $r_c$ by minimizing $|T_C - T_L|$. Reported outcomes include TTFT reduction of about 1.28× to 2.68×, KV storage reduction of about 1.33× to 2.35×, and accuracy loss below 1% on average (Wen et al., 10 Jul 2025).

DUAL-BLADE approaches restoration from the storage-stack side and is best understood as a direct-cache-style retrieval path (Jeong et al., 29 Apr 2026). It partitions KV Placement Units into a page-cache path and an NVMe-direct path according to available memory, with the usable page-cache budget computed from $M^*$ and $B_{pc}$. Group 1 uses ordinary filesystem-backed mmap, while Group 2 bypasses the filesystem and maps tensors to contiguous NVMe logical block address regions represented by $L$0. It issues native NVMe commands with io_uring_cmd, preserves the sequential structure of the KV stream, and overlaps storage I/O with GPU DMA through adaptive pipeline parallelism. Across 2–11 GB host-memory limits, it reduces prefill latency by up to 33.1% and decode latency by up to 42.4% on SSD A, and by up to 25.4% and 57.8% on SSD B; SSD utilization improves by up to 2.2× (Jeong et al., 29 Apr 2026).

A common misconception is that restoration is synonymous with reloading an exact serialized cache image. The surveyed work shows a wider design space. Some systems restore the exact state representation consumed by the runtime, as in persistent Q4 KV caches. Others restore an equivalent state from a smaller or more convenient representation, as in HCache and Krul. Storage-centric systems such as DUAL-BLADE further emphasize that the directness of restoration can refer to the software path—bypassing filesystem and page-cache mediation—as much as to the semantic identity of the restored data.

5. Scheduling, balance, and the avoidance of restoration bubbles

A recurring technical theme is that restoration latency is rarely dominated by a single resource. HCache makes this explicit through a bubble-free restoration scheduler that mixes hidden-state restoration with a resource-complementary method (Gao et al., 2024). If computation is slower than I/O, part of the model state is restored through KV offload; if I/O is slower than computation, part is restored through token recomputation. It partitions layers into $L$1 and $L$2 and uses offline profiling to balance $L$3 against $L$4. Layer-wise partitioning is preferred because token-wise partitioning creates irregular GEMM shapes and is 7%–12% slower in the reported ablations (Gao et al., 2024).

CacheFlow generalizes this balancing logic into a batch-aware two-pointer scheduler (Nian et al., 28 Apr 2026). Each request maintains a compute pointer and an I/O pointer, and the scheduler allocates I/O to the requests with the largest remaining recomputation cost. Because recomputation cost grows superlinearly with prefix length, long requests derive higher marginal benefit from loading. The paper reports that disabling multi-GPU parallelism increases average restoration latency from 0.21 s to 0.29 s, a 38% increase, and that across batch sizes CacheFlow improves TTFT by 1.6×–2.6×, with larger batches benefiting more from the batch-aware scheduler (Nian et al., 28 Apr 2026).

Krul’s bubble-free scheduler is structurally similar but operates under compressed KV layouts (Wen et al., 10 Jul 2025). It defines recomputation and loading latencies as $L$5 and $L$6, then chooses $L$7 to minimize $L$8. The resulting pyramid-shaped distribution increases loading volume with layer index and decreases recomputation volume with layer index, while preserving the hidden-state dependencies required for correctness.

DUAL-BLADE applies the same principle to storage and DMA overlap (Jeong et al., 29 Apr 2026). After a warm-up decode iteration, it compares overlap-intra with overlap-cross and selects the better strategy for later iterations. The reported example on SSD A with $L$9 shows Group 1 throughput increasing from 11.93 GB/s to 13.13 GB/s under overlap-cross, which is then selected.

These results suggest that restoration is fundamentally a scheduling problem whenever the restored state is large enough to stress both compute and data movement. Directness alone does not determine performance; overlap discipline and contention-aware arbitration are often equally important.

6. Failure recovery, residual cache state, and security

Direct cache restoration also appears in contexts where the original cache state is lost or becomes hazardous. ReplayCache enables commodity energy-harvesting systems to use a volatile write-back cache without preserving dirty cachelines across power failure (Zeng et al., 2022). The compiler partitions code into regions so that store operand registers remain intact within each region, inserts CLWB after each store, and relies on just-in-time register checkpointing already available in systems such as NVP and QuickRecall. After a failure, ReplayCache identifies the interrupted region via a region register, consults the Recovery Map and Store Counting Map, re-executes the potentially unpersisted stores, restores registers, and resumes execution. It reports about 10.72x geometric-mean speedup over a no-cache baseline without power outages and 8.95x and 8.46x speedups on two outage traces, with binary size overhead about 1.2% and dynamic instruction count increase about 2.49% (Zeng et al., 2022). Here the restored object is consistent NVM state rather than the cache itself.

On Apple M4 Pro, restoration takes the form of software rewarming after a completed GPU command (Alpay et al., 25 Jun 2026). A synchronized Metal experiment shows that the first CPU traversal of a 16 MiB, 8192-byte-stride SLC-selective probe is slower after large GPU footprints, while a second traversal removes most of the cost and returns to the no-victim range. The paper interprets this as residual shared-cache displacement rather than simultaneous DRAM contention and describes the remedy as a simple one-pass software recovery mechanism: one deliberate CPU traversal before the latency-sensitive CPU phase (Alpay et al., 25 Jun 2026).

Security-oriented work exposes a different meaning of direct restoration. “Shadow in the Cache” shows that leaked KV-cache is an information-bearing intermediate state from which the original prompt can be reconstructed (Luo et al., 13 Aug 2025). Its Direct Cache Restoration via KV-cache Inversion Attack uses

$K_L^i = W_L^k \cdot H_L^i$0

when the relevant matrices are invertible, mainly for first-layer MHA caches. The more general Collision Attack compares candidate and leaked KV-cache entries with the Frobenius norm and can approach 100% recovery with prior knowledge. The Semantic-based Injection Attack reveals semantic leakage even without explicit inversion. KV-Cloak mitigates these attacks with reversible matrix-based obfuscation, one-time permutation, additive masking, and operator fusion, while reporting negligible accuracy degradation and runtime overhead around 5%, with specific models ranging roughly from 2.9% to 10.2% (Luo et al., 13 Aug 2025).

The broader implication is that restored cache state is not merely a performance artifact. It is operational state, and in some settings it is semantically equivalent to user data or program progress. Consequently, restoration mechanisms are constrained not only by latency and bandwidth, but also by correctness, crash consistency, isolation, and confidentiality.