Krul: Dynamic LLM State Restoration

• Krul is a dynamic multi-turn LLM inference system that optimizes state restoration by employing conversation-specific cross-layer KV sharing.
• It uses a token-wise attention similarity estimator to efficiently compute pairwise layer distances, reducing time-to-first-token and storage overhead.
• Krul integrates a bubble-free restoration scheduler that balances recomputation and loading, preventing idle GPU resources while preserving generation quality.

Searching arXiv for the specified paper to ground the article in the cited source. Krul is a multi-turn LLM inference system for efficient state restoration in resumed conversations. It addresses the cost of reconstructing the attention state for historical tokens after inactive turns’ KV caches have been evicted from GPU memory. The central problem is that restoring state by either recomputing the full history or loading full KV caches inflates time-to-first-token (TTFT), while prior compressed-KV approaches commonly apply a fixed cross-layer sharing policy across all conversations despite substantial variation in attention similarity across prompts and layers. Krul introduces dynamic, conversation-specific cross-layer KV sharing, a token-wise attention similarity estimator, and a bubble-free recomputation–loading scheduler, with reported gains of $1.5\times$–$2.68\times$ TTFT reduction and $1.33\times$–$2.35\times$ KV cache storage reduction relative to state-of-the-art baselines, without compromising generation quality (Wen et al., 10 Jul 2025).

1. Problem setting and motivation

Multi-turn conversations dominate real-world LLM usage, reported as over 70% in ShareGPT, and they create a characteristic systems problem: historical KV caches are often evicted from GPU memory to save HBM, but when a conversation resumes the model must restore its internal attention state before decoding can continue (Wen et al., 10 Jul 2025). This restoration can proceed by full recomputation of the prompt history or by loading stored KV caches, but both strategies are expensive in different ways. Recompute increases prefilling work and TTFT, whereas full KV loading reduces compute at the cost of memory and I/O pressure.

The paper motivates the storage side quantitatively through an example: for Qwen1.5-7B, the KV cache for a single turn at 2K tokens is approximately 1 GB, which can quickly exhaust DRAM. SSD loading is slower than DRAM loading, so hierarchical storage does not eliminate the latency bottleneck. This makes simple prefix-cache restoration unattractive when historical turns are long or numerous.

Prior compressed-KV approaches exploit attention similarity across adjacent deep layers by storing shared KV representations rather than separate caches. Krul’s critique of these methods is not that cross-layer sharing is ineffective, but that fixed schemes choose the same layer pairs for all conversations. The paper reports that attention similarity varies widely across prompts and layers, and that different fixed strategies win on different tasks. A plausible implication is that a static compression policy over-compresses layers that are semantically important for a particular dialogue, thereby degrading future-turn accuracy.

Krul is therefore framed as a system for accurate and efficient state restoration under conversation-specific attention dynamics. Its design objective is to minimize TTFT while preserving critical context for future turns, rather than optimizing storage reduction alone.

2. Dynamic cross-layer KV sharing

The core mechanism in Krul is dynamic, conversation-specific cross-layer KV sharing selected at the end of each turn. Instead of enforcing a uniform layer-sharing map, Krul identifies which layers are likely safe to compress and then constructs a customized sharing policy from the observed attention patterns of that conversation (Wen et al., 10 Jul 2025).

A key distinction in the paper is among three layer behaviors:

• I-E layers: input-sensitive layers that selectively attend to task-relevant context.
• I-P layers: shallow processing layers with prompt-influenced structure.
• I-R layers: input-insensitive layers that consistently emphasize the initial and most recent tokens.

The system relies on the observation that I-R layers cover 75–87.5% of layers across models. Krul compresses only those KV caches likely to be safe to share, namely the I-R layers, and excludes non-I-R layers from compression. This selective scope is central to the claim that compression can be applied without harming quality.

The decision rule is based on the average attention assigned to the initial 10% and last 10% of tokens. For layer $l$, the paper defines:

$$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$

where $T_{initial}$ is the first 10% of tokens and $T_{recent}$ is the last 10% of tokens. If $\text{avg\_weight\_sum}(l) < \gamma$, Krul marks $l$ as non-I-R and excludes it from compression.

Among the remaining I-R layers, Krul computes pairwise attention similarity and greedily selects disjoint layer pairs with the smallest Euclidean distance between flattened attention maps until the number of shared layers reaches $2.68\times$0, where $2.68\times$1 is the layer compression ratio. The paper specifies the compression decision as:

$2.68\times$2

with $2.68\times$3 potentially implemented implicitly by ranking distances, and $2.68\times$4 used to keep pairs disjoint. Krul explicitly avoids sharing a layer multiple times, since repeated sharing can reduce quality.

This selector is heuristic rather than learned. It uses per-layer attention distributions over initial and recent spans, plus pairwise attention distances. The paper emphasizes that the overhead is low because the selector reads attention already computed during generation. This suggests that Krul is designed to fit into existing inference stacks with minimal perturbation to the forward path.

3. Token-wise heterogeneous attention similarity estimator

Krul’s dynamic selection requires estimating attention similarity without incurring the prohibitive memory cost of storing full attention tensors. The paper’s second innovation is a token-wise heterogeneous attention similarity estimator that splits the work across prefill and decode phases and across CPU and GPU resources (Wen et al., 10 Jul 2025).

For token $2.68\times$5, the attention vector for layer $2.68\times$6 is denoted $2.68\times$7. Krul defines the per-token distance and similarity between two layers as:

$2.68\times$8

These are then aggregated across tokens:

$2.68\times$9

The implementation ranks candidate pairs by distance, so smaller $1.33\times$0 is operationally better.

During prefilling, attention weights have shape $1.33\times$1. Krul asynchronously offloads them to CPU during the forward pass and computes distances there, optimized as $1.33\times$2 rather than direct $1.33\times$3. During decoding, the per-step attention tensor has shape $1.33\times$4. Krul retains only the last-token attention on GPU, computes $1.33\times$5 via torch.cdist, records the result, and frees memory immediately. The final similarity is an aggregation of CPU-side prefill and GPU-side decode results:

$1.33\times$6

with $1.33\times$7 and $1.33\times$8 reflecting token-count asymmetry; the paper notes that outputs are approximately $1.33\times$9 longer than inputs in ShareGPT.

The estimator is motivated by efficiency. The paper states that attention weights are 256× smaller than KV per decode step, and that offloading prefill attention to CPU avoids GPU OOM. CPU similarity computation finishes before decode ends, while GPU decode-side similarity overhead is negligible. The estimator therefore makes dynamic per-turn policy selection practical without storing full historical attention or introducing substantial latency.

A plausible implication is that Krul treats similarity estimation not as an auxiliary offline analysis step, but as a streaming systems primitive integrated into the normal generation pipeline.

4. Restoration scheduling and TTFT minimization

Krul’s third innovation is a bubble-free restoration scheduler. The problem is that once a conversation has been compressed with cross-layer sharing, restoring the state becomes a heterogeneous pipeline involving both cache loading and history recomputation. If those streams are not balanced, the system suffers stalls, termed “bubbles,” or can even produce prefilling errors when necessary hidden states are missing (Wen et al., 10 Jul 2025).

Krul formalizes the balance objective through a recomputation ratio $2.35\times$0, chosen to minimize the discrepancy between compute time and load time:

$2.35\times$1

where $2.35\times$2 and $2.35\times$3 denote total recompute FLOPs and load bytes, respectively, for layers $2.35\times$4, sequence length $2.35\times$5, and hidden dimension $2.35\times$6; $2.35\times$7 is GPU throughput and $2.35\times$8 is memory bandwidth. The paper recommends calibrating $2.35\times$9 by evaluating several candidate values on a small dataset and selecting the one minimizing $l$0.

Under Krul’s cross-layer sharing scheme, loaded KV volume increases with layer depth, while recompute volume decreases with depth. The paper describes this as a “pyramid overlap,” yielding complementary ramps that improve concurrent utilization. It defines per-layer load utilization and compute effort as:

$l$1

and

$l$2

This complementary structure enables overlapping the two streams more effectively than a flat or fixed partitioning.

Correctness is handled through hidden-state consistency. Deeper layers require shorter hidden-state prefixes than earlier layers, and Krul enforces:

$l$3

so that the union of previously computed states suffices for downstream layers. The scheduler detects possible gaps and redistributes recompute versus load to avoid missing-state errors. The reported TTFT model under overlap is:

$l$4

with

$l$5

Krul’s optimization target is therefore not merely lower load volume or lower recompute volume independently, but balanced overlap such that neither stream idles waiting for the other.

5. Architecture, implementation, and algorithms

Krul is implemented in PyTorch and targets standard transformer LLMs such as LLaMA and Qwen. The paper states explicitly that it does not modify the model’s attention math. Cross-layer KV sharing is a storage and restoration optimization: when layers $l$6 and $l$7 are shared, their K/V tensors for the same tokens are merged and represented by one KV per pair, following the compression method used in Minicache, and both layers read the shared KV during restoration (Wen et al., 10 Jul 2025).

KV caches are stored per layer, per head, per token, with tensor shapes $l$8 and $l$9, and a typical head_dim of 128. Compressed KV for I-R layers is offloaded to CPU at turn end, and restoration reloads it through separate I/O threads. Recomputed components use shorter prefixes via tensor slicing such as tensor[:target_length]. Attention weights are managed with CUDA streams for prefill offload to CPU; CPU computes similarities, while GPU handles per-token decode similarities and promptly frees temporary tensors.

The paper reports deployment on single and multiple NVIDIA GPUs, including A100, with DRAM and SSD over PCIe Gen4, and with continuous batching enabled. Qwen2-72B is evaluated using 8-bit model quantization, while KV precision follows the model default and remains compatible with Minicache.

The paper provides three algorithms in simplified form:

Algorithmic component	Main input	Main output
Preemptive compression strategy selector	model $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$0, context $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$1, threshold $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$2, layers $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$3, compression ratio $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$4	sharing map $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$5
Token-wise similarity estimation	prefill and decode attention	aggregated similarity $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$6
Restoration scheduler	$$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$7, recompute ratio $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$8, $$\text{avg\_weight\_sum}(l) = \frac{1}{L} \Big(\sum_{t \in T_{initial}} A_l(t) + \sum_{t \in T_{recent}} A_l(t)\Big),$$9, $T_{initial}$0	per-layer recompute/load plan

The selector runs a forward pass, classifies layers as I-R or non-I-R, computes pairwise distances among I-R layers, sorts pairs by increasing distance, and greedily builds a disjoint sharing policy. The similarity estimator combines CPU-side prefill distance computation with GPU-side decode-time torch.cdist, then aggregates the results at the end of the turn. The scheduler constructs per-layer recomputation and loading lengths, maintains separate compute and load queues, issues tasks through separate CUDA streams, monitors imbalance, and concatenates recomputed and loaded KV segments using torch.cat.

The paper’s practical guidance recommends setting $T_{initial}$1 from a small calibration so that roughly 70–85% of layers classify as I-R, starting with $T_{initial}$2, and sweeping $T_{initial}$3 to identify the best overlap point. It also recommends keeping pairs disjoint, prioritizing DRAM over SSD caches, and preserving shared-KV ownership in multi-GPU settings.

6. Empirical evaluation and comparative position

Krul is evaluated on LLaMA-7B, LLaMA-30B, Qwen1.5-7B, and Qwen2-72B, with Qwen2-72B using 8-bit quantization. LongBench is used for accuracy and ShareGPT for throughput and token-length analysis. The reported setting applies per-turn KV compression to historical context, with the last 128 tokens simulating new user input (Wen et al., 10 Jul 2025).

The main reported results are organized around TTFT, storage, and quality:

Metric	Reported result	Context
TTFT reduction vs SOTA	$T_{initial}$4–$T_{initial}$5	across tasks
TTFT reduction vs full recompute	avg $T_{initial}$6 for 7B/30B/Qwen1.5-7B; $T_{initial}$7 for Qwen2-72B	model-wise comparison
Prefill throughput improvement	up to $T_{initial}$8	throughput analysis
KV storage reduction	$T_{initial}$9–$T_{recent}$0	vs baselines
Average accuracy loss	$T_{recent}$1	across LongBench

The paper further states that Krul often outperforms fixed compression such as Minicache and eviction policies such as H2O, and that it matches or approaches baseline accuracy while beating Minicache on multiple tasks, including passretriev_zh. It also reports that fixed strategies do not generalize well: the dynamic selector frequently chooses layer ranges such as 9–24 rather than strictly the latter half of the network. This supports the claim that relevant similarity structure is not confined to adjacent deep layers.

The recompute/load ratio analysis reports that a ratio near 2:3 achieves the best overlap, and TTFT grows more slowly with prompt length under Krul. Overhead analysis shows that CPU similarity computation finishes before decode ends and GPU per-token similarity overhead is negligible. Together, these results position Krul as a systems method whose gains derive from combining compression, selection, and scheduling rather than from any single optimization in isolation.

7. Trade-offs, limitations, and interpretation

Krul’s limitations are described in terms of both workload dependence and systems calibration. Dynamic compression may underperform when layer attention similarities are uniformly low across a conversation, leaving few safe pairs, or when attention is highly heterogeneous and input-dependent, causing fewer layers to pass the I-R threshold $T_{recent}$2 (Wen et al., 10 Jul 2025). In such cases, the compression opportunity narrows, and the advantage over fixed policies may diminish.

The method introduces some overhead because it requires capturing attention weights, offloading prefill attention to CPU, and performing per-token decode-side similarity computation. The paper characterizes these costs as small, but they remain part of the design envelope. Similarly, the scheduler must be calibrated to choose an $T_{recent}$3 matched to the hardware. Miscalibration can reintroduce bubbles by making either the compute stream or the load stream dominant.

The approach is also sensitive to hyperparameters. The threshold $T_{recent}$4 governs I-R classification, the fraction $T_{recent}$5 controls how aggressively layers are compressed, and the aggregation weights $T_{recent}$6 and $T_{recent}$7 affect how prefill and decode behaviors influence pair selection. The paper warns that excessively aggressive $T_{recent}$8 can cause quality loss, and that inappropriate aggregation weights can bias the sharing map.

A further failure mode concerns correctness: incorrect partitioning of recompute and load, or sharing without proper hidden-state slicing, can produce prefilling errors. Krul addresses this through layer-wise consistency constraints and gap detection in the scheduler, but the existence of this failure mode is significant because it underscores that cross-layer sharing is not merely a storage transform; it affects restoration semantics.

More broadly, Krul is situated against several alternative restoration strategies. Full recompute yields high TTFT; full KV loading imposes heavy DRAM and SSD traffic; partial recompute/load schemes reduce only one side of the balance; H2O can reduce latency at the cost of future-turn accuracy by evicting tokens that later matter; and Minicache uses fixed adjacent-layer compression that may compress input-sensitive layers. Krul’s comparative claim is that conversation-adaptive selection among I-R layers, combined with token-wise similarity estimation and balanced restoration scheduling, preserves critical context while reducing both storage and TTFT.

This suggests a broader systems interpretation: Krul treats multi-turn state restoration as a joint optimization over representation sharing, hardware-aware scheduling, and prompt-specific attention structure, rather than as a purely caching problem.