KV-only Prefill: Reusable KV-State Inference

• KV-only prefill is a long-context inference technique that precomputes and reuses key/value caches to eliminate redundant prompt processing.
• It serializes per-layer key/value tensors with metadata to ensure token-exact reuse and achieve significant speedups in model decoding.
• The approach spans methods from shared-prefill modules to selective recomputation, offering cost efficiency and improved throughput in inference systems.

KV-only prefill is a family of long-context inference techniques that treat the prefill stage not as an ephemeral computation to be repeated for every request, but as a reusable KV-state construction problem. In its strictest form, a shared prefix or document is prefetched once, its per-layer key/value tensors are serialized with compatibility metadata, and later consumers load that artifact and begin decoding without rerunning the prefix forward pass. Recent work also applies the label more broadly to shared-prefill modules, stage-asymmetric pruning, and visibility policies that preserve KV-cache integrity while reducing prompt-side computation. Taken together, these papers suggest that KV-only prefill is best understood as a systems and modeling paradigm for eliminating redundant prompt processing, especially when many agents, models, or sessions repeatedly read the same context (Zhang, 11 Jun 2026, Woo et al., 12 Feb 2026, Oh et al., 7 May 2026).

1. Definition and conceptual scope

In the canonical formulation, the waste being targeted is simple: multiple agents read the same document, each reruns prefill over identical tokens, and each reconstructs an identical KV cache. The proposal in "Can I Buy Your KV Cache?" is to compute that cache once, save it as an artifact, and let later consumers load it instead of recomputing it. The paper calls this mechanism KV-only prefill and frames it as a first-class reusable artifact for long-context inference (Zhang, 11 Jun 2026).

The same label appears elsewhere with a wider meaning. In PrefillShare, the prefill stage is centralized into a frozen shared module whose cache is consumed by multiple task-specific decode modules in a disaggregated multi-LLM system; the shared prefix is computed once and reused across heterogeneous decoders (Woo et al., 12 Feb 2026). In POP, pruned layers are skipped during prefill, but independent KV projections are still materialized so that the later full-depth decode stage sees a complete cache (He et al., 3 Feb 2026). In SPEED, non-anchor prompt tokens are materialized only in lower layers, while decode-phase tokens remain full-depth; the method changes the visibility set of prompt KV rather than compressing or synthesizing upper-layer states (Oh et al., 7 May 2026). This suggests that the unifying idea is not a single implementation, but a common objective: reduce prompt-side work while keeping the downstream decode path compatible with the model’s cache semantics.

A persistent boundary condition runs through the literature. Some methods are token-exact shared-prefix reuse, whereas others are approximate selective recomputation, transfer-aware compression, or phase-asymmetric cache construction. The distinction matters because the strongest claims about exact equivalence apply only to the pure shared-prefix case, not to fragment fusion, lossy compression, or query-conditioned partial repair (Zhang, 11 Jun 2026, Wang et al., 31 Jan 2026, Sanovar et al., 8 Jun 2026).

2. Exact shared-prefix reuse

The strict shared-prefix mechanism begins with a document $c=x_{1:L}$. A publisher runs a normal forward pass to compute its cache $\mathcal{K}(c)$, then serializes the per-layer key/value tensors together with metadata such as $(\text{model id}, \text{dtype}, L)$. A consumer later deserializes that artifact, verifies that the running model and dtype match, reconstructs the cache object, and starts decoding from continuation tokens rather than recomputing the document prefix. Two implementation details are necessary for equivalence: the continuation position ids must start at $L$, and the attention mask must span the full cached prefix plus the continuation; otherwise positional encodings and attention context diverge (Zhang, 11 Jun 2026).

The exactness claim is unusually strong. The paper reports a 24/24 greedy-token match between loaded-cache continuation and from-scratch prefill plus decode, and also reports logits-level agreement in which the next-token argmax is identical, with only a small maximum absolute logit difference of approximately $0.02$, attributed to floating-point and kernel nondeterminism rather than semantic mismatch. The proof sketch is causal: the KV entries for positions $1,\dots,L$ depend only on the prefix tokens, so they are identical whether the prefix was computed alone or as part of a longer sequence; when the loaded cache is paired with the correct positional indices and full mask, the continuation sees the same keys and values at the same relative positions (Zhang, 11 Jun 2026).

The same paper also makes the main limitation explicit. Exactness holds in the prefix-only case. If independently prefetched fragments are fused, cross-attention between fragments is absent, so the result is no longer token-exact. This is the point at which KV-only prefill ceases to be exact cache reuse and becomes an approximation problem (Zhang, 11 Jun 2026).

3. Cost model and economic logic

The basic amortization model is compact. Let $C_{\text{prefill}}$ be the cost to compute the cache once and $C_{\text{load}}$ the cost of reusing it. Recomputing from scratch for $N$ reads costs

$$\text{total}_{\text{scratch}} = N\,C_{\text{prefill}}, \qquad \text{per-call}_{\text{scratch}} = C_{\text{prefill}}.$$

KV reuse instead costs

$\mathcal{K}(c)$0

The break-even reuse count is

$\mathcal{K}(c)$1

When loading is much cheaper than prefill, $\mathcal{K}(c)$2 is essentially $\mathcal{K}(c)$3, so the second read already benefits (Zhang, 11 Jun 2026).

The same paper ties this directly to attention complexity. Dense projections scale roughly linearly with sequence length $\mathcal{K}(c)$4, while self-attention contributes a term scaling as $\mathcal{K}(c)$5. The compute gap therefore widens with context length. On Qwen3-4B in fp16 on an Apple M1 Pro/MPS setup, measured reuse-versus-prefill speedups are $\mathcal{K}(c)$6 at 255 tokens, $\mathcal{K}(c)$7 at 485, $\mathcal{K}(c)$8 at 945, $\mathcal{K}(c)$9 at 1888, and $(\text{model id}, \text{dtype}, L)$0 at 3774. At 3774 tokens, the fp16 KV artifact is about 557 MB (Zhang, 11 Jun 2026).

The economic argument follows from those measurements. Serving one hot 3774-token document to 80 million agents costs about \$(\text{model id}, \text{dtype}, L)$130.4k</strong> under host-side reuse at the measured compute ratio, and about <strong>\$150.96k under a production cache-read tariff of $(\text{model id}, \text{dtype}, L)$2 input price. The paper therefore proposes an agent-native prefill CDN in which a publisher precomputes a document’s KV once, a provider hosts it, and agents pay per load. The tariff passes a $(\text{model id}, \text{dtype}, L)$3 discount to users while remaining inside the measured $(\text{model id}, \text{dtype}, L)$4–$(\text{model id}, \text{dtype}, L)$5 compute envelope, so the spread between physical reuse cost and charged price becomes provider margin (Zhang, 11 Jun 2026).

A central negative result is that shipping the cache to each consumer is economically unattractive. The paper argues that KV is nearly incompressible losslessly, so egress dominates: for the 3774-token fp16 example, transfer at (\$(\text{model id}, \text{dtype}, L)$60.049** per load, exceeding the roughly <strong>\$0.019 input-priced prefill it is supposed to replace. Provider-side hosting removes that egress term entirely and turns the KV into an internal reusable asset rather than a network payload (Zhang, 11 Jun 2026).

4. Representative formulations inside the model and serving stack

The literature contains several distinct realizations of the same prompt-side asymmetry.

Formulation	Core mechanism	Reported result
Direct shared-prefix reuse (Zhang, 11 Jun 2026)	serialize and later load document KV	$(\text{model id}, \text{dtype}, L)$7–$(\text{model id}, \text{dtype}, L)$8 cheaper than prefill
PrefillShare (Woo et al., 12 Feb 2026)	frozen shared prefill + fine-tuned decode modules	up to $(\text{model id}, \text{dtype}, L)$9 lower p95 latency
POP (He et al., 3 Feb 2026)	skip deep residual computation during prefill, but still project KV	up to $L$0 prefill speedup
SwiftKV (Qiao et al., 2024)	generate later-layer KV from an earlier hidden state	25–50% prefill compute reduction
SPEED (Oh et al., 7 May 2026)	shallow prompt KV, full-depth decode KV	33% TTFT and 22% TPOT improvement

PrefillShare is the clearest multi-model generalization. It factorizes a decoder-only Transformer into a frozen base prefill module $L$1 and specialized decode modules $L$2, with cache-conditioned fine-tuning so that heterogeneous decoders can consume the shared base cache. The memory analysis changes from $L$3 to $L$4, and the vLLM-based prototype uses a client-facing proxy, a prefill pool, a decode pool, and prefix-locality-aware routing from User ID to prefill worker. Accuracy matches Full-FT very closely, typically within about 1%, and serving results show up to 3.9× lower p95 latency and 3.6× higher throughput on ReAct, and up to 4.5× lower p95 latency and 3.9× higher throughput on Reflexion (Woo et al., 12 Feb 2026).

POP and SwiftKV both reduce prompt-side layer work while keeping decode compatible with the full model. POP prunes the last one-third of layers during prefill, bypassing their heavy Attention and FFN blocks but still generating KV through independent projections

$L$5

then hands the first decode token to the full model through a boundary handling strategy. The extra projection cost is reported as <5% of the full layer cost, and the method reaches up to 1.37× prefill speedup with minimal performance loss (He et al., 3 Feb 2026). SwiftKV instead runs only the first $L$6 layers normally and forms later-layer KV caches from an earlier hidden state,

$L$7

with lightweight distillation of the affected $L$8 parameters. It reports 25–50% prefill compute reduction, up to 2× higher aggregate throughput, and identifies 50% prefill reduction as a practical sweet spot (Qiao et al., 2024).

SPEED makes the asymmetry even sharper by changing decode-time visibility rather than synthesizing or compressing upper-layer prompt KV. With cutoff $L$9, non-anchor prefill tokens are visible only in layers $0.02$0, while decode tokens remain full-depth. Using a minimal BoS anchor, IT-SPEED-24+BoS reaches 51.2 average score against 51.4 for Full-IT, while at 128K context improving TTFT by 33%, TPOT by 22%, and reducing active KV memory by 25.0% (Oh et al., 7 May 2026).

5. Selective recomputation and KV-free relatives

Not all techniques described alongside KV-only prefill are pure cache loading. In long-context RAG, retrieved chunks are often reordered or combined in ways that break the shared-prefix assumption, so exact cache reuse gives way to partial recomputation.

ProphetKV is explicit on this point. It is a training-free, plug-and-play RAG method that assembles precomputed chunk-wise KV caches in position-independent fashion, then uses a dual-stage, user-query-driven recomputation pipeline to spend budget on tokens that matter for the query rather than on globally salient but query-irrelevant tokens. The paper reports 96%–101% of full-prefill accuracy with only a 20% recomputation ratio, and gains of 8.8%–24.9% on RULER and 18.6%–50.9% on LongBench over CacheBlend, EPIC, and KVShare (Wang et al., 31 Jan 2026).

InfoFlow KV frames the same repair problem as an information-flow problem. Chunks are prefetched independently, global RoPE positions are reconstructed at inference, and prompt-to-context attention norms

$0.02$1

select the top-$0.02$2 tokens whose KV states are recomputed under the full global causal layout. The method reports up to 3.49× speedup over the single-GPU full-prefill baseline, and under sequence parallelism reduces TTFT from 707.8 ms to 427.6 ms at 16K and from 2350.1 ms to 914.0 ms at 32K (Teng et al., 5 Mar 2026).

SIFT moves further away from traditional KV reuse. It stores no KV data at all, only two compact bit vectors per $0.02$3 that mark high-attention local tiles and cross-document key pages. Because these metadata are reported as up to 24,000× smaller than KV tensors, the method avoids the storage-to-GPU bottleneck that can make offline KV loading slower than recomputation on modern GPUs. SIFT improves TTFT by up to 1.71× while holding accuracy within 1% of full recompute (Sanovar et al., 8 Jun 2026).

These methods clarify a common misconception. Selective repair, sparse recomputation, or attention-location indexing can make prefill closer to KV-only reuse in practical RAG settings, but they are not token-exact shared-prefix loading. Their contribution is different: they recover enough cross-document or query-conditioned attention structure to avoid full prefill under a constrained compute budget (Wang et al., 31 Jan 2026, Teng et al., 5 Mar 2026, Sanovar et al., 8 Jun 2026).

6. Systems constraints, misconceptions, and open problems

A recurring systems lesson is that KV reuse is not only a modeling problem; it is also an I/O, transfer, and memory-placement problem. Cake makes this explicit for prefix caching. Pure KV-only loading from storage is often latency-suboptimal because loading latency scales linearly with cache size, whereas compute and I/O can be overlapped. Cake therefore uses bidirectional scheduling in which one worker computes from the front of the prompt and another loads cached KV from the back, meeting in the middle; across hardware and storage conditions it achieves an average 2.6× reduction in TTFT relative to compute-only and I/O-only methods (Jin et al., 2024).

In prefll-decode disaggregation, the same issue reappears as network payload. SpectrumKV argues that KV transfer should be treated as a precision-allocation problem rather than a binary keep-or-drop decision. Tokens are assigned FP16, INT8, or INT4 according to importance and a normalized budget, with a deployment-time probe to determine whether the model is INT4-tolerant. At a 50% normalized KV budget on WikiText-2, the perplexity changes are +1.97%, -0.06%, and -0.44% for Qwen2.5-7B, Mistral-7B, and Gemma-2-9B, compared with much larger degradations for PDTrim; end-to-end transfer-path timing reports 50–62% TTFT reductions at $0.02$4 (Pengju, 7 Jun 2026).

A different way to remove transfer is to avoid it altogether. Kairos, in disaggregated serving, opportunistically deflects the prefill phase to decode nodes when decode-side slack exists, so the KV cache is built directly where decode will happen. On a 2P2D A100 cluster, the paper reports that prefill execution itself accounts for only 2–23% of P95 TTFT, with queueing and KV transfer dominating; proactive deflection reduces P95 TTFT by up to 81% and raises SLO attainment by up to 79% (Arun et al., 2 Jul 2026).

The open problems in exact KV-only prefill remain specific and substantial. Full-precision artifacts are large; naive compression breaks exactness, and an int8 experiment in "Can I Buy Your KV Cache?" halves artifact size but preserves only 16/32 greedy tokens on a 692-token example. Lossless KV compression is therefore still open. Exactness also fails outside the prefix-only case, so fragment fusion remains unresolved. A real cross-party deployment further needs a payment and control layer for pricing, provenance, invalidation when models change, and fallback behavior on cache miss (Zhang, 11 Jun 2026).

The topic therefore sits at the intersection of inference semantics, systems architecture, and market design. Exact shared-prefix reuse establishes that a precomputed KV cache can be a token-exact, economically meaningful artifact. The broader literature shows how that idea can be adapted when prefixes are shared across models, when prompt tokens need only shallow visibility, when prefill layers can be skipped but KV integrity preserved, or when exact reuse is impossible and selective recomputation becomes necessary. The main unresolved question is no longer whether KV-only prefill can work, but under what deployment constraints it remains exact, economical, and operationally composable (Zhang, 11 Jun 2026, Woo et al., 12 Feb 2026, Oh et al., 7 May 2026).