Can I Buy Your KV Cache?

Abstract: Right now, across the world, AI agents are repeating the same absurd act: to read one document, they each recompute it from scratch. Every agent re-runs prefill, the most compute-intensive step a large model takes, over identical text, only to rebuild a key-value (KV) cache identical to the one the agent before it just built. The same answer, computed a million times. We make a proposal that is almost offensively simple: compute it once. Let a publisher precompute a document's KV cache, and let every other agent buy the right to load it and skip prefill. It works, and it is token-exact: loading a precomputed KV and continuing matches prefilling from scratch (24/24 greedy tokens, and at the logits level), with no accuracy cost. On Qwen3-4B, reuse is 9-50x cheaper in compute than prefill, and the gap widens with length (prefill's attention scales with L^2), so a single reuse already pays it back. Then the part that matters: where the KV lives. Shipping it fails, because KV is nearly incompressible, so per-load egress costs more than the prefill it saves. Hosting it provider-side, exactly as production prompt-caching works, removes egress entirely. The size of the prize is set by our measured compute saving: serving one hot 3774-token document to 80M agents costs ~$1.5M to re-prefill but only ~$0.03M of reuse compute (49.7x less). The 0.1x cache-read tariff APIs charge passes a 10x discount to users while sitting inside this measured envelope, so the 10x is a floor that the measured ~50x compute saving clears, and the gap to the physical ~50x is provider margin: millions of dollars per popular document. We frame the resulting agent-native prefill CDN and leave lossless KV compression and a cross-party payment layer as the open problems.

• The paper introduces a method for reusing prefill KV caches in LLMs to drastically reduce computational cost and latency.
• It formalizes artifact production and consumption processes that guarantee deterministic, token-exact outcomes during generation.
• Empirical results using Qwen3-4B show up to a 50× speedup and immediate cost benefits when reusing caches for repeated inputs.

Summary and Analysis of "Can I Buy Your KV Cache?" (2606.13361)

Problem Statement and Motivation

The paper interrogates the computational inefficiency inherent in current AI agent workflows involving LLMs, with a particular focus on the costly prefill phase. Prefilling, which computes key-value (KV) caches for all tokens in an input context, constitutes both a latency bottleneck and the dominant compute expense. This becomes acutely wasteful in scenarios where thousands or millions of independent agents repeatedly ingest the same content (e.g., knowledge-base articles, regulatory filings, product manuals) and redundantly compute identical KV caches. The author postulates a straightforward, economically motivated solution: precompute the KV cache once, publish it as a portable artifact, and allow downstream consumers (agents) to load and reuse it, paying a fraction of the original computation cost.

Technical Methodology

The paper formally defines two operations: (i) artifact production (by the publisher), which involves running a forward pass over the input context and serializing the resultant per-layer KV tensors alongside model-specific metadata; (ii) artifact consumption (by the agent), which entails loading the artifact, reconstructing the model's cache object in memory, and starting token generation from the cached prefix. Two critical points guarantee token-exactness: (a) correct assignment of position IDs for continuations, and (b) proper attention masking covering all sequence positions. The correctness argument hinges on the causal nature of transformer attention—prefix KV entries, whether prefetched independently or together with continuations, remain intact and thus enable deterministic, reproducible generation.

Numerical Results and Cost Characterization

Empirical measurements using Qwen3-4B (36 layers, fp16, Apple M1 Pro/MPS) reveal substantial computational gains from prefix KV reuse:

• For context lengths ranging from 255 to 3774 tokens, the compute cost of reuse (resident KV, no disk I/O) achieves $8.6$–$49.7\times$ speedup over fresh prefill, with the ratio expanding super-linearly as context length increases due to the quadratic scaling in prefill's attention cost.
• Token-exactness is maintained under greedy decoding ($24/24$ token match), with logit-level divergence attributable only to floating-point non-determinism, not semantic variation.
• The break-even reuse count is essentially $N^\star \approx 1$, meaning that reuse begins yielding net savings immediately upon a second consumer.

  Figure 1: Amortized per-call compute cost versus reuse count $N$: KV-reuse becomes advantageous almost immediately, with per-call cost falling rapidly toward the KV-load floor.

These results demonstrate that the economic incentive for KV artifact publication and provider-side caching (as envisioned in a prefill CDN model) is robust and practically viable, particularly for long contexts and popular documents.

Artifact Size, Compression, and Exactness Trade-offs

Artifact size poses the main challenge to widespread KV distribution: for Qwen3-4B, a 3.8K-token context yields $\sim$557 MB in fp16. Uniform int8 quantization halves this footprint, but breaks token-exact equivalence (greedy token match drops to $16/32$), illustrating the trade-off between size reduction and inference reproducibility. The paper suggests that selective or mixed-precision schemes (e.g., SpectrumKV (Pengju, 7 Jun 2026), SIFT (Sanovar et al., 8 Jun 2026)) are necessary for practical lossless compression without compromising correctness.

Deployment Architecture and Economics

The market-oriented framing pivots on artifact hosting location:

• Shipping KV artifacts directly to consumers incurs prohibitive egress costs; artifact sizes are nearly incompressible, and lossless distribution is economically unattractive at scale unless accuracy requirements are relaxed.
• Hosting provider-side (in-system, datacenter memory) enables efficient caching (akin to prompt-caching in production APIs): consumers are billed a cache-read rate per load, which can be set substantially below the price of context prefill while maintaining a sizeable provider margin (e.g., a $10\times$ user-facing discount against measured $\sim$50\times$$compute savings).</li> <li>Aggregate savings are immense: serving a single$$49.7\times$0-token document to $49.7\times$1M agents costs $49.7\times$2\$49.7\times$3\sim$49.7\times$40.03M for reuse—a $49.7\times$5 reduction, with further discounting available to users.

Relation to Prior Work

The artifact-publishing approach complements and builds upon recent advances in:

• Chunk fusion and KV reuse: CacheBlend (Yao et al., 2024), CacheCraft (Agarwal et al., 5 Feb 2025), CacheClip (2510.10129) address the harder case of merging independently prefetched fragments (with selective recomputation to restore correct attention), accepting controlled accuracy degradation for position-independent reuse.
• KV persistence and transfer: MatKV (Shin et al., 20 Dec 2025), InstInfer (Pan et al., 2024), NetKV (Ojewale, 2 Jun 2026), ConServe (Ding et al., 1 Jun 2026) demonstrate that KV can be materialized, stored, and moved across infrastructure, underpinning the "publishable artifact" concept.
• Compression techniques: SIFT (Sanovar et al., 8 Jun 2026), SpectrumKV (Pengju, 7 Jun 2026) explore lossy/lossless KV reduction to minimize storage and transfer costs, crucial to prefill CDN viability.
• The paper distinguishes itself by focusing exclusively on the "exact prefix reuse" case—where correctness is trivially guaranteed—and by articulating the economics, deployment structure, and break-even points.

Limitations

• The current study is restricted to prefix-only reuse: cross-fragment fusion is not addressed (though prior works handle this via selective recomputation).
• Artifacts are model-bound; exact reuse is undefined across models or precision mismatches, limiting deployment to standardized-model domains.
• All measurements are conducted on a single device and model; further validation across server-grade GPUs and larger models is recommended.
• Lossless distribution demands more sophisticated compression; naive quantization forfeits reproducibility.

Implications and Future Directions

The practical implications are far-reaching:

• Prefill KV caching at scale is immediately economically advantageous for long and frequently accessed content, allowing providers to drastically reduce compute expense and enabling consumer-side discounts.
• The concept can be extended to multimodal (video, audio, etc.) domains where prefill is even more computationally prohibitive.
• Open problems include cross-device/model validation; systems and economics design for cache provenance, billing, invalidation, and fallback; integration of smart compression schemes for lossless, scalable artifact hosting.

Theoretically, the KV artifact as a published, reusable object opens new avenues in agent economics, content-layer standardization, and infrastructure design for LLM serving ecosystems.

Conclusion

Treating the prefill KV cache as a reusable, publishable artifact for exact prefix reuse in LLMs yields strong numerical economies (up to $49.7\times$6 compute reduction), immediate amortization, and deterministic correctness in outputs. These findings motivate deployment architectures such as agent-native prefill CDNs, wherein content is cached as computation and served to millions of consumers under economically grounded pricing schemes. Lossless compression, fusion strategies for non-prefix fragments, and robust cross-party payment protocols remain as open directions for future research and development.