VeriCache: Exact LLM Inference via KV Verification

• VeriCache is a large-language-model inference framework that converts lossy KV-cache compression into lossless outputs through a draft-and-verify process.
• It uses compressed KV caches for drafting and full KV caches off-GPU for verification, achieving up to 4X higher throughput than full-KV inference.
• The system supports long-context decoding and remote prefix caching by balancing GPU HBM and transfer constraints with asynchronous scheduling.

Searching arXiv for the primary VeriCache paper and closely related LLM KV-cache work. arxiv_search(query="VeriCache KV cache lossless LLM inference", max_results=5) arxiv_search(query="VeriCache KV cache lossless LLM inference", max_results=10) VeriCache is a large-language-model inference framework that converts lossy KV-cache compression into lossless inference by using the compressed KV cache only for drafting and then verifying drafted tokens against the full KV cache. It is presented as the first framework that ensures the same output as full-KV-cache decoding while largely preserving the high decoding throughput of a range of KV cache compression algorithms. The system targets two settings—long-context decoding and remote prefix caching—and is designed to keep the full KV cache out of GPU memory, swapping it in only when verification is due. Reported results show up to 4X higher throughput than full-KV inference while producing identical outputs under greedy decoding, aside from hardware nondeterminism (Yao et al., 17 May 2026).

1. Problem setting and motivation

VeriCache addresses the fact that the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In long-context inference, the KV cache grows linearly with context length, increasing both memory footprint and the per-token bandwidth cost of decoding. In prefix-caching scenarios, the bottleneck shifts from recomputation to transfer, because loading large stored KV caches from CPU memory, storage, or remote instances can dominate latency (Yao et al., 17 May 2026).

The framework is motivated by a limitation of existing KV-cache compression methods. Token dropping and KV quantization reduce memory or transfer cost, but they are inherently lossy because the compressed cache is not the same as the original cache. VeriCache explicitly frames this as a distribution-shift problem: even small next-token discrepancies accumulate over long generations. The paper uses the per-step divergence

$$\mathrm{KL}_t=\sum_{x_t} p_\text{full}(x_t\mid x_{<t}) \log \frac{p_\text{full}(x_t\mid x_{<t})}{p_\text{lossy}(x_t\mid x_{<t})},$$

and argues via the KL chain rule that small token-level bias compounds as decoding proceeds. The reported practical consequence is that outputs can remain superficially fluent while failing on syntax, structured output, exact argument matching, code generation, and tool calling (Yao et al., 17 May 2026).

This motivates a separation between acceleration and correctness. Rather than serving lossy compressed-cache outputs directly, VeriCache treats lossy compression as a fast proposal mechanism and restores exactness through verification. A plausible implication is that the framework repositions compression from a terminal approximation to an intermediate systems primitive.

2. Drafting with compressed KV and verifying with full KV

The core algorithm is a draft-and-verify procedure. Let $\text{KV}_\text{comp}$ denote the compressed KV cache and $\text{KV}_\text{full}$ the full KV cache. VeriCache first drafts $x$ tokens autoregressively using $\text{KV}_\text{comp}$. It then performs one parallel forward pass using $\text{KV}_\text{full}$ over those drafted positions, producing verifier predictions for the same positions plus one additional bonus position. The system accepts the longest prefix of drafted tokens that matches the verifier, replaces the first mismatch with the verifier token, and discards the remaining draft tokens. If all drafted tokens match, it accepts all $x$ tokens and also the bonus token (Yao et al., 17 May 2026).

This procedure is conceptually similar to speculative decoding, but the drafter and verifier use the same model weights. The only difference is whether attention is computed from compressed or full KV. That distinction matters because VeriCache is not approximating the model with a smaller auxiliary network; it is comparing two cache representations for the same base model. The result is that every emitted token is checked against the full-KV next-token prediction, so the final output is identical to full-KV greedy decoding, up to hardware nondeterminism (Yao et al., 17 May 2026).

The paper attributes much of the system’s efficiency to a long drafting horizon. Because the compressed-KV drafter tracks the full model more closely than a conventional small-model speculative drafter, many drafted tokens are often accepted before verification finds a mismatch. Reported acceptance lengths are often around 25–40 tokens per verification round, compared with 2–3 for typical small-model speculative drafters. This long horizon amortizes the cost of each full-KV swap (Yao et al., 17 May 2026).

3. Off-GPU full KV, overlap, and runtime scheduling

A naive implementation would keep both compressed and full KV caches in GPU HBM, but that would nullify the memory savings that make compression attractive. VeriCache instead keeps the compressed KV resident on GPU for drafting and keeps the full KV off GPU: in CPU memory for long-context decoding, or in storage or local remote GPU memory for prefix caching. The full KV is reloaded only when verification is due (Yao et al., 17 May 2026).

The runtime is built on top of vLLM and LMCache as a thin layer that manages scheduling, reservation of bandwidth and HBM, asynchronous full-KV reloads, and compressed-KV allocations. Its central systems observation is that drafting and verification stress different resources. Drafting with compressed KV is mostly HBM-bandwidth-bound, because tokens are generated one at a time and the GPU repeatedly reads model weights and compressed KV. Verification, by contrast, is interconnect-bound and compute-bound, because it requires loading the full KV over PCIe, CPU–GPU links, or the network and then running a parallel forward pass. The paper summarizes the iteration-time model as

$T_\text{iter}=\max(T_\text{gpu},T_\text{xfer}),$

with the scheduling objective being to keep both the GPU and the transfer path busy rather than serializing them (Yao et al., 17 May 2026).

To realize this overlap, VeriCache maintains two reservation rings over a lookahead window. The BW ring tracks interconnect reservation and ensures that transfers do not overload the PCIe or storage link. The HBM ring tracks GPU memory occupancy and ensures that full-KV reloads can coexist with resident compressed caches. On request arrival and after each verification, the runtime calls Admit(r), estimates when the next verification should occur, searches around the target draft horizon, reserves a future window if both bandwidth and HBM constraints are satisfied, and otherwise places the request back in a waiting queue. At each iteration, it kicks off reloads whose deadlines are approaching, drafts on compressed KV and verifies requests whose full KV has arrived, re-admits requests after verification, and slides the window forward (Yao et al., 17 May 2026).

This design makes the framework fundamentally different from a static swap-in/swap-out mechanism. The full KV is neither permanently resident nor loaded in lock-step with every decode step; instead, transfer and compute are explicitly staggered across requests.

4. Compressor interface, supported methods, and composition

VeriCache is designed to apply to a broad family of KV compression methods through a uniform compressor interface. The paper instantiates the framework for two compression families: token dropping and quantization. Token-dropping examples include Keyformer, H2O, Ada-KV, KVzip, FastKVzip, KVzap, SnapKV, PyramidKV, and DuoAttention. Quantization examples include KIVI, KVQuant, TurboQuant, CacheGen, and LLM.265. In evaluation, the concrete baselines include KVzip, KVZap, ExpectedAttention, SnapKV, KIVI, KVQuant, and RotateKV (Yao et al., 17 May 2026).

The interface is specified as follows:

$\text{KV}_\text{comp}$3

The runtime uses compress(...) to produce a compressed cache, decompress(...) to materialize full cache state when needed, and update(...) to support online token-dropping decisions during inference. The metadata fields dropped_indices and bit_scheme are sufficient for memory allocation and attention execution. The interface assumes that one compression mode is used at a time, that the bit scheme is uniform, that the compression method is fixed at deployment, and that quantization and token dropping are not mixed across concurrent requests (Yao et al., 17 May 2026).

VeriCache also composes with traditional speculative decoding. The paper lists Eagle, MTP, and n-gram speculators as examples of auxiliary drafters. In the composed design, VeriCache drafts outer tokens using compressed KV, a traditional speculative drafter proposes additional inner tokens, and VeriCache verifies the combined draft against the full KV cache. The paper gives the expected accepted draft length as

$$\gamma(x,c)\cdot x \cdot \big[1 + \gamma_e(d_e)\cdot(d_e - 1)\big],$$

where $\gamma(x,c)$ is VeriCache’s acceptance rate, $\text{KV}_\text{comp}$0 is its draft length, $\text{KV}_\text{comp}$1 is the auxiliary drafter’s acceptance, and $\text{KV}_\text{comp}$2 is the auxiliary drafter’s extra speculative length. This expresses a compositional split of labor: VeriCache reduces KV-related overhead, while the auxiliary drafter reduces compute per token (Yao et al., 17 May 2026).

5. Evaluation methodology and empirical behavior

The evaluation uses Mistral-24B, Qwen-32B, and Llama-70B. Hardware includes an NVIDIA RTX PRO 6000 96GB, 2× H100 NVL for Llama-70B, and a PCIe 5.0 x16 CPU–GPU link, with storage links at different bandwidths depending on whether the setup is local or remote. Two serving pipelines are reported. In the long-context pipeline, one serving instance operates with precomputed KV stored off-GPU in CPU memory or storage. In the remote-prefix pipeline, one local and four remote instances reuse KV caches across requests over slower links (Yao et al., 17 May 2026).

The quality metrics include KL divergence from full-KV outputs, function-call exact-match accuracy, prompt injection defense success, completion rate, and math answer accuracy. Efficiency is measured through end-to-end request latency and throughput in tokens per second. The central correctness claim is that VeriCache produces outputs identical to full-KV inference under greedy decoding, except for hardware nondeterminism. The reported KL divergence from full-KV is typically under 0.01 nats, which the paper attributes to nondeterminism rather than algorithmic divergence (Yao et al., 17 May 2026).

The headline performance result is up to 4X throughput improvement over full-KV inference. For long-context decoding, the reported gains are around 1.92×–2.73× over full-KV on representative settings, with examples such as 256 tok/s versus 102 tok/s on Llama-70B. For remote prefix caching, reported gains are around 1.33×–2.11× over full-KV. When composed with a traditional drafter, VeriCache reaches up to 4.26× speedup, and in some settings VeriCache alone reaches around 3.7× ideal speedup in the throughput model (Yao et al., 17 May 2026).

The paper further argues that lossy compression alone can be faster, but only at substantial quality cost. Compression baselines often preserve token-level similarity while failing on task-level correctness, especially on long outputs. VeriCache therefore occupies a distinct operating point: near-full-KV correctness with compression-like throughput. This suggests that the framework is best understood not as another compressor, but as a correctness-preserving execution layer over compressors.

6. Limitations, assumptions, and broader position in KV-cache research

VeriCache has several explicit limitations. It stores compressed KV on GPU and full KV off GPU, so total storage overhead is higher than in pure full-KV serving or pure compression-only serving. It uses a fixed draft horizon per workload rather than a per-request adaptive policy. The paper notes that current compressors are optimized for direct serving accuracy rather than for maximizing acceptance length under long draft horizons, which creates an objective mismatch. The compressor interface also assumes a single compression mode at a time, a uniform bit scheme, and no mixing of token dropping and quantization across concurrent requests (Yao et al., 17 May 2026).

Its performance is also hardware- and deployment-dependent. The design is most effective when HBM is the bottleneck during drafting, transfer bandwidth is the bottleneck during verification, and the runtime can overlap those two phases. This means the realized speedup depends on the balance among HBM capacity, PCIe or network bandwidth, and workload shape. The paper additionally notes that other forms of approximate KV reuse, such as non-prefix reuse, remain future work rather than solved cases (Yao et al., 17 May 2026).

Within the broader KV-cache literature, VeriCache is distinct from methods that expand effective KV capacity by changing memory allocation rather than correcting approximation. For example, MIRAGE reclaims model-parameter memory and repurposes it for KV cache, avoiding KV-cache swapping through parameter remapping in multi-tenant serving; its gains derive from capacity expansion and swap avoidance, not from verification of lossy drafts (Li et al., 15 Jul 2025). VeriCache instead starts from the premise that lossy compression is useful but unsafe as a final execution path, and it restores exactness through full-KV verification.

Taken together, these properties define VeriCache as a verification-centric systems framework for LLM serving. Its central contribution is not a new compressor, quantizer, or token-dropping heuristic, but a runtime architecture that turns lossy KV-cache approximations into exact greedy decoding by combining compressed drafting, off-GPU full-KV storage, asynchronous reload, and amortized verification (Yao et al., 17 May 2026).