vToken: Token-Level Virtualization for Reclaimable KV Caches

Abstract: LLM serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.

• The paper introduces a token-level virtualization layer that separates logical KV-token eviction from physical block reclamation, enabling asynchronous compaction without modifying attention kernels or CUDA Graph execution.
• vToken reduces retained KV blocks by 27.2–72.3%, improves SLA-constrained throughput by up to 37.3%, and extends feasible concurrency by up to 2× under constrained GPU memory budgets.
• The system preserves correctness and near-native overhead through token tables, lazy relocation, CUDA-event synchronization, and conservative prefix-cache handling, while reducing per-policy integration work from more than 500 lines to under 50.

vToken addresses a granularity mismatch in LLM serving systems: token-level KV eviction policies make per-token retention decisions, while PagedAttention-style runtimes allocate and reclaim memory only at block granularity. The result is intra-block fragmentation—partially live blocks that cannot be returned to the allocator—which traps a substantial fraction of GPU memory even after aggressive logical eviction. vToken inserts a token-level virtualization layer between eviction policies and the block-managed substrate, decoupling logical token liveness from physical block placement and enabling asynchronous physical reclamation without modifying attention kernels or CUDA Graph execution.

The granularity mismatch

PagedAttention organizes the KV cache into fixed-size blocks (typically 16 tokens) and maps logical to physical blocks via a block table, reducing external fragmentation and enabling prefix sharing. Token-level eviction algorithms such as H2O, StreamingLLM, Scissorhands, and FastGen instead decide which individual tokens to retain. When composed naively on a block-based runtime, a block containing both evicted and retained tokens cannot be released.

The paper formalizes this as an intra-block waste ratio $F = 1 - \frac{1}{N}\sum_i u_i$, where $u_b = n_b/S$ is block utilization. Preliminary experiments running H2O-style eviction on vLLM with Llama-3.1-8B at 16K context show that most allocated blocks sit at or below 50% utilization across policies and workloads, with waste ratios of 40–60%. This is a strong claim about where the loss lies: the paper argues the dominant cost is not the eviction policy but the runtime's inability to convert token-level liveness into reusable physical capacity. The mismatch also imposes an integration burden—a direct H2O integration into vLLM requires over 500 lines of code spread across core modules, per policy.

The paper considers and rejects two alternatives. A fully token-granular allocator would weaken contiguous-access assumptions of PagedAttention kernels and multiply metadata; shrinking block size merely redistributes the fragmentation–bandwidth tradeoff (smaller blocks increase metadata overhead and degrade bandwidth efficiency in copy and offload paths).

Design

vToken defines a contract: above the boundary, policies emit evict_token calls on logical token identities and never reason about blocks; below it, the runtime maintains a per-sequence token table mapping each logical token ID to a (block ID, offset) pair plus a liveness bit, refreshes attention slot mappings, and decides when reclamation is profitable. Three challenges shape the design: dual-view consistency between token liveness and block occupancy, safe reclamation during live decoding, and policy-neutral amortized cost.

The token table is the central metadata structure, exposing three interfaces: evict_token, sync_new_tokens, and apply_moves. Marking a token dead updates metadata only; no memory moves until the reclamation backend acts. Metadata cost is $O(L)$ per sequence—one entry per token, independent of layers and heads—with a canonical CPU table and a GPU-resident lookup cache for steady-state slot translation.

The reclamation backend implements lazy compaction in four stages. Reclamation eligibility uses incrementally maintained per-block liveness counts. Headroom-aware admission triggers only when global waste exceeds $\theta_F$ (default 0.25), enough low-utilization blocks exist, and free-block count approaches a low watermark; bounded evacuation headroom within the same KV budget provides destination blocks, preventing relocation from borrowing unbounded memory under pressure. Relocation planning is request-local and admitted only when projected block reduction is strictly positive. Stage-aware asynchronous copy launches KV copies after forward returns, on a dedicated stream, overlapping them with sampling and scheduling phases that do not read the KV cache; a stream-level CUDA event guards subsequent attention launches, replacing any host synchronization.

Scheduler integration modifies slot mapping to consult the token table ($slot = \text{block\_id} \times S + \text{offset}$ from the table rather than direct computation) and adds three hooks: scheduler-side headroom reservation, worker-side reclamation launch, and pre-attention synchronization. CUDA Graph compatibility is preserved by keeping captured buffers stable and updating only the mutable slot-mapping buffer before replay—no graph bypass or recapture.

Correctness rests on four stated invariants: token conservation (I1), unique logical-to-physical mapping (I2), pre-attention visibility via CUDA-event dependencies (I3), and layout-aware planning from consistent snapshots with strict profitability (I4). I1 and I2 are checked online during evaluation; I3 is structural and unit-tested; I4 is enforced by planner gates.

Evaluation

The prototype runs on vLLM v0.18.0 with PyTorch 2.10.0 on a single H100, evaluated on Mistral-7B and Llama-3.1-8B (plus a Qwen2.5-14B capacity check) over ShareGPT and LongBench, using H2O, Random, and Scissorhands policies. All comparisons are paired against Naive-Evict—the same eviction decisions on the same runtime with reclamation disabled—so gains isolate the virtualization layer itself. An indirection-only ablation changes throughput and p95 latency by less than 1% relative to native vLLM, confirming the steady-state lookup path is nearly free.

Memory efficiency: vToken increases average memory utilization by 21.88% (Llama-3.1-8B) and 21.67% (Mistral-7B) and reduces retained blocks per request by 27.2–72.3%. Because decoding is memory-bound, reclaimed blocks translate directly into admission capacity.

SLA-constrained throughput: with the SLA threshold anchored at 1.05× Naive-Evict p95 latency, vToken improves selected feasible throughput by 9.9–37.3% on Mistral-7B and 18.9% on average on Llama-3.1-8B, reducing p95 latency by up to 27.5%. Gains are largest under Random eviction (up to 37.0%), where retained tokens scatter across blocks, and strongest for Scissorhands (33.3–103.7% throughput improvement), whose persistent-attention retention leaves live tokens highly dispersed. Notably, gains are realized at the maximum-throughput feasible concurrency point, not merely at larger batch sizes.

Capacity frontier: under controlled KV budgets, vToken extends verified feasible concurrency from C=5 to C=8 at gpu_mem_util=0.35 (a 60% extension) and from C=11 to C=22 at 0.50 (a 2× extension), with boundary throughput degrading gracefully (180.3 vs. peak 203.2 tokens/s). A Qwen2.5-14B check confirms a 2× frontier extension, indicating the benefit is not model-specific. The paper is explicit that native full-KV serving remains preferable when feasible; vToken matters only once full retention exhausts the block pool.

Overhead: planner-side opportunity checking dominates CPU overhead; async copies incur no explicit synchronization waits, though copy/decode contention can appear under heavier pressure—an honest concession that overlap is not entirely free.

Sensitivity: block size 32 yields best throughput; larger blocks trade capacity against performance and do not substitute for reclamation. Higher eviction ratios convert directly into more reclaimed capacity, while results are comparatively insensitive to the fragmentation threshold, suggesting fine-grained trigger tuning is unnecessary.

Prefix caching and correctness: shared-prefix blocks are conservatively excluded from relocation. With sharing degrees 2–8, all shared candidates are skipped, consistency checks pass, and retained blocks still drop 28.6–42.2% via private-suffix reclamation; throughput improves up to 146.5% relative to Naive-Evict with prefix caching enabled. Relocation correctness is validated by hashing ordered retained token IDs and K/V contents before and after every relocation event—all checks pass—and paired generation comparison shows mean ROUGE-L F1 difference of −0.0016 with 93.1% of pairs differing by at most 0.01, indicating no measurable quality drift attributable to the runtime layer.

Limitations and open questions

Several constraints bound the current results. The prototype targets single-node, single-GPU decoding; tensor-parallel operation is argued to be straightforward (shard-local block IDs) but is not implemented or measured. Shared-prefix blocks are excluded from relocation rather than handled via copy-on-write, leaving potential reclamation capacity unused when sharing is heavy. The implementation supports one runtime KV cache group, and multi-group runtimes would need group-specific placement arrays. Overhead measurements show copy/decode contention under stress, and planner-side checks dominate CPU cost—batching these checks remains an engineering target. The evaluation also depends on backend-exposed signals such as attention scores for policy inputs, and the block-size sweep is restricted to sizes ≥16 by the vLLM version used. An open question the paper leaves explicit: whether a less conservative shared-prefix mechanism (copy-on-write gated on expected reclamation benefit) preserves the correctness guarantees while recovering the excluded capacity.

Conclusion

vToken contributes a token-level virtualization abstraction that makes token-level KV eviction physically effective in block-managed serving runtimes. By separating the decision that a token is dead from the act of reclaiming its memory, it converts logically evicted tokens into reusable physical capacity: 27.2–72.3% fewer retained blocks, up to 1.37× SLA-constrained throughput, up to 2× feasible concurrency under constrained budgets, and per-policy integration reduced from 500+ to under 50 lines of code—all without kernel modifications or CUDA Graph disruption. The abstraction is positioned as a pressure-activated complement to native full-KV serving rather than a replacement, and its portability hinges on three runtime hooks—block allocation/release tracking, pre-attention slot-map updates, and dependency-tracked asynchronous copy—that other PagedAttention-style systems could instantiate.