PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

Abstract: Autoregressive LLM serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication. Modern paged-attention systems reduce KV-cache fragmentation and mature kernels such as FlashInfer provide highly optimized native-paged decode attention. However, the best single-kernel implementation is not always the best serving schedule: low-active long-context decode can under-utilize commodity GPUs, while mixed sequence lengths introduce a tension between many exact-length launches and coarse padded batches. We present PersistentKV, a native block-table decode attention engine and page-aware scheduling study for grouped-query attention (GQA). PersistentKV maps work by KV-head group, is designed to reuse K,V tiles across grouped query heads, supports native page tables, and adds a compact workqueue schedule that executes only non-empty row-KV-head-sequence-split tasks. On an RTX 3060 with FP16, page size 16, Hq=32, Hkv=8, d=128, and identical correctness tolerance against FlashInfer, a calibrated adaptive policy selects FlashInfer for small active batches, PersistentKV sequence splitting for B1 long-context steps, and PersistentKV workqueue scheduling for B8 long-context steps. With thresholds and split counts fixed on calibration traces, one held-out trace seed improves synchronized wall throughput by 1.063-1.265x on B8 bimodal, uniform, and Zipf-like workloads and by 1.399x on a B1 bucketed trace. On the B4 bimodal boundary case, the policy avoids the PersistentKV regression by selecting FlashInfer. These results identify a concrete systems niche for adaptive page-aware decode scheduling and show that work assignment, not only attention math, is a decisive serving-system variable.

• The paper introduces a page-aware block-table decode scheduling strategy that maps CUDA thread arrays across KV-head groups to optimize low-active, long-context LLM inference.
• It employs sequence splitting and a compact workqueue scheduler to reduce launch overhead and improve throughput by up to 1.399× in select decoding regimes.
• The adaptive serving policy dynamically selects optimal kernel routes, maintaining numerical correctness while addressing heterogeneous sequence workloads.

PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

Background and Motivation

The serving landscape for autoregressive LLMs has undergone substantial changes, with the compute bottleneck shifting from dense matrix multiplication to key-value (KV) cache management during decode iterations. Prefill allows efficient matrix-multiplication kernels, but decode steps impose memory-bound workloads, especially under long-context and low-active regimes, where each sequence contributes only a single token. Modern paged-attention approaches—including FlashInfer—have matured to optimize native paged decode attention, yet uniform single-kernel strategies fail to adapt to diverse serving workloads characterized by varied sequence lengths and batch sizes. PersistentKV addresses this gap by introducing a page-aware, block-table decode scheduling strategy, targeting grouped-query attention (GQA) while directly operating on native paged layouts.

Methodological Advances

PersistentKV is engineered to optimize low-active, long-context inference by mapping CUDA thread arrays (CTAs) across KV-head groups and sequence splits, leveraging GQA ($H_q=32$, $H_{kv}=8$, $G=4$, $d=128$). KV pages are maintained in a block table, providing logical-to-physical mapping that fully exploits memory layout without repacking costs. Sequence splitting is implemented, enabling the decomposition of long-context single-sequence requests into $S$ splits, with exact online-softmax merging ensuring correctness. The merged approach allows for occupancy gains where native baselines underutilize GPU resources.

A compact workqueue scheduler is further introduced for ragged batch workloads, emitting only non-empty row-KV-head-split tasks. This design sidesteps the excessive launch fan-out associated with length bucketing, minimizing empty or inefficient split work—an explicit correction for serving traces with highly heterogeneous sequence lengths. Ablation studies confirm that row-local bounds, indexed Q/O, zero-tile early exit, and careful workqueue management collectively contribute to scaling throughput on commodity GPUs.

Calibrated Adaptive Serving Policy

The serving harness employs an explicit calibration/held-out protocol, selecting kernel routes based on request state and batch profile. The calibrated adaptive policy assigns:

• PersistentKV sequence splitting for B1 long-context steps,
• PersistentKV workqueue scheduling for B8 long-context steps,
• FlashInfer for small or boundary-case batches.

Thresholds and split counts are fixed from calibration traces and validated on held-out seeds. Correctness is rigorously maintained (max $|e| < 2 \cdot 10^{-3}$ and mean $|e| < 3 \cdot 10^{-4}$ versus FlashInfer), with identical page layouts and context lengths across comparisons.

Experimental Results

On an NVIDIA RTX 3060 (FP16, page size 16), PersistentKV demonstrates significant throughput gains in select serving regimes. Isolated native-paged single-request tests confirm that FlashInfer remains the fastest kernel, with PersistentKV competitive but not dominant. The core improvement manifests in actual serving traces:

• On B1 bucketed long-context decode, PersistentKV achieves a synchronized wall throughput improvement of $1.399\times$.
• For B8 bimodal, uniform, and Zipf-like traces, improvements range from $1.063\times$ to $1.265\times$.
• Scheduler-derived workload counters show launch fan-out reduction from 16.00 to 2.00 per decode step in B8 bimodal traces, with substantial reductions in merge-state traffic and launch overhead.
• In the boundary B4 case, the policy falls back to FlashInfer, avoiding regressions due to persistent merge overhead.

Sequence split calibration is critical: too few splits lead to underoccupancy, while too many increase merge overhead. Fixed split counts (e.g., 32 for B1, 20–28 for various B8 traces) yield wall-time improvements without introducing padding waste or excessive launch overhead.

Discussion and Implications

PersistentKV exposes the sensitivity of LLM decode serving to hardware-aware scheduling and memory layout. In low-active, long-context decode, it leverages sequence splitting for improved occupancy, whereas in mixed/bimodal batch regimes, the compact workqueue scheduler eliminates the pathologies of launch fan-out and padding. The adaptive route selection is necessary—the kernel is not universally superior; its strengths are workload-dependent.

The implications are that work assignment and scheduling are decisive variables in LLM serving, not just kernel efficiency. Serving systems should employ dynamic, trace-aware scheduling policies and optimize for occupancy, launch reduction, and memory locality. Practically, deploying PersistentKV or similar page-aware serving modules could yield substantial efficiency gains on commodity GPUs, which remain a significant deployment target outside datacenter-bound accelerators.

On the theoretical front, the separation of kernel execution and scheduling exposes a new axis for LLM inference optimization, inviting future work on adaptive split policies, online cost modeling, and exploitation of hardware features (e.g., Hopper-generation GPU primitives). PersistentKV’s methodology points toward dynamic kernel fusion, dequantization-aware scheduling, and further integration into production runtimes with real request traces.

Limitations and Threats to Validity

The experimental scope is deliberately controlled:

• Synthetic traces are used for isolating kernel-scheduler behaviors; production-level systems include additional serving stack complexities.
• Single GPU target: tuning is RTX 3060-specific; results should be replicated on datacenter-class hardware.
• Baseline versions are fixed and compatible with local environments; results are not assertions against all future releases.
• GQA ratio is fixed; a broader sweep is needed for generalization.
• No hardware performance counters are reported due to permission restrictions; all structural counters are scheduler-derived.
• Kernel-level harness: MLP layers, sampling, networking, and full-request latency distributions are not included.

These caveats highlight that the findings are kernel-scheduler centric and are best read as a reproducible methodology for future integration and benchmarking.

Conclusion

PersistentKV provides a principled kernel-scheduling approach for long-context LLM serving on commodity GPUs, demonstrating significant synchronized wall throughput gains in low-active and mixed-sequence regimes. The calibrated adaptive policy ensures selection of optimal kernel routes, avoiding regressions while maintaining numerical correctness. The work establishes page-aware scheduling and compact workqueue execution as key contributors to LLM serving efficiency, delineates concrete systems research problems, and lays the foundation for further developments in adaptive, trace-informed LLM inference systems.