PersistentKV: Decode Scheduling & Storage Insights

• PersistentKV is a term defining both page-aware decode scheduling for long-context LLM serving and persistent key-value storage architectures, bridging computation and durability.
• In LLM serving, PersistentKV enhances cache management by reducing redundant KV memory traffic and optimizing grouped-query attention on commodity GPUs.
• In storage systems, PersistentKV underpins durable key-value stores that blend memory-like latency with persistence, replication, and efficient recovery mechanisms.

Searching arXiv for "PersistentKV" and closely related papers to ground the article in current literature. PersistentKV is a term with multiple technical uses across systems research. In its most specific contemporary usage, it denotes a page-aware decode scheduling system for long-context LLM serving on commodity GPUs, introduced as a native block-table decode attention engine and page-aware scheduling study for grouped-query attention (GQA) (Ahmed, 25 Jun 2026). In storage literature, closely related usage refers more broadly to persistent key-value stores that combine durability, indexing, and replication on non-volatile media (Mikrou et al., 2020), while adjacent LLM-serving work uses the term in a looser sense to describe persistent or reusable KV-cache management across requests, workflow steps, or memory tiers (Zou et al., 20 Jan 2026). The resulting landscape is not unified by a single abstraction: in one line of work, “KV” denotes durable key–value objects; in another, it denotes transformer attention key/value cache state.

1. Terminology and semantic scope

The strongest paper-specific meaning of PersistentKV is the 2026 system titled “PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs”, which argues that during autoregressive decode, performance is often limited not by dense matrix multiplication but by how KV-cache pages are moved, indexed, partitioned, and scheduled (Ahmed, 25 Jun 2026). In that setting, PersistentKV is explicitly not presented as a universally faster attention kernel; it is presented as a page-aware decode scheduling system whose benefits arise in particular low-active, long-context regimes.

A second usage appears in the storage-systems literature, where “PersistentKV” functions as a category label for persistent key-value stores. In “Power and Performance Analysis of Persistent Key-Value Stores”, the term refers not to a single named system but to persistent KVS designs that store data on flash or SSD media and must balance CPU overhead, I/O overhead, tail latency, and energy efficiency (Mikrou et al., 2020). Related papers on persistent memory, RDMA replication, and NVM-native engines occupy the same conceptual space, even when they use different names such as MCAS, Rowan-KV, or FlintKV (Waddington et al., 2021).

A third, newer usage arises in LLM serving, where “persistent KV” often means long-lived KV cache state retained across requests or across execution phases rather than recomputed from scratch. ContiguousKV, PBKV, PiKV, and CONF-KV all fall into this family: each treats KV tensors as a managed resource subject to offloading, eviction, prediction, routing, compression, or scheduling policies (Zou et al., 20 Jan 2026). This suggests that the term has bifurcated into two major research traditions: durable key–value storage on non-volatile hardware, and persistent attention-state management in transformer inference.

2. Persistent key-value storage lineage

In storage systems, the central problem is to combine memory-like latency with durability and, in distributed settings, with replication. MCAS is a representative design: it is presented as a network-attached, persistent-memory key-value store that combines the low-latency access of memory, the durability of storage, and the scalability of RDMA networking, while adding a near-memory compute model through Active Data Objects (ADO) (Waddington et al., 2021). Its base organization is a sharded key-value store in which each shard owns persistent memory regions, CPU resources, one network endpoint, and a pool of key-value data. The paper emphasizes a $1:N$ shard-to-pool mapping, meaning a pool is serviced by only one shard, simplifying concurrency and enabling a lock-free design.

MCAS stores the hash table index, keys, and values in persistent memory, using a hopscotch hash table as the core index. Because keys and values are variable-sized, it uses a heap allocator, but the allocator for key-value data is maintained in volatile DRAM to support fast insert/delete rates (Waddington et al., 2021). Persistent memory regions may be backed by devdax partitions or fsdax files, and the implementation described in the paper runs Intel Optane PMM in App-Direct mode. For networking, MCAS uses libfabric, with RDMA verbs as the transport, yielding kernel-bypass access to a persistent pool over RDMA.

Durability in such systems typically has two components: memory persistence and replication. MCAS states this explicitly and evaluates both 2-way replication and 3-way replication. For 12 shards and 6 client threads, the reported throughput reaches 3.92M updates/sec for 2-way replication and 3.07M updates/sec for 3-way replication, corresponding to 20% degradation and 37% degradation, respectively; the conclusion further notes that with three replicas the system can support over 6M updates/sec in a three-node cluster of 2-socket servers (Waddington et al., 2021). The same paper reports single-thread update RTT of 7.14 μs, 140K updates/sec from a single client thread, and 99.77% of 1-thread writes under 10 μs, underscoring the emphasis on memory-like latency rather than storage-like latency.

Rowan-KV addresses a narrower but fundamental issue in replicated PM-backed KVSs: the interaction between RDMA and device-level write amplification (DLWA) on persistent memory (Wang et al., 2022). The paper argues that RPC-based replication is too slow for PM KVSs, while one-sided RDMA WRITE can generate many small PM write streams that overflow PM write-combining buffers and trigger read-modify-write behavior. Rowan therefore provides an RDMA abstraction that aggregates concurrent remote writes and lands them to PM in a sequential and one-sided manner. Rowan-KV, the KVS built on top of this abstraction, is a log-structured PM KVS whose architecture uses per-thread primary logs in PM, one backup log in PM, and DRAM-resident per-shard hash indexes. Under write-intensive workloads, compared with PM KVSs using RPC and RDMA WRITE for replication, Rowan-KV improves throughput by 1.22X and 1.39X, lowers median PUT latency by 1.77X and 2.11X, and keeps DLWA below 1.032× versus 1.54× for the RDMA WRITE baseline (Wang et al., 2022).

FlintKV extends this lineage toward production database semantics. It is an NVM-optimized skiplist-based storage engine that natively supports atomic multi-key write batches, snapshot-consistent iteration, and durable linearizability (Egorov et al., 2 Jul 2026). The design splits state into a volatile DRAM skiplist index and a persistent NVM linked list of nodes, and it uses flat combining with multi-versioning to reduce synchronization overhead while preserving correctness. The paper reports up to 73% improvement over PMemRocksDB and up to 75% over ListDB, while also supporting the richer API required by modern databases (Egorov et al., 2 Jul 2026). This places FlintKV at the point where “persistent KV” ceases to mean only Put/Get/Delete durability and instead encompasses snapshots, iterators, and batched atomicity.

3. Distributed persistence, object identity, and recovery

A conceptually broader precursor is the peer-to-peer middleware framework for resilient persistent programming, which is described as a global persistent object infrastructure in which an object can be referenced by abstract identity, replicated across a P2P overlay, re-instantiated after failure, and resolved transparently whether it is local, remote, or stored on disk (Dearle et al., 2010). The paper explicitly frames this as extending orthogonal persistence beyond a single host and notes that the design is “very close in spirit” to a PersistentKV-style abstraction, except that the system manages typed objects, object graphs, version histories, code, and policy rather than only key/value pairs.

Its architecture decomposes persistence into multiple distributed services: Object Directory, Version Directory, Data Store, Code Store, Name Directory, and Policy Store (Dearle et al., 2010). Resolution of an abstract reference follows a concrete workflow: extract the GUID, search the Object Directory, consult the Version Directory if no live instance exists, fetch serialized state from the Data Store, fetch the class from the Code Store, re-instantiate the object, publish the new instance, and return the resulting reference. The separation between GUID as object identity and PID as serialized version identity is central. This is a richer semantic model than a plain key-value store, but it shares several persistent-storage ideas: logical identity, versioned state, replication, and transparent recovery.

The paper is careful to distinguish atomic replication of object graphs from full application-level transactional semantics. It guarantees that individual replica storage is atomic and that the overall effect of commit is atomic, but it deliberately does not hard-code ACID transactions into the middleware (Dearle et al., 2010). Instead, it exposes a policy framework in which application developers can provide custom resilience policies. A plausible implication is that this work occupies an ancestral position relative to later durable-KV systems: it treats persistence not merely as media durability but as a network-wide recovery substrate.

4. PersistentKV as page-aware decode scheduling for LLM serving

In the LLM-serving literature, the paper actually titled PersistentKV defines the term in a distinct way. The system is built for grouped-query attention (GQA) and keeps K/V in paged layout,

$K,V[physical\_page, H_{kv}, page\_size, d],$

with each request carrying a block table mapping logical pages to physical pages (Ahmed, 25 Jun 2026). The target problem is decode, where each active sequence contributes only one new query token per step but must attend over the entire prefix:

$$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$

The paper emphasizes that in this regime the main bottlenecks are KV-cache memory traffic, page-table traversal, launch overhead, and the difficulty of exposing enough parallel work to keep the GPU occupied.

PersistentKV’s decode engine maps work by KV-head group and sequence split, assigning CTAs over

$$(\text{request}, \text{KV head}, \text{sequence split}).$$

For the model shape used in the paper, $H_q = 32$, $H_{kv} = 8$, $G = 4$, $d = 128$, FP16, and page size $P=16$ (Ahmed, 25 Jun 2026). This mapping is designed so that when a CTA loads a K/V tile for one KV head, that tile can be reused across the grouped query heads, reducing redundant K/V traffic. The engine processes 32-token tiles and maintains the exact online-softmax state $(m,\ell,\mathrm{acc})$ in FP32 directly over native page tables.

A central mechanism is sequence splitting, introduced because one CTA per request/KV-head is insufficient when only a few long sequences are active. PersistentKV splits the sequence dimension into $K,V[physical\_page, H_{kv}, page\_size, d],$0 disjoint ranges and computes split-local softmax states

$K,V[physical\_page, H_{kv}, page\_size, d],$1

which are then merged exactly:

$K,V[physical\_page, H_{kv}, page\_size, d],$2

This provides more parallel work without sacrificing exactness (Ahmed, 25 Jun 2026). The paper also stresses row-local bounds based on the true per-row sequence length $K,V[physical\_page, H_{kv}, page\_size, d],$3, so empty splits can return early with neutral states $K,V[physical\_page, H_{kv}, page\_size, d],$4.

The distinctive scheduling contribution is the workqueue path. Rather than launching a rectangular grid over $K,V[physical\_page, H_{kv}, page\_size, d],$5, the system emits a compact task list containing only non-empty row–KV-head–split intervals. This is motivated by the observation that exact length bucketing can create many launches for ragged batches, while coarse bucketing can create large amounts of empty work (Ahmed, 25 Jun 2026). The resulting policy is explicitly adaptive:

$K,V[physical\_page, H_{kv}, page\_size, d],$6

The calibrated split counts are fixed on calibration traces before held-out evaluation: split 32 for the B1 bucket route, split 20 for B8 bimodal workqueue, split 24 for B8 uniform workqueue, and split 28 for B8 Zipf-like workqueue.

The reported gains are deliberately conditional rather than universal. On a single-request native-paged benchmark, FlashInfer remains fastest, and PersistentKV is competitive with vLLM but does not beat FlashInfer in that isolated test (Ahmed, 25 Jun 2026). The main serving improvements appear in end-to-end traces: the calibrated policy yields 1.399× wall throughput on a Bucketed B1 trace, 1.265× on Bimodal B8, 1.082× on Uniform B8, and 1.063× on Zipf B8, while on the Bimodal B4 boundary case it selects FlashInfer and obtains 1.000×, thereby avoiding regression. The paper also reports a strong structural reduction in scheduler overhead on the held-out bimodal B8 trace: 16.00 launches/step for exact length buckets versus 2.00 launches/step for the compact workqueue, 4.06 MB/step of merge-state traffic versus 2.54 MB/step, and 8.00 merge launches versus 1.00 (Ahmed, 25 Jun 2026).

5. Persistent KV-cache management in neighboring LLM systems

Several adjacent systems extend the “persistent KV” idea beyond decode scheduling into cache retention, offloading, routing, and prediction. ContiguousKV addresses the Re-Prefill Phase for workloads with shared prefixes, where the system must load precomputed prefix KV from slower storage, compute request-specific suffix KV, and run attention over the reused prefix (Zou et al., 20 Jan 2026). It introduces ContiguousChunk, a unified granularity of consecutive tokens with chunk count

$K,V[physical\_page, H_{kv}, page\_size, d],$7

using $K,V[physical\_page, H_{kv}, page\_size, d],$8 in the experiments. The paper argues that aligning pruning, storage, transfer, and eviction at this same granularity eliminates read amplification. It further uses intra-period and inter-period asynchronous prefetching, based on the observation that important ContiguousChunk indices are highly similar across layers and across adjacent periods. On Qwen2.5 series models, ContiguousKV reports a 3.85× speedup in the Re-Prefill phase over IMPRESS at a 5% KV budget, and an average I/O reduction of about 16.33× (Zou et al., 20 Jan 2026).

PBKV studies a different persistence unit: workflow-level cache reuse in dynamic multi-agent systems (Zheng et al., 7 May 2026). Here the challenge is that the sequence of invoked agents is not fixed in advance, so the system cannot know which cache entries will be needed later. PBKV therefore predicts the next $K,V[physical\_page, H_{kv}, page\_size, d],$9 agent invocations by combining a global call graph estimated from offline traces, a semantic signal from the current prefill, and a prefix-history summary. Its multi-step reuse score for a cache node $$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$0 is

$$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$1

where the survival term is

$$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$2

Eviction is hierarchical: retired cache is reclaimed first, and active cache is evicted only if necessary. Prefetching is conservative and uses only free or retired GPU space together with otherwise-idle PCIe bandwidth. Across three workflow benchmarks, the paper reports up to 1.85× latency speedup over LRU on dynamic workflows and up to 1.26× over KVFlow on a static workflow (Zheng et al., 7 May 2026).

PiKV addresses persistent KV cache management for Mixture-of-Experts (MoE) models by making cache retention expert-aware, distributed, and query-aware (Liu et al., 2 Aug 2025). Its architecture comprises distributed expert-sharded KV storage, PiKVRouting, PiKVCompression, and PiKVScheduling. The storage layer uses a shard-mapping function over token and expert identifiers, a circular buffer per shard, and a memory model in which

$$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$3

with optimal shard size

$$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$4

PiKV’s cache-aware routing biases expert selection away from high-miss experts, while scheduling assigns each entry a utility score derived from attention, recency, frequency, or learned features. The paper reports throughput improvements of about 1.8× to 3.2×, memory reductions of about 2.8× to 3.5×, and latency improvements of about 1.9× to 2.4× across Switch-Transformer-1.6T, GLaM-1.2T, PaLM-540B, and Mixtral-8x7B, with accuracy staying within roughly 1.5% of full-KV baselines (Liu et al., 2 Aug 2025).

CONF-KV focuses on long-horizon decoding and uses the model’s next-token distribution to select the per-step cache budget (Li et al., 24 May 2026). Its confidence score is

$$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$5

and the cache budget switches between $$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$6 and $$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$7 depending on whether $$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$8 crosses a threshold $$O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.$$9. Tokens are then ranked by a composite of accumulated attention mass and recency, with a protected recent window. The system combines this policy with blockwise online-softmax attention, mixed FP16/INT8 storage, and an optional pyramidal per-layer budget. On Needle-in-a-Haystack up to 32K tokens, CONF-KV reports 91.4% retrieval accuracy versus 53.8% for sliding windows and 80.6% for H2O; on 75 VisualWebArena tasks it retains 95.3% of full-KV success at 2.8 times lower peak memory (Li et al., 24 May 2026).

6. Design tensions, misconceptions, and broader significance

One common misconception is that PersistentKV denotes a single stable systems abstraction. The literature instead shows at least two major meanings. In storage systems, PersistentKV-like work is about durability, replication, recovery, and increasingly richer APIs such as snapshots and atomic batches (Egorov et al., 2 Jul 2026). In LLM serving, it is about persistent KV-cache state and the management of that state under constraints imposed by GPU memory, host-device I/O, page tables, or workflow structure (Ahmed, 25 Jun 2026). The shared phrase “KV” obscures the fact that one branch concerns application-visible key–value objects while the other concerns internal attention tensors.

A second misconception is that persistence necessarily implies “keep everything forever.” The LLM-serving papers consistently reject that interpretation. PersistentKV in decode scheduling keeps native-paged K/V in the serving layout and changes how work is launched and merged rather than insisting on maximal retention (Ahmed, 25 Jun 2026). ContiguousKV persistently stores shared prefix KV across tiers but selectively reloads semantically important chunks (Zou et al., 20 Jan 2026). PBKV keeps workflow-prefix cache alive across agent steps but uses prediction to decide what should remain resident (Zheng et al., 7 May 2026). PiKV explicitly characterizes persistence as long-lived, reusable KV state under explicit memory/latency constraints, not indefinite retention (Liu et al., 2 Aug 2025).

A third recurring tension is between operational performance and recovery or correctness guarantees. The persistent-memory porting study of Redis and Memcached shows that a hybrid design, with key/value data in persistent memory and indexing structures in volatile memory, generally improves throughput and reduces latency at the cost of slower recovery and additional complexity (Choi et al., 2020). The paper concludes that the hybrid design nearly doubles operational throughput at the cost of increased time to recovery and formulates three design principles: prefer hybrid design when operational performance dominates, allocate persistent data in large chunks, and use full-featured persistent memory libraries. This suggests a durable systems lesson that also echoes in NVM-native engines such as FlintKV: the right persistence boundary is often not “persist every structure,” but rather “persist the structures whose recovery cost and semantics justify it.”

Across these lines of work, the broader significance of PersistentKV is that it marks a shift from treating KV state as ephemeral implementation detail to treating it as a first-class managed resource. In storage systems, that means durable, recoverable, and often replicated state on PM or NVM. In LLM serving, it means page-aware, workflow-aware, or uncertainty-aware management of K/V tensors whose layout and lifetime decisively affect throughput, latency, memory footprint, and quality (Ahmed, 25 Jun 2026). This suggests that the term’s enduring importance lies less in a single named artifact than in a recurring systems insight: KV state, once made long-lived, forces co-design across storage layout, scheduling, consistency, and recovery.