---
title: 'PersistentKV: Decode Scheduling & Storage Insights'
url: https://www.emergentmind.com/topics/persistentkv
type: topic
---

# PersistentKV: Decode Scheduling & Storage Insights

Searching arXiv for "PersistentKV" and closely related papers to ground the article in current literature.
PersistentKV is a term with multiple technical uses across systems research. In its most specific contemporary usage, it denotes a **page-aware decode scheduling system** for **long-context LLM serving on commodity GPUs**, introduced as a **native block-table decode attention engine and page-aware scheduling study for grouped-query attention (GQA)** [2606.26666]. In storage literature, closely related usage refers more broadly to **persistent key-value stores** that combine durability, indexing, and replication on non-volatile media [2008.13402], while adjacent LLM-serving work uses the term in a looser sense to describe **persistent or reusable KV-cache management** across requests, workflow steps, or memory tiers [2601.13631]. The resulting landscape is not unified by a single abstraction: in one line of work, “KV” denotes durable key–value objects; in another, it denotes transformer attention key/value cache state.

## 1. Terminology and semantic scope

The strongest paper-specific meaning of PersistentKV is the 2026 system titled **“PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs”**, which argues that during autoregressive decode, performance is often limited not by dense matrix multiplication but by **how KV-cache pages are moved, indexed, partitioned, and scheduled** [2606.26666]. In that setting, PersistentKV is explicitly **not** presented as a universally faster attention kernel; it is presented as a **page-aware decode scheduling system** whose benefits arise in particular low-active, long-context regimes.

A second usage appears in the storage-systems literature, where “PersistentKV” functions as a category label for **persistent key-value stores**. In **“Power and Performance Analysis of Persistent Key-Value Stores”**, the term refers not to a single named system but to persistent KVS designs that store data on flash or SSD media and must balance CPU overhead, I/O overhead, tail latency, and energy efficiency [2008.13402]. Related papers on persistent memory, RDMA replication, and NVM-native engines occupy the same conceptual space, even when they use different names such as MCAS, Rowan-KV, or FlintKV [2104.06225].

A third, newer usage arises in LLM serving, where “persistent KV” often means **long-lived KV cache state** retained across requests or across execution phases rather than recomputed from scratch. ContiguousKV, PBKV, PiKV, and CONF-KV all fall into this family: each treats KV tensors as a managed resource subject to offloading, eviction, prediction, routing, compression, or scheduling policies [2601.13631]. This suggests that the term has bifurcated into two major research traditions: durable key–value storage on non-volatile hardware, and persistent attention-state management in transformer inference.

## 2. Persistent key-value storage lineage

In storage systems, the central problem is to combine **memory-like latency** with **durability** and, in distributed settings, with **replication**. MCAS is a representative design: it is presented as a **network-attached, persistent-memory key-value store** that combines the **low-latency access of memory**, the **durability of storage**, and the **scalability of RDMA networking**, while adding a near-memory compute model through **Active Data Objects (ADO)** [2104.06225]. Its base organization is a **sharded key-value store** in which each shard owns persistent memory regions, CPU resources, one network endpoint, and a pool of key-value data. The paper emphasizes a **$1:N$ shard-to-pool mapping**, meaning a pool is serviced by only one shard, simplifying concurrency and enabling a lock-free design.

MCAS stores the **hash table index**, **keys**, and **values** in persistent memory, using a **hopscotch hash table** as the core index. Because keys and values are variable-sized, it uses a heap allocator, but the allocator for key-value data is maintained in **volatile DRAM** to support fast insert/delete rates [2104.06225]. Persistent memory regions may be backed by **devdax** partitions or **fsdax** files, and the implementation described in the paper runs Intel Optane PMM in **App-Direct mode**. For networking, MCAS uses **libfabric**, with RDMA verbs as the transport, yielding kernel-bypass access to a persistent pool over RDMA.

Durability in such systems typically has two components: **memory persistence** and **replication**. MCAS states this explicitly and evaluates both **2-way replication** and **3-way replication**. For 12 shards and 6 client threads, the reported throughput reaches **3.92M updates/sec** for 2-way replication and **3.07M updates/sec** for 3-way replication, corresponding to **20% degradation** and **37% degradation**, respectively; the conclusion further notes that with three replicas the system can support **over 6M updates/sec** in a three-node cluster of 2-socket servers [2104.06225]. The same paper reports **single-thread update RTT of 7.14 μs**, **140K updates/sec** from a single client thread, and **99.77% of 1-thread writes under 10 μs**, underscoring the emphasis on memory-like latency rather than storage-like latency.

Rowan-KV addresses a narrower but fundamental issue in replicated PM-backed KVSs: the interaction between **RDMA** and **device-level write amplification (DLWA)** on persistent memory [2209.09459]. The paper argues that RPC-based replication is too slow for PM KVSs, while one-sided **RDMA WRITE** can generate many small PM write streams that overflow PM write-combining buffers and trigger read-modify-write behavior. Rowan therefore provides an RDMA abstraction that aggregates concurrent remote writes and lands them to PM in a **sequential** and **one-sided** manner. Rowan-KV, the KVS built on top of this abstraction, is a **log-structured PM KVS** whose architecture uses per-thread primary logs in PM, one backup log in PM, and DRAM-resident per-shard hash indexes. Under write-intensive workloads, compared with PM KVSs using RPC and RDMA WRITE for replication, Rowan-KV improves throughput by **1.22X** and **1.39X**, lowers median PUT latency by **1.77X** and **2.11X**, and keeps DLWA below **1.032×** versus **1.54×** for the RDMA WRITE baseline [2209.09459].

FlintKV extends this lineage toward production database semantics. It is an **NVM-optimized skiplist-based storage engine** that natively supports **atomic multi-key write batches**, **snapshot-consistent iteration**, and **durable linearizability** [2607.02401]. The design splits state into a volatile DRAM skiplist index and a persistent NVM linked list of nodes, and it uses **flat combining** with multi-versioning to reduce synchronization overhead while preserving correctness. The paper reports up to **73% improvement over PMemRocksDB** and up to **75% over ListDB**, while also supporting the richer API required by modern databases [2607.02401]. This places FlintKV at the point where “persistent KV” ceases to mean only `Put`/`Get`/`Delete` durability and instead encompasses snapshots, iterators, and batched atomicity.

## 3. Distributed persistence, object identity, and recovery

A conceptually broader precursor is the peer-to-peer middleware framework for resilient persistent programming, which is described as a **global persistent object infrastructure** in which an object can be referenced by abstract identity, replicated across a P2P overlay, re-instantiated after failure, and resolved transparently whether it is local, remote, or stored on disk [1006.3724]. The paper explicitly frames this as extending **orthogonal persistence** beyond a single host and notes that the design is “very close in spirit” to a PersistentKV-style abstraction, except that the system manages **typed objects, object graphs, version histories, code, and policy** rather than only key/value pairs.

Its architecture decomposes persistence into multiple distributed services: **Object Directory**, **Version Directory**, **Data Store**, **Code Store**, **Name Directory**, and **Policy Store** [1006.3724]. Resolution of an abstract reference follows a concrete workflow: extract the GUID, search the Object Directory, consult the Version Directory if no live instance exists, fetch serialized state from the Data Store, fetch the class from the Code Store, re-instantiate the object, publish the new instance, and return the resulting reference. The separation between **GUID** as object identity and **PID** as serialized version identity is central. This is a richer semantic model than a plain key-value store, but it shares several persistent-storage ideas: logical identity, versioned state, replication, and transparent recovery.

The paper is careful to distinguish **atomic replication of object graphs** from full application-level transactional semantics. It guarantees that individual replica storage is atomic and that the overall effect of `commit` is atomic, but it deliberately does not hard-code ACID transactions into the middleware [1006.3724]. Instead, it exposes a policy framework in which application developers can provide custom resilience policies. A plausible implication is that this work occupies an ancestral position relative to later durable-KV systems: it treats persistence not merely as media durability but as a network-wide recovery substrate.

## 4. PersistentKV as page-aware decode scheduling for LLM serving

In the LLM-serving literature, the paper actually titled PersistentKV defines the term in a distinct way. The system is built for **grouped-query attention (GQA)** and keeps K/V in paged layout,
$$
K,V[physical\_page, H_{kv}, page\_size, d],
$$
with each request carrying a **block table** mapping logical pages to physical pages [2606.26666]. The target problem is decode, where each active sequence contributes only one new query token per step but must attend over the entire prefix:
$$
O_{i,h} = \sum_{j=1}^{N_i} \frac{\exp(q_{i,h}^{\top} k_{i,j}/\sqrt{d})} {\sum_{r=1}^{N_i}\exp(q_{i,h}^{\top} k_{i,r}/\sqrt{d})} v_{i,j}.
$$
The paper emphasizes that in this regime the main bottlenecks are **KV-cache memory traffic**, **page-table traversal**, **launch overhead**, and the difficulty of exposing enough parallel work to keep the GPU occupied.

PersistentKV’s decode engine maps work by **KV-head group** and **sequence split**, assigning CTAs over
$$
(\text{request}, \text{KV head}, \text{sequence split}).
$$
For the model shape used in the paper, \(H_q = 32\), \(H_{kv} = 8\), \(G = 4\), \(d = 128\), FP16, and page size \(P=16\) [2606.26666]. This mapping is designed so that when a CTA loads a K/V tile for one KV head, that tile can be **reused across the grouped query heads**, reducing redundant K/V traffic. The engine processes 32-token tiles and maintains the exact online-softmax state \((m,\ell,\mathrm{acc})\) in FP32 directly over native page tables.

A central mechanism is **sequence splitting**, introduced because one CTA per request/KV-head is insufficient when only a few long sequences are active. PersistentKV splits the sequence dimension into \(S\) disjoint ranges and computes split-local softmax states
$$
m_s=\max_j x_j,\quad \ell_s=\sum_j \exp(x_j-m_s),\quad a_s=\sum_j \exp(x_j-m_s)v_j,
$$
which are then merged exactly:
$$
m=\max_s m_s,\quad \ell=\sum_s \exp(m_s-m)\ell_s,\quad a=\sum_s \exp(m_s-m)a_s,\quad O=a/\ell.
$$
This provides more parallel work without sacrificing exactness [2606.26666]. The paper also stresses **row-local bounds** based on the true per-row sequence length \(N_i^\star = seq\_lens[i]\), so empty splits can return early with neutral states \((m=-\infty,\ell=0,a=0)\).

The distinctive scheduling contribution is the **workqueue** path. Rather than launching a rectangular grid over \((B,H_{kv},S)\), the system emits a compact task list containing only **non-empty** row–KV-head–split intervals. This is motivated by the observation that exact length bucketing can create many launches for ragged batches, while coarse bucketing can create large amounts of empty work [2606.26666]. The resulting policy is explicitly adaptive:
$$
route(B,N_{\max}) = \begin{cases} \text{PersistentKV length bucket}, & B \le 1,\ \text{PersistentKV workqueue}, & B \ge 8 \wedge N_{\max}\ge 16\mathrm{K},\ \text{FlashInfer}, & \text{otherwise}. \end{cases}
$$
The calibrated split counts are fixed on calibration traces before held-out evaluation: split 32 for the B1 bucket route, split 20 for B8 bimodal workqueue, split 24 for B8 uniform workqueue, and split 28 for B8 Zipf-like workqueue.

The reported gains are deliberately conditional rather than universal. On a single-request native-paged benchmark, **FlashInfer remains fastest**, and PersistentKV is competitive with vLLM but does not beat FlashInfer in that isolated test [2606.26666]. The main serving improvements appear in end-to-end traces: the calibrated policy yields **1.399×** wall throughput on a **Bucketed B1** trace, **1.265×** on **Bimodal B8**, **1.082×** on **Uniform B8**, and **1.063×** on **Zipf B8**, while on the **Bimodal B4** boundary case it selects FlashInfer and obtains **1.000×**, thereby avoiding regression. The paper also reports a strong structural reduction in scheduler overhead on the held-out bimodal B8 trace: **16.00 launches/step** for exact length buckets versus **2.00 launches/step** for the compact workqueue, **4.06 MB/step** of merge-state traffic versus **2.54 MB/step**, and **8.00** merge launches versus **1.00** [2606.26666].

## 5. Persistent KV-cache management in neighboring LLM systems

Several adjacent systems extend the “persistent KV” idea beyond decode scheduling into cache retention, offloading, routing, and prediction. ContiguousKV addresses the **Re-Prefill Phase** for workloads with shared prefixes, where the system must load precomputed prefix KV from slower storage, compute request-specific suffix KV, and run attention over the reused prefix [2601.13631]. It introduces **ContiguousChunk**, a unified granularity of **consecutive tokens** with chunk count
$$
m = \lceil n / c \rceil,
$$
using **\(c=16\)** in the experiments. The paper argues that aligning pruning, storage, transfer, and eviction at this same granularity eliminates read amplification. It further uses **intra-period** and **inter-period asynchronous prefetching**, based on the observation that important ContiguousChunk indices are highly similar across layers and across adjacent periods. On Qwen2.5 series models, ContiguousKV reports a **3.85× speedup** in the Re-Prefill phase over IMPRESS at a 5% KV budget, and an average I/O reduction of about **16.33×** [2601.13631].

PBKV studies a different persistence unit: **workflow-level cache reuse** in dynamic multi-agent systems [2605.06472]. Here the challenge is that the sequence of invoked agents is not fixed in advance, so the system cannot know which cache entries will be needed later. PBKV therefore predicts the next \(K\) agent invocations by combining a **global call graph** estimated from offline traces, a semantic signal from the current prefill, and a prefix-history summary. Its multi-step reuse score for a cache node \(c\) is
$$
\mathrm{Score}(c)
=\sum_{k=1}^{K} \gamma^{k-1} \sum_{w \in W(c)} s_w^{(k)} \cdot A_w(c) \cdot P_w^{(k)},
$$
where the survival term is
$$
s_w^{(k)}=\prod_{j=1}^{k-1} \bigl(1-p_{w,\langle\text{END}\rangle}^{(j)}\bigr).
$$
Eviction is hierarchical: retired cache is reclaimed first, and active cache is evicted only if necessary. Prefetching is conservative and uses only free or retired GPU space together with otherwise-idle PCIe bandwidth. Across three workflow benchmarks, the paper reports up to **1.85× latency speedup** over LRU on dynamic workflows and up to **1.26×** over KVFlow on a static workflow [2605.06472].

PiKV addresses persistent KV cache management for **Mixture-of-Experts (MoE)** models by making cache retention **expert-aware**, **distributed**, and **query-aware** [2508.06526]. Its architecture comprises **distributed expert-sharded KV storage**, **PiKVRouting**, **PiKVCompression**, and **PiKVScheduling**. The storage layer uses a shard-mapping function over token and expert identifiers, a circular buffer per shard, and a memory model in which
$$
\mathcal{M}_{\text{total}} = \frac{2d}{\rho}\left(\frac{L}{GS} + KS\right),
$$
with optimal shard size
$$
S^* = \sqrt{\frac{L}{KG}}.
$$
PiKV’s cache-aware routing biases expert selection away from high-miss experts, while scheduling assigns each entry a utility score derived from attention, recency, frequency, or learned features. The paper reports throughput improvements of about **1.8× to 3.2×**, memory reductions of about **2.8× to 3.5×**, and latency improvements of about **1.9× to 2.4×** across Switch-Transformer-1.6T, GLaM-1.2T, PaLM-540B, and Mixtral-8x7B, with accuracy staying within roughly **1.5%** of full-KV baselines [2508.06526].

CONF-KV focuses on long-horizon decoding and uses the model’s **next-token distribution** to select the per-step cache budget [2605.24786]. Its confidence score is
$$
c = w_H(1-\hat{H}) + w_m \sigma(m) + w_p p_{(1)}, \quad (w_H,w_m,w_p)=(0.4,0.3,0.3),
$$
and the cache budget switches between \(N_{\mathrm{high}}\) and \(N_{\mathrm{low}}\) depending on whether \(c\) crosses a threshold \(\tau\). Tokens are then ranked by a composite of accumulated attention mass and recency, with a protected recent window. The system combines this policy with blockwise online-softmax attention, mixed **FP16/INT8** storage, and an optional pyramidal per-layer budget. On Needle-in-a-Haystack up to 32K tokens, CONF-KV reports **91.4%** retrieval accuracy versus **53.8%** for sliding windows and **80.6%** for H2O; on 75 VisualWebArena tasks it retains **95.3%** of full-KV success at **2.8 times lower peak memory** [2605.24786].

## 6. Design tensions, misconceptions, and broader significance

One common misconception is that PersistentKV denotes a single stable systems abstraction. The literature instead shows at least two major meanings. In storage systems, PersistentKV-like work is about **durability**, **replication**, **recovery**, and increasingly richer APIs such as snapshots and atomic batches [2607.02401]. In LLM serving, it is about **persistent KV-cache state** and the management of that state under constraints imposed by GPU memory, host-device I/O, page tables, or workflow structure [2606.26666]. The shared phrase “KV” obscures the fact that one branch concerns application-visible key–value objects while the other concerns internal attention tensors.

A second misconception is that persistence necessarily implies “keep everything forever.” The LLM-serving papers consistently reject that interpretation. PersistentKV in decode scheduling keeps native-paged K/V in the serving layout and changes how work is launched and merged rather than insisting on maximal retention [2606.26666]. ContiguousKV persistently stores shared prefix KV across tiers but selectively reloads semantically important chunks [2601.13631]. PBKV keeps workflow-prefix cache alive across agent steps but uses prediction to decide what should remain resident [2605.06472]. PiKV explicitly characterizes persistence as **long-lived, reusable KV state under explicit memory/latency constraints**, not indefinite retention [2508.06526].

A third recurring tension is between **operational performance** and **recovery or correctness guarantees**. The persistent-memory porting study of Redis and Memcached shows that a **hybrid** design, with key/value data in persistent memory and indexing structures in volatile memory, generally improves throughput and reduces latency at the cost of slower recovery and additional complexity [2002.02017]. The paper concludes that the hybrid design nearly doubles operational throughput at the cost of increased time to recovery and formulates three design principles: prefer hybrid design when operational performance dominates, allocate persistent data in large chunks, and use full-featured persistent memory libraries. This suggests a durable systems lesson that also echoes in NVM-native engines such as FlintKV: the right persistence boundary is often not “persist every structure,” but rather “persist the structures whose recovery cost and semantics justify it.”

Across these lines of work, the broader significance of PersistentKV is that it marks a shift from treating KV state as ephemeral implementation detail to treating it as a **first-class managed resource**. In storage systems, that means durable, recoverable, and often replicated state on PM or NVM. In LLM serving, it means page-aware, workflow-aware, or uncertainty-aware management of K/V tensors whose layout and lifetime decisively affect throughput, latency, memory footprint, and quality [2606.26666]. This suggests that the term’s enduring importance lies less in a single named artifact than in a recurring systems insight: KV state, once made long-lived, forces co-design across storage layout, scheduling, consistency, and recovery.

Source: https://www.emergentmind.com/topics/persistentkv