---
title: 'KV-Passthrough: Direct KV State Management'
url: https://www.emergentmind.com/topics/kv-passthrough
type: topic
---

# KV-Passthrough: Direct KV State Management

KV-Passthrough denotes a family of mechanisms for moving, reusing, or preserving transformer key–value state without re-encoding or incurring the full cost of conventional cache handling. The term is not standardized across the literature. In "Unified KV Pooling to Accelerate Long-Context LLM Serving" [2606.14779], KV-passthrough is the SSD access fast path that bypasses the Linux kernel filesystem and directly accesses SSD-resident KV caches from user space via SPDK. In "Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse" [2606.23581], KV-Passthrough is the authors’ name for a training-free, position-invariant multimodal KV-cache reuse operator that combines exact RoPE re-rotation with a low-rank conditioning patch. Other papers describe closely related continuity, recurrence, retention, or bypass behaviors without using the term explicitly, which indicates that KV-Passthrough is better understood as a cross-cutting design pattern than as a single canonical algorithm [2604.12171][2605.12471][2606.03075][2605.13734][2505.24133].

## 1. Terminological scope and main usages

The literature uses KV-Passthrough in at least two distinct technical senses. The first is an **I/O-path sense**: KV state already offloaded to storage is retrieved through a direct user-space path rather than a filesystem-mediated file read. This is the meaning used in unified KV pooling, where KV-passthrough is the SSD-side mechanism that removes the filesystem bottleneck for SSD-backed KV retrieval while the KV orchestrator decides placement across host memory and SSD devices [2606.14779].

The second is a **reuse-operator sense**: cached KV is stored in a form that can be moved, re-instantiated, or reused under changed positional or contextual conditions without full recomputation. This is the meaning used by Kamera, where KV-Passthrough separates a chunk’s position from the conditioning it absorbed from antecedent context, then reconstructs reuse as exact relocation plus a low-rank conditioning restoration [2606.23581].

A broader survey of adjacent work shows that several papers instantiate passthrough-like behavior without naming it as such. PipeLive describes a live KV continuity path during pipeline-parallel reconfiguration through non-contiguous KV access, incremental KV patching, and atomic commit after convergence; the paper explicitly states that it does **not** use the term KV-Passthrough [2604.12171]. KV-Fold passes prior KV forward unchanged as prefix state across chunk boundaries in a recurrent left fold over chunks [2605.12471]. TGV-KV defines a text-prioritised retention policy that effectively lets text KV “pass through” aggressive vision-side eviction [2606.03075]. KVServe can effectively bypass compression when compression is non-beneficial, but does not introduce a dedicated passthrough primitive [2605.13734]. R-KV is a redundancy-aware retention policy rather than a passthrough method in the architectural sense [2505.24133].

This distribution of meanings suggests that KV-Passthrough is not a single settled term. A plausible implication is that current usage clusters around three technical objectives: eliminating mediation overhead in the KV data path, preserving KV identity across execution boundaries, and retaining useful KV under strict memory or communication budgets.

## 2. KV-Passthrough as filesystem-bypassing SSD access

In unified KV pooling, the need for KV-passthrough arises from profiling long-context serving. The paper reports high serving latency of approximately \(30.7\) s at \(128\)K context on the Qwen3-30B-A3B setup, exceeding the typical TTFT requirement of \(10\) s by more than \(3\times\). It attributes this to two compounding issues: retrieval is serialized through a narrow host-memory/SSD path, and SSD-based KV retrieval spends \(84\%\) of its time in the kernel filesystem rather than actual device access [2606.14779].

KV-passthrough changes the retrieval path from filesystem-mediated file read to direct LBA access from user space. In the conventional path, evicted KV caches go from GPU to host memory and then get written to SSD via the kernel filesystem, which maps them to LBAs; retrieval reverses that path as SSD \(\rightarrow\) filesystem \(\rightarrow\) host memory \(\rightarrow\) GPU. KV-passthrough removes the filesystem middle layer by using SPDK, a user-space NVMe framework that bypasses the kernel filesystem entirely. After the KV orchestrator assigns a KV shard to an SSD pool device, KV-passthrough allocates a contiguous LBA range on that SSD and writes the KV caches there using asynchronous NVMe writes. The resulting LBA is recorded in the global lookup table (GLT). Retrieval later uses the GLT’s device index and location fields to issue an asynchronous NVMe read directly against the SSD, with completion handled by SPDK polling on a dedicated CPU core [2606.14779].

The broader design couples this SSD fast path to a bandwidth-aware placement scheme. The KV orchestrator computes placement ratios
\[
p_i = \frac{b_i}{\sum_{j=1}^{N} b_j},
\]
where \(b_i\) is the bandwidth of pool device \(i\). For decoding step \(t\) with offload set \(\mathcal{K}^{(t)}\), the target per-device offload size is
\[
|\mathit{offload\_set}_i^{(t)}| = \left\lfloor p_i \cdot |\mathcal{K}^{(t)}| \right\rfloor + r_i^{(t)}.
\]
KV caches are partitioned in layerwise order, then written in parallel to the assigned pool devices. Host-memory devices use direct memory store, while SSD devices use KV-passthrough. The GLT stores, for each KV cache, a lookup key, device index, device type, and in-device location; retrieval groups requests by device and reads them in parallel, again routing SSD reads through KV-passthrough instead of the filesystem [2606.14779].

The paper reports that unified KV pooling reduces blocked I/O time by up to \(23.2\times\), and for SSD-heavy cases this improvement is directly tied to bypassing filesystem overhead. It further reports TTFTs at \(128\)K context of \(5.6\) s on LLaMA 3.1-8B, \(5.2\) s on GPT-OSS-20B, and \(9.8\) s on Qwen3-30B-A3B. The evaluation also shows that the full unified KV pooling design outperforms both “orchestrator only” and “passthrough only,” which establishes that KV-passthrough alone is insufficient if retrieval remains concentrated on one device, while pooling alone is insufficient if SSD access still pays filesystem overhead [2606.14779].

The implementation assumptions are explicit. SPDK-based passthrough relies on polling, which reduces latency but increases CPU usage; the design dedicates a CPU core to polling. The approach also assumes that the serving engine can maintain and use its own metadata, so it can address KV objects by internal identifiers rather than relying on filesystem services. More advanced completion mechanisms such as user-level interrupts or hybrid polling are mentioned as future work for reducing CPU overhead further [2606.14779].

## 3. KV-Passthrough as position-invariant multimodal KV reuse

Kamera uses KV-Passthrough to denote a training-free, position-invariant multimodal KV-cache reuse operator. The central claim is that standard prefix caches entangle two distinct factors: the position of a chunk in the prompt or window, and the conditioning the chunk has absorbed from what came before it. The method separates these two factors so that multimodal chunks can be reused across sliding windows, reordered inputs, and later recalls without re-encoding the vision or video backbone [2606.23581].

For the positional component, the method relies on exact RoPE composition:
\[
R(\delta)R(p_0)=R(p_0+\delta)=R(p_1).
\]
If a chunk was cached at position \(p_0\) and must be reused at \(p_1\), a rotation by \(\delta=p_1-p_0\) is applied to its cached keys. The paper emphasizes that keys need RoPE re-rotation to move them to the new target position, whereas values are reused directly in the content channel. This operator is formulated to work across MLA, GQA, and MHA once each architecture is split into a content channel and a RoPE or positional channel [2606.23581].

Naive reuse, however, loses the cross-chunk conditioning that a chunk has already absorbed from its antecedent context. Kamera distinguishes **readout**, which is recovered exactly by standard attention state-merge,
\[
o=(1-\mu)o_B+\mu o_A,
\]
from **conditioning**, which is the antecedent-dependent portion already embedded inside the KV state. The missing signal is written as
\[
\Delta = KV(B\mid A) - KV(B\mid \varnothing).
\]
The paper identifies this term as the cross-chunk conditioning deficit or cross-chunk conditioning residue. It reports that blind reuse leaves single-hop recall intact while sharply degrading tasks that require cross-chunk binding [2606.23581].

The repair is a low-rank conditioning patch stored alongside each canonical chunk. At compile time, one conditioned forward on \([prefix \cdot A \cdot B]\) is used to extract \(KV(B\mid A)\), subtract the relocated solo chunk, and factorize the residual:
\[
\Delta = KV(B\mid A) - R(\delta)\,KV(B\mid \varnothing), \qquad \Delta \approx U_m V_m^\top.
\]
Serve-time reuse is then
\[
\widehat{KV}(B\mid A) = R(\delta)\,KV(B\mid \varnothing) + U_m V_m^\top.
\]
This is the paper’s fundamental KV-Passthrough operator: exact relocation plus low-rank conditioning restoration [2606.23581].

The empirical motivation for this decomposition is that the lost conditioning signal is reported as low-rank in feature space, diffuse over tokens, and concentrated in middle or deep layers. The paper states that the conditioning deficit has a knee around rank \(m \approx 8\text{–}16\) and saturates around \(m \approx 32\); rank-32 usually reaches the KL plateau, while rank-16 already recovers much of the gap. It also reports that the patch can optionally be stored only for the deepest roughly half of layers, yielding a “deep-half” variant that preserves about \(95\%\) of full fidelity with reduced storage [2606.23581].

Kamera frames three operations that this operator makes cheap. In **reorder**, the same chunks can be reused in a different order because the positional part is re-rotated and the conditioning patch is stored per chunk or orbit. In **sliding-window survival**, surviving chunks often require RoPE re-rotation only, with no conditioning patch in many cases. In **recall**, an evicted chunk is rehydrated from the canonical chunk plus a fresh patch computed on the then-valid antecedent context [2606.23581].

## 4. Continuity and recurrence mechanisms that resemble passthrough

PipeLive and KV-Fold do not define KV-Passthrough as a named component, but both describe mechanisms in which KV state remains continuously usable across an execution boundary. PipeLive addresses live in-place pipeline-parallel reconfiguration. The paper states that GPUs are already saturated by model weights plus KV cache, leaving little room for new layer placements, and that in-place reconfiguration therefore requires KV resizing and KV synchronization without interrupting inference [2604.12171].

PipeLive’s first enabling mechanism is a redesigned KV cache layout with non-contiguous block access. Instead of a single contiguous GPU buffer per layer, KV is represented as a list of independently allocated GPU blocks. PipeLive modifies the PageAttention block table so that it stores resolved block addresses directly, enabling access to non-contiguous physical KV blocks on the fly and preserving PageAttention-like access efficiency “with no measurable performance degradation in practice.” Shrinking is done by compacting live KV blocks toward the front of the block list and releasing blocks in batches, with overhead “less than 1ms,” while expansion is done by appending newly allocated blocks [2604.12171].

The second mechanism is incremental KV patching, inspired by live virtual machine migration. Sender and receiver threads are spawned for each source–destination GPU pair; the sender maintains a dirty bitmap for newly written KV slots, periodically drains it, gathers the corresponding KV data, and sends a KV patch to the destination. The receiver applies each incoming patch directly into the local layer KV cache. Convergence is tracked by a scheduler-side cumulative token counter \(T_{\mathrm{sched}}\) and a receiver-side applied counter \(T_{\mathrm{applied}}\), with commit gated by
\[
T_{\mathrm{sched}} - T_{\mathrm{applied}} < \tau
\]
for all destination GPUs, where the testbed sets \(\tau = 50\) tokens [2604.12171]. The paper explicitly says that it does **not** use the term KV-Passthrough, yet functionally the design forms a continuous pathway for KV state movement while inference remains live.

KV-Fold presents a different analogue: KV cache itself becomes the accumulator in a recurrent left fold over chunks. A sequence is divided into chunks \(\mathbf{x}_t\), and the transformer performs
\[
(\mathcal{K}^{(t)}, \mathcal{V}^{(t)}) = \mathcal{F}_\theta\big((\mathcal{K}^{(t-1)}, \mathcal{V}^{(t-1)}), \mathbf{x}_t\big),
\]
equivalently
\[
(\mathcal{K}^{(N-1)}, \mathcal{V}^{(N-1)}) = foldl\big(\mathcal{F}_\theta,\;(\emptyset,\emptyset),\;[\mathbf{x}_0,\mathbf{x}_1,\ldots,\mathbf{x}_{N-1}]\big).
\]
At every layer, the keys and values from chunk \(t-1\) are passed into chunk \(t\) unchanged as prefix state with continuous position IDs across the boundary, and chunk \(t\)’s KV is appended and carried forward with “no copy, no transformation” [2605.12471].

The paper directly interprets this as preserving the entire KV history rather than using bounded-memory streaming or sliding-window approximations. It reports stable recurrence dynamics, including a drift plateau whose change between depths \(15\) and \(60\) is \(-0.0003\) nats on Qwen2.5-7B-Instruct with \(T=16{,}384\) and \(C=256\), and \(152/152\) exact retrieval overall on needle-in-a-haystack trials spanning contexts from \(16\)K to \(128\)K tokens and chain depths up to \(511\) on Llama-3.1-8B-Instruct [2605.12471]. This is not passthrough in the storage or reuse-operator sense, but it is a direct form of KV state continuity across chunk boundaries.

These systems motivate a broader interpretation: passthrough-like design often means that KV is neither recomputed wholesale nor collapsed into a lossy summary at the boundary where it must move.

## 5. Retention, compression, and adaptive bypass as neighboring paradigms

Several papers address the same bottleneck—KV state becoming the dominant memory, bandwidth, or quality constraint—by deciding what should remain in the cache rather than by forwarding the cache unchanged. These are adjacent to KV-Passthrough but are not equivalent to it.

TGV-KV is a text-grounded KV cache eviction framework for vision-language models. Its retention policy, Text-Prioritised Retention (TPR), keeps text KV first and fills remaining budget with top-scoring vision KV:
\[
\mathcal{I}_l =
\begin{cases}
\mathcal{T} \cup \text{TopK}(\{s_{l,j}^{(\text{V})}\}_{j \in \mathcal{V}}, \, b_l - N_t), & \text{if } b_l > N_t \\
\text{TopK}(\{s_{l,j}^{(\text{T})}\}_{j \in \mathcal{T}}, \, b_l), & \text{if } b_l \le N_t .
\end{cases}
\]
The paper does not introduce KV-Passthrough as a separate module name, but the retention behavior is its closest analogue: text KV are protected by design because text tokens are described as highly sensitive to eviction, whereas vision tokens are highly redundant [2606.03075].

R-KV similarly focuses on selective retention during long reasoning traces. It scores tokens by
\[
Z_i^h = \lambda I_i^h - (1-\lambda) R_i^h,
\]
combining attention-derived importance and key-similarity-derived redundancy. The paper reports near-100% full-cache performance with only \(10\%\) KV cache, \(105\%\) of full-cache performance with \(16\%\) KV cache, \(90\%\) memory saving, and \(6.6\times\) throughput over standard chain-of-thought reasoning inference [2505.24133]. This is not a passthrough operator; it is a redundancy-aware retention policy.

KVServe addresses disaggregated serving, where KV becomes an explicit payload crossing network and storage boundaries. Its controller decides whether to compress, how to compress, and whether compression is worthwhile. Compression is beneficial only below a profile-specific threshold
\[
B_p^\star \triangleq \left(1-\frac{1}{cr_p}\right)s_p,
\qquad
T_p(c) < T_0(c)\ \Longleftrightarrow\ B < B_p^\star.
\]
When compression is non-beneficial, the system effectively falls back to the default uncompressed path. The paper is explicit that this is not a dedicated passthrough transport path, but it yields passthrough-like behavior through adaptive bypass of compression [2605.13734].

Together, these works delimit the conceptual boundary of KV-Passthrough. Passthrough preserves or reuses KV with minimal semantic alteration; eviction and compression instead decide which fraction of KV should survive. The technical problems are closely related, but the mechanisms are distinct.

## 6. Performance characteristics, trade-offs, and misconceptions

A recurring misconception is that KV-Passthrough denotes a universal optimization that can be applied independently of system context. The papers instead show that its effect depends on the bottleneck being addressed. In unified KV pooling, passthrough is beneficial because long-context TTFT is dominated by storage stalls once the KV footprint spills to SSD; filesystem traversal, not raw device bandwidth, is the residual bottleneck [2606.14779]. In Kamera, KV-Passthrough is beneficial because naive position-independent reuse preserves direct readout but loses cross-chunk conditioning, which selectively damages multi-hop reasoning rather than single-hop recall [2606.23581].

Another misconception is that direct reuse of cached KV is always semantically exact. Kamera explicitly shows that the direct readout of a cached chunk is recovered exactly and for free by the standard state-merge, but blind reuse leaves a cross-chunk conditioning residue inside the KV state, so single-hop recall may remain intact while multi-hop accuracy drops sharply. On two-page document QA, the paper reports single-hop accuracy staying around \(0.57\) while multi-hop accuracy drops from \(0.41\) to \(0.28\) for MLA and from \(0.28\) to \(0.15\) for GQA under blind reuse [2606.23581].

Conversely, another misconception is that bypassing the filesystem or forwarding KV state is by itself sufficient to restore service-level performance. Unified KV pooling reports that “passthrough only” underperforms the full design because retrieval can still be concentrated on one device, while “orchestrator only” underperforms because SSD access still pays filesystem overhead [2606.14779]. PipeLive makes an analogous point in a different context: non-contiguous access without controlled synchronization would not solve KV consistency during live reconfiguration, and synchronization without resize-friendly layout would not solve the memory-pressure problem [2604.12171].

The trade-offs are implementation-specific. Unified KV pooling dedicates a CPU core to SPDK polling and notes the latency-versus-CPU trade-off of polling-based filesystem bypass [2606.14779]. Kamera trades patch rank against storage footprint: rank-16 uses about \(6\%\) of the segment KV bytes, rank-64 about \(25\%\), and deep-half storage reduces bytes further with only a small accuracy drop [2606.23581]. PipeLive trades layer stacking factor \(k\) between memory utilization and reconfiguration granularity [2604.12171]. These patterns suggest that KV-Passthrough is best treated as a systems interface between KV semantics and the dominant hardware or serving bottleneck, not as a universally fixed primitive.

## 7. Relation to long-context and multimodal inference

KV-Passthrough becomes most salient when the cost of recomputation or mediated access grows faster than the model’s useful compute. Long-context LLM serving is one such regime. Unified KV pooling reports TTFT explosions at \(128\)K context and shows that filesystem-heavy SSD retrieval can dominate service time; KV-passthrough is introduced precisely because the serving engine already knows exactly which KV object it wants and therefore does not need the general-purpose file abstraction, metadata handling, and block management of a conventional filesystem [2606.14779].

Multimodal agents provide a second regime. Kamera begins from the observation that agents repeatedly re-examine the same video frames, UI screenshots, and rendered artifacts as their context window slides and reasoning iterates, yet every look-back re-encodes from scratch because prefix caches serve reuse only at a fixed leading position. KV-Passthrough addresses this by making three window operations cheap—reorder, sliding-window survival, and recall—through exact RoPE re-rotation plus conditioning restoration [2606.23581]. TGV-KV reinforces the same systems pressure from a different angle: VLMs are especially exposed to KV growth because images and videos can contribute thousands of visual tokens, and text-grounded retention is needed because vision tokens are highly redundant while text tokens are semantically fragile under eviction [2606.03075].

Reasoning-heavy decoding provides a third regime. R-KV argues that reasoning models generate excessively long and repetitive chain-of-thought traces, producing very large KV caches not because every token is equally informative but because many tokens are redundant reflections, re-checks, and verbose self-dialogue [2505.24133]. KV-Fold, by contrast, shows that when exact recall matters and linear cache growth is acceptable, a frozen pretrained transformer can reuse the entire prior KV history across hundreds of chunk transitions without retraining [2605.12471]. These papers do not share a common passthrough term, but they converge on the same systems insight: once KV becomes the state that determines memory footprint, retrieval latency, or cross-chunk fidelity, efficient inference depends on treating KV as a first-class object rather than as an incidental by-product of attention.

Taken together, the literature presents KV-Passthrough as an emerging umbrella concept for direct KV data paths, position-invariant KV reuse, and continuity-preserving movement of attention state. The exact mechanism differs by setting, but the underlying principle is consistent: avoid unnecessary mediation, preserve the part of KV state that already encodes useful computation, and expose that state to the next stage of serving with as little distortion and overhead as the deployment regime allows.

Source: https://www.emergentmind.com/topics/kv-passthrough