Unified KV Pooling: Principles & Architectures

• Unified KV pooling is a design principle that redefines the key–value cache as a pooled resource, allowing dynamic selection, pruning, and exact attention.
• It is applied across long-context acceleration, serving systems, multimodal reuse, and compression frameworks to improve efficiency and scalability.
• Experimental setups report significant KV cache memory reduction, throughput gains, and minimal impact on accuracy across diverse architectures.

Unified KV pooling denotes a family of mechanisms that treat the key–value cache as a pooled resource rather than a fixed, per-request, monolithic artifact. In the cited literature, the term spans hardware-native pruning-and-selection pipelines for long-context attention, shared and virtualized memory pools for multi-model serving, position-invariant multimodal reuse, compression-and-reuse frameworks, and global pools that factor shared structure away from request-specific or user-specific residuals [2504.07479] [2606.24506] [2606.23581] [2601.19178] [2604.24971]. This suggests a common abstraction: KV state is preserved, filtered, reconstructed, migrated, or shared according to utility, bandwidth, and reuse structure, while exact attention or downstream computation is restricted to the retained or rehydrated subset.

1. Conceptual scope and recurring design pattern

Across the literature, unified KV pooling is used for several technically distinct but structurally related designs. In hardware-centric work, it denotes an end-to-end pooling flow that performs approximate selection, accumulative eviction, and exact attention on a selected subset [2504.07479]. In serving systems, it denotes a shared, virtualized KV-cache pool or a global pool spanning multiple devices, models, or roles [2606.24506] [2606.14779] [2606.31093]. In reuse-oriented work, it denotes a position-free, content-addressed store in which position and conditioning are separated, or a write-once, read-many shared prefix cache for concurrent agents [2606.23581] [2604.24971]. In compression-oriented work, it denotes channel pooling, structural pooling, or cross-user pooling [2512.06727] [2601.19178].

Setting	Unified KV pooling mechanism	Representative paper
Long-context accelerator	O(1) array-time CAM selection, charge-domain accumulative static eviction, current-domain exact attention	[2504.07479]
Cold MoE serving	Shared, virtualized KV-cache pool plus separate weights pool	[2606.24506]
Long-context offload	Multiple host-memory modules and SSDs aggregated into one logical pool	[2606.14779]
Multimodal reuse	Exact RoPE re-rotation plus a low-rank conditioning patch	[2606.23581]
Cross-user recommendation	Learnable global KV pool plus low-dimensional user-specific KV	[2601.19178]
Multi-agent inference	Single shared, asymmetrically-compressed KV cache pool	[2604.24971]

The common design pattern is that KV is elevated to a first-class resource. Some systems pool across tokens, heads, and layers; others pool across models, requests, agents, users, devices, or tiers. Some preserve exactness only for a selected set; others preserve exactness algebraically by re-rotation; others trade exactness for compression while keeping the attention interface unchanged. A plausible implication is that unified KV pooling is best understood as a general resource-management and representation-management principle rather than a single algorithmic primitive.

2. Attention-side pooling, pruning, and selective exactness

In long-context Transformer inference, unified KV pooling often appears as a staged replacement for dense attention. UniCAIM defines the most explicit version of this pattern. Its workflow has three array-level modes: CAM mode for dynamic pruning via approximate similarity, charge-domain CIM for static pruning via accumulative similarity, and current-domain CIM for exact attention on the pruned set [2504.07479]. The baseline attention equation remains

$$
\mathrm{Attn}(Q,K,V) = \mathrm{softmax}!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V.
$$

For a single query $q$ and keys ${k_i}$, the pruning score is the dot product

$$
s_i = q^{\top} k_i.
$$

UniCAIM first derives a candidate set $C_d$ through CAM-mode approximate similarity, then maintains an accumulative similarity score per row,

$$
S_i \leftarrow S_i + s_i^{(t)}, \qquad S_i = \sum_t s_i^{(t)},
$$

and evicts

$$
i^* = \arg\min_i S_i.
$$

Exact attention is then restricted to the selected set $C$,

$$
\alpha_i = \mathrm{softmax}!\left(\frac{q^{\top}k_i}{\sqrt{d}}\right), \; i \in C,
\qquad
y = \sum_{i \in C}\alpha_i v_i.
$$

The architectural significance is that dynamic selection, static eviction, and exact attention are all carried out in-place in one FeFET-based CAM/CIM fabric. UniCAIM reports circuit-level AEDP reductions of $8.2\times$ to $831\times$ over Sprint, TranCIM, and CIMFormer, while keeping application-level accuracy comparable with dense attention on LongBench tasks [2504.07479].

LeanKV presents a different but related unification. It treats compression as a continuum driven by token importance: the most important tokens are kept at higher precision, moderately important tokens at lower precision, and the least important tokens are pruned [2412.03131]. The framework differentiates keys and values, uses attention-derived token significance, and allocates memory adaptively per head. Its hetero-KV configurations include K8V4 and K4V2, and it reports KV cache compression by $3.0\times$–$5.0\times$ without accuracy loss, up to $11.0\times$ with under $5\%$ loss, together with throughput gains of $1.9\times$–$2.5\times$ with no accuracy loss and up to $6.9\times$ with $\leq 5\%$ loss [2412.03131].

A related architecture-agnostic variant is KV-CAR, which combines autoencoder-based compression along the embedding dimension with similarity-driven KV reuse across adjacent layers and heads [2512.06727]. Here, unified pooling means that the cache holds fewer, lower-dimensional KV tensors, and some heads in adjacent layers may reuse prior KV instead of storing new tensors. The paper reports up to $47.85$ percent KV cache memory reduction with minimal impact on perplexity and zero-shot accuracy [2512.06727].

3. Shared pools for serving, disaggregation, and tiered memory

In serving systems, unified KV pooling usually means that KV capacity is provisioned to aggregate active demand rather than to worst-case demand per model or per request. CrossPool targets cold MoE models and separates FFN weights and KV-cache into two GPU memory pools: a weights pool and a KV-cache pool [2606.24506]. Its planner models active KV tokens for request $i$ at age $u$ as

$$
Q_i(u) = \big(O_{p,i} + O_{d,i}\cdot(u/T_i)\big)\cdot \mathbb{1}_{0 \le u < T_i},
$$

then defines

$$
K_M(t) = \kappa(M)\cdot \sum_i Q_i(t-A_i), \qquad
K_{\mathrm{pool}}(t) = \sum_M K_M(t),
$$

and provisions the pool budget by a quantile estimate,

$$
B = \left\lceil K_{\mathrm{pool}}^{(q)} / P \right\rceil.
$$

This design keeps attention local to the KV pool, virtualizes allocation with CUDA VMM, and uses admission control rather than evicting active pages. At $0.8$ RPS/model, it reports P99 TBT reductions of $7.6\times$, $10.4\times$, and $7.3\times$ over kvcached for Qwen3-30B-A3B, DeepSeek-V2-Lite, and GLM-4.7-Flash, respectively [2606.24506].

For very long contexts, unified KV pooling has also been formulated as a host-memory/SSD pooling problem. “Unified KV Pooling to Accelerate Long-Context LLM Serving” aggregates multiple host-memory modules and SSDs into a single logical pool and distributes KV caches across devices based on their bandwidth [2606.14779]. The scheduler uses

$$
p_i = \frac{b_i}{\sum_{j=1}^{N} b_j},
$$

and assigns

$$
|offload_set_i^{(t)}| = \lfloor p_i \cdot |K^{(t)}| \rfloor + r_i^{(t)}.
$$

The system also introduces KV-passthrough, which bypasses the kernel filesystem and directly accesses SSD-resident KV caches from user space via SPDK. The motivating baseline shows TTFT reaching about $30.7$ s at $128$K input tokens and SSD retrieval spending $84\%$ of its time in the kernel filesystem. Unified KV pooling reduces TTFT in long contexts about $4.1\times$ over state-of-the-art techniques, with all tested models under $10$ s, and reduces blocked I/O time by up to $23.2\times$ [2606.14779].

BanaServe applies the same abstraction to prefill–decode disaggregation. Its Global KV Cache Store allows routers to perform purely load-aware scheduling, unconstrained by cache placement, while attention-level KV migration and layer-level weight migration rebalance memory-bound and compute-bound tiers [2510.13223]. The system reports $1.2\times$–$3.9\times$ higher throughput and $3.9\%$–$78.4\%$ lower total processing time than vLLM, and $1.1\times$–$2.8\times$ higher throughput with $1.4\%$–$70.1\%$ latency reduction over DistServe [2510.13223].

4. Position-invariant reuse and multimodal or cross-task pooling

A different branch of unified KV pooling treats reuse itself as the primary problem. Kamera analyzes why naive position-independent KV reuse fails in multimodal agents: the direct readout of a cached chunk is recovered exactly by the standard state-merge, but the cross-chunk conditioning that a chunk absorbs from its neighbours is lost [2606.23581]. The missing term is the conditioning deficit

$$
\Delta_\ell = KV_\ell(B!\mid!A) - R(\theta(p_1-p_0))\,KV_\ell(B!\mid!\varnothing),
$$

and the unified operator is

$$
\widehat{KV}\ell(B!\mid!A,p_1) =
R(\theta(p_1-p_0))\,KV\ell(B!\mid!\varnothing,p_0) + U_{\ell,m}V_{\ell,m}^{\top}.
$$

This combines exact RoPE re-rotation with a training-free low-rank conditioning patch. The same operator supports reorder, sliding-window survival, and recall, and the paper states that a rank-$m$ patch recovers full task accuracy on cross-chunk-binding benchmarks, MM-NIAH across two attention families and two-page doc-QA, at a fraction of the KV footprint [2606.23581].

In distributed multimodal inference, Omni-Flow generalizes the pool to a system-wide abstraction. Its Global Params Pool is a global, paged, and distributed store backed by three-tier paged storage: L1 GPU HBM, L2 CPU RAM, and L3 SSD [2606.31093]. Allocation, migration, reuse, and reclamation are centralized in the data plane; cross-role transmission is direct; and KV reuse is lifted from local prefix hits to cross-role, cross-request, cross-node prefix matching. The same framework uses a unified SGLang interface and Chained Paged Hashing, and it allows diffusion models to reuse the LLM forward path under unified parallel semantics [2606.31093].

OxyGen specializes unified KV cache management to Mixture-of-Transformers vision-language-action models. It defines the backbone-produced KV segment for a frame as a first-class shared resource across tasks and over time, enabling cross-task KV sharing and cross-frame continuous batching [2603.14371]. A single observation frame is prefetched once, action and language experts consume the same backbone cache, and multiple active language requests are decoded in parallel from resumable states. On a single NVIDIA RTX 4090 GPU, OxyGen reports up to $3.7\times$ speedup over isolated execution, over $200$ tokens/s language throughput, and $70$ Hz action frequency simultaneously without action quality degradation [2603.14371].

5. Compression, collaborative sharing, and write-once read-many pools

Several works define unified KV pooling through compression and shared structure. KV-CAR combines learned embedding-dimension compression with similarity-driven structural reuse across adjacent layers and heads [2512.06727]. For a decoder-only transformer, it writes the KV cache size as

$$
KV_Cache_Size = 2 \times P \times N_{\mathrm{layers}} \times d_{\mathrm{model}} \times L_{\mathrm{seq}} \times B,
$$

and then replaces full-dimensional cached tensors with compressed latents and reused tensors when head similarity falls below thresholds. The framework is architecture-agnostic and leaves the base attention computation unchanged [2512.06727].

CollectiveKV moves the same idea to sequential recommendation. It starts from an SVD analysis in which the KV matrix is decomposed into a shared and a user-specific part,

$$
X_{\mathrm{shared}} = U_r \Sigma_r V_r^{\top}, \qquad
X_{\mathrm{spec}} = X - X_{\mathrm{shared}},
$$

and implements a learnable global KV pool plus low-dimensional user-specific KV [2601.19178]. The final cached representation is assembled by concatenation,

$$
K_{\mathrm{final}} = \mathrm{concat}(K_{\mathrm{user}}, K_{\mathrm{shared}}), \qquad
V_{\mathrm{final}} = \mathrm{concat}(V_{\mathrm{user}}, V_{\mathrm{shared}}).
$$

With $k=10$, the principal components preserve $\geq 90\%$ of the variance while the residual contains $< 10\%$. Across five sequential recommendation models and three datasets, the method compresses the per-user KV cache to as low as $0.8\%$ of the original size while maintaining or even enhancing model performance [2601.19178].

PolyKV applies the pool abstraction to concurrent inference agents. It constructs one write-once, read-many shared KV cache over a shared document context and injects it into multiple HuggingFace DynamicCache objects [2604.24971]. Compression is asymmetric: Keys use int8 q8_0 and Values use TurboQuant MSE with FWHT rotation and 3-bit Lloyd–Max quantization. The theoretical compression ratio is

$$
CR = \frac{16 + 16}{8 + 3} = \frac{32}{11} \approx 2.91.
$$

The paper reports a stable $2.91\times$ compression ratio across all configurations. For Llama-3-8B with $15$ agents sharing a $4$K-token context, KV cache memory is reduced from $19.8$ GB to $0.45$ GB, a $97.7\%$ reduction, with only $+0.57\%$ perplexity degradation and a mean BERTScore F1 of $0.928$ [2604.24971].

A related but more indirect unification appears in coordinated routing and eviction. “Randomization Boosts KV Caching, Learning Balances Query Load” does not build a monolithic global KV memory; instead, it gives a unified mathematical model coupling per-worker cache eviction and query routing [2601.18999]. This effective unified abstraction over coordinated local caches yields up to $6.92\times$ improvement in cache hit rate, $11.96\times$ reduction in latency, $14.06\times$ reduction in time-to-first-token, and $77.4\%$ increase in throughput over state-of-the-art methods [2601.18999]. This suggests that unified pooling can also be realized through coordinated policy over distributed caches rather than by physical centralization alone.

6. Limitations, semantics, and common misconceptions

A recurrent misconception is that unified KV pooling is synonymous with a particular replacement policy or an automatic speedup. “Resident KV Claims” explicitly rejects that interpretation: it introduces a conformance contract for future reuse under active KV pressure, not a production speedup or a new cache-replacement algorithm [2605.24259]. Its central inequality is the feasibility condition between protected resident KV and active live KV,

$$
\text{protected_resident_kv} + \text{active_live_kv} \leq \text{usable_kv},
$$

and hard protected resident claims convert unreported resident loss into scheduler-visible active refusal with direct blocking-claim attribution [2605.24259]. The conceptual importance is that pooling also requires lifecycle semantics, materialization predicates, and telemetry, not just capacity sharing.

Another misconception is that larger pools always imply exact reuse. Kamera shows that blind reuse preserves single-hop recall but can halve multi-hop accuracy because the conditioning deficit is missing; the low-rank patch is required to restore cross-chunk binding, and non-RoPE schemes such as ALiBi are out of scope for its re-rotation operator [2606.23581]. CrossPool similarly makes explicit that active KV pages are never evicted: if the pooled page budget is exhausted, admission control queues or rejects new requests [2606.24506]. In hardware realizations such as UniCAIM, device variability, endurance, and bit precision remain active constraints: the cell design is demonstrated up to 3-bit multilevel keys, and very large KV caches require array tiling and hierarchical selection [2504.07479].

There are also systems-level limits. Unified host-memory/SSD pooling depends on multiple devices and on user-space SSD access; short contexts that never spill to SSD benefit less [2606.14779]. Omni-Flow validates architectural support for unified pooling, weight sharing, and direct transmission, but identifies quantitative speedups, hit ratios, and GPU memory reduction measurements as future work [2606.31093]. BanaServe focuses on performance rather than fault recovery, and its policies are reactive rather than predictive [2510.13223]. PolyKV does not report system-level TTFT or throughput metrics, and its shared pool remains a stable prefix while per-agent decoded tails stay private [2604.24971].

Taken together, these constraints indicate that unified KV pooling is a broad design space with several orthogonal axes: selection versus reconstruction, centralization versus coordination, exactness versus approximation, and static capacity versus lifecycle-aware arbitration. The literature converges on the view that KV is not merely cached state but a structured resource whose value depends on sparsity, reuse topology, position, modality, and memory hierarchy.