PolyKV: Shared KV Cache & PolyKervNets

• PolyKV is a term describing two distinct concepts: a shared, asymmetrically-compressed KV cache for multi-agent LLM inference and a PolyKervNet approach for private CNN inference.
• In the transformer setting, PolyKV compresses shared prefix keys with 8-bit quantization and values with TurboQuant, reducing memory usage by over 97% via a single KV pool.
• In the privacy domain, PolyKervNets replace standard convolutions with polynomial kernels and activations to eliminate expensive nonlinear operations for secure inference.

Searching arXiv for papers on PolyKV and PolyKervNets to ground the article. PolyKV denotes two distinct research usages on arXiv. In current transformer-systems literature, it refers to a system in which multiple concurrent inference agents share a single, asymmetrically compressed KV cache pool for multi-agent LLM inference (Patel et al., 27 Apr 2026). In earlier privacy-preserving CNN literature, “PolyKV” also appears as shorthand for PolyKervNets, which replace linear convolutions and explicit activations with polynomial kernels or polynomial activations (Aremu, 2023). The dominant contemporary meaning is the KV-cache system: a method for building the KV cache for a shared prefix once, compressing it asymmetrically, and injecting it into independent agent contexts through HuggingFace DynamicCache objects.

1. Terminology and scope

The two usages of PolyKV address different technical objects. One concerns transformer key–value caches; the other concerns polynomial-kernel neural networks for private inference. The naming overlap is therefore terminological rather than architectural.

Usage	Domain	Core object
PolyKV	Multi-agent LLM inference	Shared asymmetrically-compressed KV cache pool
“PolyKV”-style PolyKervNets	Privacy-preserving CNN inference	Polynomial kernels or polynomial activations

In the LLM setting, the problem begins with the standard transformer cache. For each layer $\ell$ and token position $t$, the model computes keys $K_{\ell,t} \in \mathbb{R}^{H \times D}$ and values $V_{\ell,t} \in \mathbb{R}^{H \times D}$. For a single sequence of length $T$, the KV cache size per layer is approximately

$$\text{size}_\ell \sim T \times H \times D \times (\text{bytes per K} + \text{bytes per V}),$$

and for a model with $L$ layers,

$$\text{KV size} \sim L \times T \times H \times D \times (\text{bytes per K} + \text{bytes per V}).$$

When $N$ agents process the same long prefix and then diverge, most serving stacks allocate $N$ separate caches, so total KV memory grows approximately as

$t$0

PolyKV targets the redundant $t$1 term by storing the shared prefix once and making its memory cost effectively $t$2 in the number of agents (Patel et al., 27 Apr 2026).

2. Problem formulation for multi-agent LLM inference

PolyKV is designed for a setting with one shared long document context and multiple concurrent agents, each with its own prompt, decoding loop, and continuation. The agents share identical prefix tokens but have independent suffix tokens. In this regime, memory rather than arithmetic often becomes the bottleneck, especially for 8B-scale models and 4K-token contexts.

The system addresses three issues. The first is memory blow-up with many agents: for Llama-3-8B with 4K tokens, 15 agents need about 19.8 GB just for KV cache under the naive paradigm. The second is redundant bandwidth and compute for the shared prefix, because each agent separately performs a full prefill and stores its own prefix KV. The third is the resulting scaling limitation, since KV memory often determines how many agents or how much context can be supported on a given GPU (Patel et al., 27 Apr 2026).

The paper’s central design choice is to “pay almost once” for the shared context. A single prefill pass produces full-precision K and V tensors for the common prefix, after which PolyKV compresses them asymmetrically and stores them in a single shared pool. Each agent subsequently initializes from that pool and then accumulates only its own suffix KV in a private cache. This preserves logical independence across agents while removing duplication of the common prefix.

3. System architecture and cache lifecycle

PolyKV organizes the system around two abstractions: a SharedKVPool and a per-agent cache context. The shared pool stores one compressed representation of the prefix KV per transformer layer. A logical agent then creates a fresh HuggingFace DynamicCache, reads each layer’s K and V from the pool, dequantizes them to the runtime dtype, and injects them through cache.update(...). Generation proceeds with standard past_key_values mechanisms (Patel et al., 27 Apr 2026).

This produces an important separation between physical sharing and logical independence. Physical sharing means that the compressed prefix resides once in memory and is read by all agents. Logical independence means that each agent has a distinct decoding process, distinct suffix tokens, and a distinct cache object into which the shared prefix has been loaded. Agents’ suffix KVs are never written back to the shared pool; they remain private to the respective agent.

The method is therefore a system-level intervention rather than a change to transformer attention itself. Attention runs in the model’s normal float dtype after dequantization. No custom low-precision attention kernel is required. This allows implementation entirely in Python and PyTorch, and the paper presents integration through HuggingFace Transformers and DynamicCache objects (Patel et al., 27 Apr 2026).

A plausible implication is that PolyKV is easiest to adopt in stacks that already expose an explicit cache abstraction. The paper’s formulation assumes standard transformer KV shapes and indicates that extension to MHA, GQA, or MQA primarily requires adapting to model-specific cache layouts.

4. Asymmetric compression scheme

PolyKV uses asymmetric compression because keys and values have different numerical sensitivities. Keys enter the attention score computation

$t$3

so aggressive key quantization can destabilize softmax. Values are more robust. Accordingly, PolyKV stores keys with 8-bit linear quantization and values with TurboQuant MSE: a Fast Walsh–Hadamard Transform rotation followed by 3-bit Lloyd–Max quantization (Patel et al., 27 Apr 2026).

For keys, PolyKV applies per-tensor linear quantization. If $t$4 is a floating-point key tensor, the scale is

$t$5

the quantized tensor is

$t$6

and dequantization is

$t$7

The paper refers to this as q8_0, with one scale per K tensor per layer.

For values, each vector $t$8 is first rotated by FWHT and normalized,

$t$9

The resulting coordinates are then quantized against the 3-bit Lloyd–Max centroids tuned to $K_{\ell,t} \in \mathbb{R}^{H \times D}$0,

$K_{\ell,t} \in \mathbb{R}^{H \times D}$1

For each coordinate,

$K_{\ell,t} \in \mathbb{R}^{H \times D}$2

Dequantization maps each stored index back to its centroid and applies inverse FWHT with the stated scaling convention. The paper reports the TurboQuant bound

$K_{\ell,t} \in \mathbb{R}^{H \times D}$3

and for $K_{\ell,t} \in \mathbb{R}^{H \times D}$4 gives $K_{\ell,t} \in \mathbb{R}^{H \times D}$5.

The resulting average storage cost over keys and values is

$K_{\ell,t} \in \mathbb{R}^{H \times D}$6

so relative to a 16-bit baseline the theoretical compression ratio is

$K_{\ell,t} \in \mathbb{R}^{H \times D}$7

The paper reports this 2.91× ratio as stable across all tested configurations (Patel et al., 27 Apr 2026).

5. Empirical characteristics

The reported evaluation spans SmolLM2-1.7B-Instruct and Llama-3-8B-Instruct, three context-length regimes from about 600 tokens to 7,194 tokens, and up to 15 concurrent agents. Metrics include perplexity on the last 30% of tokens, BERTScore F1 for Llama-3-8B, token overlap in the SmolLM2 experiments, and KV memory usage (Patel et al., 27 Apr 2026).

The headline result is that PolyKV maintains a stable 2.91× compression ratio while reducing total shared-prefix KV memory dramatically as the number of agents grows. For Llama-3-8B with 15 agents sharing a 4K-token context, the paper reports KV cache memory dropping from 19.798 GB to 0.454 GB, a 97.7% reduction. On that same configuration, perplexity degradation is reported as +0.57% and mean BERTScore F1 as 0.928. At 10 agents and the same 7,194-token context, memory falls from 13.199 GB to 0.454 GB, a 96.6% reduction (Patel et al., 27 Apr 2026).

A notable empirical pattern is agent-count invariance. At fixed context length, perplexity deltas do not grow with the number of agents. For SmolLM2 at about 600 tokens, the delta is +0.53% for both 3 and 5 agents; at 1,851 coherent tokens, it is $K_{\ell,t} \in \mathbb{R}^{H \times D}$8 for both 3 and 5 agents. For Llama-3-8B at 1,837 tokens, the delta remains +1.59% for 3, 5, and 10 agents. This follows directly from the architecture: the shared prefix is stored once, and agent multiplicity does not change the compressed representation of that prefix (Patel et al., 27 Apr 2026).

The paper also reports a context-length effect. At longer contexts, the perplexity delta improves rather than worsening: for SmolLM2 it inverts to $K_{\ell,t} \in \mathbb{R}^{H \times D}$9 at 1,851 coherent tokens, and for Llama-3-8B it improves from +1.59% at about 2K tokens to +0.57% at 7,194 tokens. The paper presents a hypothesis rather than a proof: FWHT-domain value quantization may inject small uniform noise that disrupts spurious correlations and “attention sinks,” acting somewhat like regularization during inference (Patel et al., 27 Apr 2026).

6. Position within KV-cache research and current limitations

PolyKV sits at the intersection of two earlier lines of work: per-request KV compression and multi-agent KV sharing. Prior KV compression methods listed by the paper include KIVI, KVQuant, LeanKV, AsymKV, RotateKV, KVLinC, TurboAngle, and KVTC. Prior multi-agent or multi-session sharing mechanisms listed include KVFlow, KVCOMM, LRAgent, Agent Memory, and RelayCaching. The paper’s claim of novelty is specifically their combination: asymmetric K/V quantization, FWHT-based value compression, a single shared KV pool, and concurrent multi-agent inference with multi-reader access to a lossy-compressed shared prefix (Patel et al., 27 Apr 2026).

The article’s evidence base is explicitly bounded. The experiments cover SmolLM2-1.7B and Llama-3-8B, but not 70B-scale models. WikiText-2 evaluation is performed on a fixed context window rather than the standard sliding-window protocol used in some KV-compression baselines. The paper does not report TTFT, throughput, or end-to-end latency, even though those are natural system-level questions for a cache-sharing method. It also notes a practical ceiling in the reported hardware environment: on T4 GPUs, Llama-3-8B prefills beyond about 8K tokens produced OOMs, which the paper attributes to hardware rather than to a PolyKV design limit (Patel et al., 27 Apr 2026).

The reported failure modes remain prospective. The paper suggests that very noisy or incoherent contexts may not exhibit the same beneficial perplexity behavior as longer coherent contexts, and that tasks with extreme token-level sensitivity, such as code generation or mathematics, may respond differently. Those statements are framed as robustness concerns and future work rather than demonstrated empirical conclusions.

7. PolyKV as shorthand for PolyKervNets

A separate line of research uses “PolyKV” as shorthand for PolyKervNets in privacy-preserving inference. There, the objective is not KV-cache sharing but elimination of expensive nonlinear operations such as ReLU and max-pooling under HE, secret sharing, or garbled-circuit regimes. PolyKervNets replace a standard linear convolution

$V_{\ell,t} \in \mathbb{R}^{H \times D}$0

with a polynomial kernel

$V_{\ell,t} \in \mathbb{R}^{H \times D}$1

where $V_{\ell,t} \in \mathbb{R}^{H \times D}$2 is the polynomial degree and $V_{\ell,t} \in \mathbb{R}^{H \times D}$3 is a learnable balance factor (Aremu, 2023).

The 2023 paper studies the difficulty of scaling these polynomial models to deeper architectures. Its main intervention is a learnable regularization scalar,

$V_{\ell,t} \in \mathbb{R}^{H \times D}$4

with $V_{\ell,t} \in \mathbb{R}^{H \times D}$5, together with the derived quadratic activation

$V_{\ell,t} \in \mathbb{R}^{H \times D}$6

called React-PKN. That work reports that original PolyKervNets could not train beyond about 18 residual layers without collapse, whereas the regularized formulation, careful optimization, and knowledge distillation extended training to ResNet-32 and ResNet-50 on CIFAR-10 (Aremu, 2023).

This suggests that “PolyKV” now has a genuine cross-domain ambiguity in arXiv-facing discussion. In one usage it denotes polynomial, activation-free or React-PKN-style CNNs for private inference; in the other it denotes a shared, asymmetrically-compressed transformer KV pool for concurrent agents. The overlap is lexical, not methodological.