---
title: 'BatchQuantizedKVCache: Batched Inference Engine'
url: https://www.emergentmind.com/topics/batchquantizedkvcache
type: topic
---

# BatchQuantizedKVCache: Batched Inference Engine

BatchQuantizedKVCache denotes a batched key-value cache representation and execution layer in which multiple sequences or agents share a quantized KV-cache runtime. In a concrete system for multi-agent inference on edge devices, it is the in-memory engine that holds multiple agents’ KV caches in compact 4-bit (“Q4”) form, merges multiple agents’ Q4 blocks into a single batch tensor, runs `fused_scaled_dot_product_attention` on that batch, and splits results back out [2603.04428]. Related work uses the same term for page-based mixed-precision caches, batched additive vector quantization, quantile-based block quantization, and batched residual vector quantization [2511.18643], [2506.18879], [2505.16210], [2603.16435]. A plausible implication is that the term functions less as a single canonical algorithm than as an implementation-level abstraction for batched, compressed KV-cache execution.

## 1. Conceptual scope and variants

Across the recent literature, BatchQuantizedKVCache appears at the intersection of three requirements: compressed KV storage, batched execution, and direct compatibility with autoregressive attention. In the edge-device multi-agent setting, the motivation is explicit: device RAM is too small to hold every agent’s KV cache simultaneously, so quantized persistence and direct cache restoration are used to avoid redundant re-prefill computation [2603.04428]. In high-throughput GPU settings, the same abstraction is adapted to mixed-precision page layouts, commutative codebooks, or vector-quantized indices that can be reconstructed or attended over efficiently [2511.18643], [2506.18879], [2603.16435].

| Variant | Representation | Batched execution |
|---|---|---|
| Agent Memory Below the Prompt | fixed-size 256-token Q4 blocks in safetensors | merge multiple agents’ Q4 blocks into a single batch tensor |
| Kitty | two unified 2-bit tensors plus metadata | replace the default KVCache with `BatchQuantizedKVCache` |
| CommVQ / NQKV / VQKV | packed bits or codebook indices | minibatches, fused bit-unpack, or on-demand reconstruction |

The common denominator is not a single quantizer. Rather, it is the combination of compressed KV state with a runtime that can batch multiple cache instances without reverting to full-precision cache residency. This is particularly explicit in the multi-agent system, where the BatchQuantizedKVCache works hand-in-glove with disk persistence so that evicted caches can be reloaded in sub-second time rather than incur a full $O(n)$ prefill [2603.04428].

## 2. Data structures, storage layout, and cache lifecycle

In the persistent multi-agent design, the storage hierarchy is centered on a `BlockPool` that partitions every agent’s KV cache into fixed-size 256-token blocks, stored in safetensors format per-agent and per-layer [2603.04428]. The core structures are `KVBlock`, `AgentBlocks`, `ModelCacheSpec`, and `BlockPool`. A `KVBlock` stores packed `uint32` data together with per-group `bfloat16` scales and biases, and each block becomes six tensors: packed data, scales, and biases for both K and V. Loading and saving are performed by `mx.load_safetensors()` and `mx.save_safetensors()` [2603.04428].

The same paper specifies a high-level cache lifecycle: `save_agent_cache`, `load_agent_cache`, and `quantize_and_store`. During prefill, layer outputs are sliced into 256-token blocks, quantized by `Q4Quantizer::quantize`, appended into the per-agent block list, and then serialized to disk [2603.04428]. This organization makes the in-memory batch engine inseparable from the persistence layer.

Kitty adopts a different layout. Each mixed-precision Key page is decomposed into `Tensor_low`, which stores the low 2 bits of every channel, and `Tensor_high`, which stores the high 2 bits only for boosted channels, together with `Boost_IDX` metadata [2511.18643]. The runtime also maintains full-precision `Sink`, a `Local` window, and a `Q-Buffer`. This page-centric design is intended to preserve coalescing and avoid divergence under dynamic 4-bit channel boosts [2511.18643].

For Q4 blockwise storage, the memory formulas are stated explicitly as
$$
\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n
$$
and
$$
\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.
$$
Assuming $G=64$, the ratio is
$$
Q4/FP16 = (1 + 8/64)/4 = 0.281,
$$
corresponding to a $72\%$ reduction [2603.04428].

## 3. Quantization formulations

The quantization schemes associated with BatchQuantizedKVCache differ substantially across systems.

In the Q4 persistent-cache system, blockwise quantization uses group size $G = 64$ tokens and computes
$$
\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),
$$
followed by
$$
\mathrm{packed\_data}[i] = \mathrm{round}((x[i]-\mathrm{bias})/\mathrm{scale})
$$
clipped to $[-8\ldots +7]$ [2603.04428]. This is a signed 4-bit blockwise scheme oriented toward persistence and direct reload.

Kitty uses Dynamic Channel-wise Precision Boost. For a page $X\in\mathbb{R}^{D\times T}$, channel sensitivity is approximated by
$$
s_i = \frac{1}{T}\sum_{t=1}^{T}|X_{i,t}|,
$$
after which the top $f$ fraction of channels are boosted to 4 bits while the remainder stay at 2 bits [2511.18643]. Reconstruction combines the low and high parts through
$$
q_{i,t} = x_{low}[i,t] + (x_{high}[k,t] \ll 2),
$$
and dequantization is
$$
K_{i,t} = \alpha_i \cdot (q_{i,t} - z_i).
$$
The paper summarizes the effective average precision as
$$
B_{avg}=2+2f,
$$
with examples such as $f=12.5\% \Rightarrow B_{avg}=2.25$ bits and $f=25\% \Rightarrow B_{avg}=2.5$ bits [2511.18643].

CommVQ replaces scalar quantization with additive vector quantization. A lightweight encoder maps each token vector $t_i\in\mathbb{R}^d$ to a binary code
$$
s_i = E(t_i)\in\{0,1\}^{N_c},
$$
and reconstruction is
$$
\hat t_i = s_i C.
$$
The reduction rate is given by
$$
RR=1-\frac{N_c}{16d},
$$
so that $N_c=2d$ corresponds to $2$-bit quantization and $N_c=d$ to $1$-bit quantization [2506.18879]. Its key technical refinement is a RoPE-commutative codebook, which enables reordered decoding inside self-attention rather than separate per-token decode-then-attend [2506.18879].

NQKV adopts a per-block NormalFloat quantizer under the assumption that block elements follow a normal distribution. For block statistics $(\mu_k,\sigma_k)$, quantization is
$$
z=\frac{x-\mu_k}{\sigma_k},\qquad u=\Phi(z),\qquad q=\mathrm{round}(u\cdot(2^b-1)),
$$
and dequantization is
$$
\hat u=\frac{q}{2^b-1},\qquad \hat z=\Phi^{-1}(\hat u),\qquad \hat x=\mu_k+\sigma_k\cdot \hat z.
$$
The method is described as information-theoretically optimal under the normality assumption [2505.16210].

VQKV uses batched multi-stage residual vector quantization. Keys and values are flattened into matrices of shape $[B\cdot L\cdot H]\times D$, quantized into codebook indices, and reconstructed on demand by
$$
\hat X[m] = \sum_{i=1}^{N^k} Q_i^k[\mathrm{idx}_i^k[m]]\,{W_i^k}^\top,
$$
with an analogous expression for $\hat Y[m]$ on the value side [2603.16435]. Here the stored object is not a low-bit scalar tensor but a compact index tensor.

## 4. Batched execution and attention integration

The defining runtime feature of BatchQuantizedKVCache is batched execution over multiple cache instances. In the multi-agent Q4 system, the interface is organized around three primitive operations:
- `merge(batch_blocks: Vec<AgentBlocks>) → (merged_K, merged_V, mask)`
- `update_and_fetch(merged_K, merged_V, mask) → (new_K, new_V, output_token_logits)`
- `extract(new_K, new_V) → Vec<AgentBlocks>` [2603.04428]

The corresponding batched decode procedure first gathers each agent’s cached K,V blocks and pads to max length, then launches a single fused Q4 attention call on the GPU, then splits updated caches back to per-agent blocks [2603.04428]. A single-threaded `ConcurrentScheduler` interleaves 512-token “chunked prefill” with decode so that multiple agents’ decode kernels can be dispatched as a single Metal kernel launch for merged batch tensors [2603.04428].

The same system extends batching across conversational phases through cross-phase context injection. If a new prompt `EXTEND`-matches the old prefix at character granularity, the runtime loads the old Q4 cache from disk, skips prefill for the overlapping prefix, quantizes only the new suffix tokens, and saves the extended state [2603.04428]. Because multi-phase prompts are templated to append rather than rewrite, `EXTEND` is almost always satisfied, giving up to $1.9$x TTFT improvements by Phase 5 in the prisoner’s dilemma scenario [2603.04428].

Other BatchQuantizedKVCache formulations differ mainly in where decompression occurs. CommVQ processes tokens in minibatches and shares the same $(qR_t)C_K^T$ across the batch; decoding is implemented in Triton as a fused bit-unpack + matrix-multiply across all tokens in a batch [2506.18879]. NQKV quantizes streaming appends but dequantizes the full cache for attention during decode [2505.16210]. VQKV reconstructs only the most recent $L_{local}$ vectors for each head on demand and then feeds $\hat K,\hat V$ into FlashAttention [2603.16435]. These are distinct execution strategies, but all preserve batched attention semantics over compressed cache state.

## 5. Empirical behavior and operating regimes

The edge-device multi-agent implementation reports that cache restoration reduces time-to-first-token by up to $136$x, with Gemma showing $22$--$136$x at $4$K--$32$K, DeepSeek $11$--$76$x at $4$K--$32$K, and Llama $24$--$111$x at $4$K--$16$K; at $1$K context, the same paper reports $3$--$10$x [2603.04428]. It also states that Q4 quantization fits $4$x more agent contexts into fixed device memory than FP16, and that perplexity measured with actual Q4 KV caches shows $-0.7\%$ for Gemma, $+2.8\%$ for Llama, and $+3.0\%$ for DeepSeek [2603.04428]. For two-agent warm-cache streaming, the reported system throughput is $19.6$ tok/s for Gemma, $52.6$ tok/s for DeepSeek, and $34.5$ tok/s for Llama [2603.04428].

Kitty reports that KV cache memory is cut by nearly $8$x with Kitty-Pro $(f=25\%)$ on Qwen3/LLaMA3, enabling up to $8$x larger batches and $2.1$x-$4.1$x higher throughput under the same memory budget [2511.18643]. CommVQ reports that 2-bit quantization reduces FP16 KV cache size by $87.5\%$, that 1-bit quantization is viable with minimal accuracy loss, that a LLaMA-3.1 8B model can run with a 128K context length on a single RTX 4090 GPU, that batch size at 32K grows from $8\to128$, and that the commutative reorder gives up to $9.6$x speedup over naive decode-then-attend [2506.18879]. NQKV reports that the OPT model can perform inference with an $2$x larger batch size or a $4$x longer context length, and that throughput improves by $9.3$x compared to when the KV cache is not used [2505.16210]. VQKV reports an $82.8\%$ compression ratio on LLaMA3.1-8B while retaining $98.6\%$ of the baseline performance on LongBench and enabling $4.3$x longer generation length on the same memory footprint [2603.16435].

These results indicate that BatchQuantizedKVCache implementations are evaluated along at least four axes: memory ratio, batch scaling, latency reduction, and fidelity under long-context or reasoning workloads. The preferred operating regime varies with the quantizer. Scalar low-bit schemes prioritize simplicity, mixed-precision layouts prioritize robustness, and vector-quantized schemes prioritize higher compression at acceptable reconstruction overhead.

## 6. Relation to adjacent KV-cache compression methods

A common misconception is that batched KV-cache compression is necessarily a quantization problem. Batch-Max provides a direct counterexample: it compresses the KV cache during both the input processing phase and the generation phase by eviction, not by bit-level quantization [2412.05693]. The paper states explicitly that it does not perform bit-level quantization of K/V tensors, that “compression” is purely by eviction, and that no $8\to4$-bit or other quant mapping is employed [2412.05693]. This distinction matters because BatchQuantizedKVCache systems are defined by their quantized representation and batched execution path, whereas eviction-based methods alter cache cardinality instead of numeric precision.

The broader low-bit KV-cache literature provides the algorithmic context in which BatchQuantizedKVCache designs emerged. RotateKV develops Outlier-Aware Rotation, Pre-RoPE Grouped-Head Rotation, and Attention-Sink-Aware Quantization, achieving less than $0.3$ perplexity degradation with 2-bit quantization on WikiText-2 using LLaMA-2-13B, less than $1.7\%$ degradation on GSM8K, a $3.97$x reduction in peak memory usage, support for $5.75$x larger batch sizes, and a $2.32$x speedup in decoding stage [2501.16383]. OTT, by contrast, uses channel-wise quantization for K, token-wise quantization for V, and a fixed-size outlier pool $O$ of capacity $N$ to exclude unusual tokens from quantization; it reports a $6.4$ times reduction in memory usage and a $2.3$ times increase in throughput under 2-bit quantization [2505.10938].

This suggests that BatchQuantizedKVCache should be situated within a wider design space rather than treated as a standalone algorithmic endpoint. Its distinguishing role is the systems interface between compressed KV representations and batched inference, whether the underlying compression is blockwise Q4, mixed-precision page quantization, additive vector quantization, quantile quantization, or residual vector quantization.

Source: https://www.emergentmind.com/topics/batchquantizedkvcache