BatchQuantizedKVCache: Batched Inference Engine

• BatchQuantizedKVCache is a runtime abstraction that integrates compressed key-value storage with batched execution to support efficient autoregressive attention.
• It employs 4-bit quantization and mixed-precision techniques to merge multiple agents’ KV caches, optimizing memory usage and reducing inference latency.
• Empirical evaluations demonstrate up to 136x speedup in cache restoration and a 72% reduction in memory footprint, improving throughput on resource-constrained devices.

BatchQuantizedKVCache denotes a batched key-value cache representation and execution layer in which multiple sequences or agents share a quantized KV-cache runtime. In a concrete system for multi-agent inference on edge devices, it is the in-memory engine that holds multiple agents’ KV caches in compact 4-bit (“Q4”) form, merges multiple agents’ Q4 blocks into a single batch tensor, runs fused_scaled_dot_product_attention on that batch, and splits results back out (Shkolnikov, 17 Feb 2026). Related work uses the same term for page-based mixed-precision caches, batched additive vector quantization, quantile-based block quantization, and batched residual vector quantization (Xia et al., 23 Nov 2025, Li et al., 23 Jun 2025, Cai et al., 22 May 2025, Wang et al., 17 Mar 2026). A plausible implication is that the term functions less as a single canonical algorithm than as an implementation-level abstraction for batched, compressed KV-cache execution.

1. Conceptual scope and variants

Across the recent literature, BatchQuantizedKVCache appears at the intersection of three requirements: compressed KV storage, batched execution, and direct compatibility with autoregressive attention. In the edge-device multi-agent setting, the motivation is explicit: device RAM is too small to hold every agent’s KV cache simultaneously, so quantized persistence and direct cache restoration are used to avoid redundant re-prefill computation (Shkolnikov, 17 Feb 2026). In high-throughput GPU settings, the same abstraction is adapted to mixed-precision page layouts, commutative codebooks, or vector-quantized indices that can be reconstructed or attended over efficiently (Xia et al., 23 Nov 2025, Li et al., 23 Jun 2025, Wang et al., 17 Mar 2026).

Variant	Representation	Batched execution
Agent Memory Below the Prompt	fixed-size 256-token Q4 blocks in safetensors	merge multiple agents’ Q4 blocks into a single batch tensor
Kitty	two unified 2-bit tensors plus metadata	replace the default KVCache with BatchQuantizedKVCache
CommVQ / NQKV / VQKV	packed bits or codebook indices	minibatches, fused bit-unpack, or on-demand reconstruction

The common denominator is not a single quantizer. Rather, it is the combination of compressed KV state with a runtime that can batch multiple cache instances without reverting to full-precision cache residency. This is particularly explicit in the multi-agent system, where the BatchQuantizedKVCache works hand-in-glove with disk persistence so that evicted caches can be reloaded in sub-second time rather than incur a full $O(n)$ prefill (Shkolnikov, 17 Feb 2026).

2. Data structures, storage layout, and cache lifecycle

In the persistent multi-agent design, the storage hierarchy is centered on a BlockPool that partitions every agent’s KV cache into fixed-size 256-token blocks, stored in safetensors format per-agent and per-layer (Shkolnikov, 17 Feb 2026). The core structures are KVBlock, AgentBlocks, ModelCacheSpec, and BlockPool. A KVBlock stores packed uint32 data together with per-group bfloat16 scales and biases, and each block becomes six tensors: packed data, scales, and biases for both K and V. Loading and saving are performed by mx.load_safetensors() and mx.save_safetensors() (Shkolnikov, 17 Feb 2026).

The same paper specifies a high-level cache lifecycle: save_agent_cache, load_agent_cache, and quantize_and_store. During prefill, layer outputs are sliced into 256-token blocks, quantized by Q4Quantizer::quantize, appended into the per-agent block list, and then serialized to disk (Shkolnikov, 17 Feb 2026). This organization makes the in-memory batch engine inseparable from the persistence layer.

Kitty adopts a different layout. Each mixed-precision Key page is decomposed into Tensor_low, which stores the low 2 bits of every channel, and Tensor_high, which stores the high 2 bits only for boosted channels, together with Boost_IDX metadata (Xia et al., 23 Nov 2025). The runtime also maintains full-precision Sink, a Local window, and a Q-Buffer. This page-centric design is intended to preserve coalescing and avoid divergence under dynamic 4-bit channel boosts (Xia et al., 23 Nov 2025).

For Q4 blockwise storage, the memory formulas are stated explicitly as

$$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$

and

$$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$

Assuming $G=64$, the ratio is

$Q4/FP16 = (1 + 8/64)/4 = 0.281,$

corresponding to a $72\%$ reduction (Shkolnikov, 17 Feb 2026).

3. Quantization formulations

The quantization schemes associated with BatchQuantizedKVCache differ substantially across systems.

In the Q4 persistent-cache system, blockwise quantization uses group size $G = 64$ tokens and computes

$$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$

followed by

$$\mathrm{packed\_data}[i] = \mathrm{round}((x[i]-\mathrm{bias})/\mathrm{scale})$$

clipped to $[-8\ldots +7]$ (Shkolnikov, 17 Feb 2026). This is a signed 4-bit blockwise scheme oriented toward persistence and direct reload.

Kitty uses Dynamic Channel-wise Precision Boost. For a page $$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$0, channel sensitivity is approximated by

$$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$1

after which the top $$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$2 fraction of channels are boosted to 4 bits while the remainder stay at 2 bits (Xia et al., 23 Nov 2025). Reconstruction combines the low and high parts through

$$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$3

and dequantization is

$$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$4

The paper summarizes the effective average precision as

$$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$5

with examples such as $$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$6 bits and $$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$7 bits (Xia et al., 23 Nov 2025).

CommVQ replaces scalar quantization with additive vector quantization. A lightweight encoder maps each token vector $$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$8 to a binary code

$$\mathrm{mem}_{FP16} = 2\,\mathrm{bytes}\cdot 2\cdot h\cdot d\cdot n$$9

and reconstruction is

$$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$0

The reduction rate is given by

$$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$1

so that $$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$2 corresponds to $$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$3-bit quantization and $$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$4 to $$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$5-bit quantization (Li et al., 23 Jun 2025). Its key technical refinement is a RoPE-commutative codebook, which enables reordered decoding inside self-attention rather than separate per-token decode-then-attend (Li et al., 23 Jun 2025).

NQKV adopts a per-block NormalFloat quantizer under the assumption that block elements follow a normal distribution. For block statistics $$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$6, quantization is

$$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$7

and dequantization is

$$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$8

The method is described as information-theoretically optimal under the normality assumption (Cai et al., 22 May 2025).

VQKV uses batched multi-stage residual vector quantization. Keys and values are flattened into matrices of shape $$\mathrm{mem}_{Q4} = \frac{1}{2}\,\mathrm{byte}\cdot h\cdot d\cdot n + 2\,\mathrm{bytes}\cdot 2\cdot\frac{h\cdot d\cdot n}{G}.$$9, quantized into codebook indices, and reconstructed on demand by

$G=64$0

with an analogous expression for $G=64$1 on the value side (Wang et al., 17 Mar 2026). Here the stored object is not a low-bit scalar tensor but a compact index tensor.

4. Batched execution and attention integration

The defining runtime feature of BatchQuantizedKVCache is batched execution over multiple cache instances. In the multi-agent Q4 system, the interface is organized around three primitive operations:

• merge(batch_blocks: Vec<AgentBlocks>) → (merged_K, merged_V, mask)
• update_and_fetch(merged_K, merged_V, mask) → (new_K, new_V, output_token_logits)
• extract(new_K, new_V) → Vec<AgentBlocks> (Shkolnikov, 17 Feb 2026)

The corresponding batched decode procedure first gathers each agent’s cached K,V blocks and pads to max length, then launches a single fused Q4 attention call on the GPU, then splits updated caches back to per-agent blocks (Shkolnikov, 17 Feb 2026). A single-threaded ConcurrentScheduler interleaves 512-token “chunked prefill” with decode so that multiple agents’ decode kernels can be dispatched as a single Metal kernel launch for merged batch tensors (Shkolnikov, 17 Feb 2026).

The same system extends batching across conversational phases through cross-phase context injection. If a new prompt EXTEND-matches the old prefix at character granularity, the runtime loads the old Q4 cache from disk, skips prefill for the overlapping prefix, quantizes only the new suffix tokens, and saves the extended state (Shkolnikov, 17 Feb 2026). Because multi-phase prompts are templated to append rather than rewrite, EXTEND is almost always satisfied, giving up to $G=64$2x TTFT improvements by Phase 5 in the prisoner’s dilemma scenario (Shkolnikov, 17 Feb 2026).

Other BatchQuantizedKVCache formulations differ mainly in where decompression occurs. CommVQ processes tokens in minibatches and shares the same $G=64$3 across the batch; decoding is implemented in Triton as a fused bit-unpack + matrix-multiply across all tokens in a batch (Li et al., 23 Jun 2025). NQKV quantizes streaming appends but dequantizes the full cache for attention during decode (Cai et al., 22 May 2025). VQKV reconstructs only the most recent $G=64$4 vectors for each head on demand and then feeds $G=64$5 into FlashAttention (Wang et al., 17 Mar 2026). These are distinct execution strategies, but all preserve batched attention semantics over compressed cache state.

5. Empirical behavior and operating regimes

The edge-device multi-agent implementation reports that cache restoration reduces time-to-first-token by up to $G=64$6x, with Gemma showing $G=64$7--$G=64$8x at $G=64$9K--$Q4/FP16 = (1 + 8/64)/4 = 0.281,$0K, DeepSeek $Q4/FP16 = (1 + 8/64)/4 = 0.281,$1--$Q4/FP16 = (1 + 8/64)/4 = 0.281,$2x at $Q4/FP16 = (1 + 8/64)/4 = 0.281,$3K--$Q4/FP16 = (1 + 8/64)/4 = 0.281,$4K, and Llama $Q4/FP16 = (1 + 8/64)/4 = 0.281,$5--$Q4/FP16 = (1 + 8/64)/4 = 0.281,$6x at $Q4/FP16 = (1 + 8/64)/4 = 0.281,$7K--$Q4/FP16 = (1 + 8/64)/4 = 0.281,$8K; at $Q4/FP16 = (1 + 8/64)/4 = 0.281,$9K context, the same paper reports $72\%$0--$72\%$1x (Shkolnikov, 17 Feb 2026). It also states that Q4 quantization fits $72\%$2x more agent contexts into fixed device memory than FP16, and that perplexity measured with actual Q4 KV caches shows $72\%$3 for Gemma, $72\%$4 for Llama, and $72\%$5 for DeepSeek (Shkolnikov, 17 Feb 2026). For two-agent warm-cache streaming, the reported system throughput is $72\%$6 tok/s for Gemma, $72\%$7 tok/s for DeepSeek, and $72\%$8 tok/s for Llama (Shkolnikov, 17 Feb 2026).

Kitty reports that KV cache memory is cut by nearly $72\%$9x with Kitty-Pro $G = 64$0 on Qwen3/LLaMA3, enabling up to $G = 64$1x larger batches and $G = 64$2x-$G = 64$3x higher throughput under the same memory budget (Xia et al., 23 Nov 2025). CommVQ reports that 2-bit quantization reduces FP16 KV cache size by $G = 64$4, that 1-bit quantization is viable with minimal accuracy loss, that a LLaMA-3.1 8B model can run with a 128K context length on a single RTX 4090 GPU, that batch size at 32K grows from $G = 64$5, and that the commutative reorder gives up to $G = 64$6x speedup over naive decode-then-attend (Li et al., 23 Jun 2025). NQKV reports that the OPT model can perform inference with an $G = 64$7x larger batch size or a $G = 64$8x longer context length, and that throughput improves by $G = 64$9x compared to when the KV cache is not used (Cai et al., 22 May 2025). VQKV reports an $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$0 compression ratio on LLaMA3.1-8B while retaining $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$1 of the baseline performance on LongBench and enabling $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$2x longer generation length on the same memory footprint (Wang et al., 17 Mar 2026).

These results indicate that BatchQuantizedKVCache implementations are evaluated along at least four axes: memory ratio, batch scaling, latency reduction, and fidelity under long-context or reasoning workloads. The preferred operating regime varies with the quantizer. Scalar low-bit schemes prioritize simplicity, mixed-precision layouts prioritize robustness, and vector-quantized schemes prioritize higher compression at acceptable reconstruction overhead.

6. Relation to adjacent KV-cache compression methods

A common misconception is that batched KV-cache compression is necessarily a quantization problem. Batch-Max provides a direct counterexample: it compresses the KV cache during both the input processing phase and the generation phase by eviction, not by bit-level quantization (Metel et al., 2024). The paper states explicitly that it does not perform bit-level quantization of K/V tensors, that “compression” is purely by eviction, and that no $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$3-bit or other quant mapping is employed (Metel et al., 2024). This distinction matters because BatchQuantizedKVCache systems are defined by their quantized representation and batched execution path, whereas eviction-based methods alter cache cardinality instead of numeric precision.

The broader low-bit KV-cache literature provides the algorithmic context in which BatchQuantizedKVCache designs emerged. RotateKV develops Outlier-Aware Rotation, Pre-RoPE Grouped-Head Rotation, and Attention-Sink-Aware Quantization, achieving less than $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$4 perplexity degradation with 2-bit quantization on WikiText-2 using LLaMA-2-13B, less than $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$5 degradation on GSM8K, a $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$6x reduction in peak memory usage, support for $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$7x larger batch sizes, and a $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$8x speedup in decoding stage (Su et al., 25 Jan 2025). OTT, by contrast, uses channel-wise quantization for K, token-wise quantization for V, and a fixed-size outlier pool $$\mathrm{scale} = \frac{\max - \min}{15}, \qquad \mathrm{bias} = 0.5\cdot(\max+\min),$$9 of capacity $$\mathrm{packed\_data}[i] = \mathrm{round}((x[i]-\mathrm{bias})/\mathrm{scale})$$0 to exclude unusual tokens from quantization; it reports a $$\mathrm{packed\_data}[i] = \mathrm{round}((x[i]-\mathrm{bias})/\mathrm{scale})$$1 times reduction in memory usage and a $$\mathrm{packed\_data}[i] = \mathrm{round}((x[i]-\mathrm{bias})/\mathrm{scale})$$2 times increase in throughput under 2-bit quantization (Su et al., 16 May 2025).

This suggests that BatchQuantizedKVCache should be situated within a wider design space rather than treated as a standalone algorithmic endpoint. Its distinguishing role is the systems interface between compressed KV representations and batched inference, whether the underlying compression is blockwise Q4, mixed-precision page quantization, additive vector quantization, quantile quantization, or residual vector quantization.