RaBitQCache: Sparse KVCache for Long-Context LLMs

• RaBitQCache is a sparse attention framework that mitigates the KV cache bottleneck in long-context LLM inference by using rotated binary quantization for efficient token representation.
• It employs an unbiased proxy score and adaptive Top-p retrieval, leveraging a custom binary-INT4 GEMV kernel to dynamically adjust the token budget based on attention sparsity.
• The design integrates hardware-aware optimizations, such as concurrent index construction and lazy KV-cache updates, to achieve significant speedup with minimal quality loss.

RaBitQCache is a sparse attention framework for long-context LLM inference that targets the Key-Value (KV) cache bottleneck by combining randomized rotated binary quantization, high-throughput binary-INT4 arithmetic, adaptive Top-p retrieval, and a hardware-aware execution design. It is introduced in "RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference" (Li et al., 30 Jun 2026), where the method is positioned against sparse attention schemes that use static fixed-budget Top-k retrieval or computationally expensive and biased proxy scores. The framework uses a proxy score that serves as an unbiased estimator with a proven error bound, enabling adaptive retrieval based on actual attention sparsity rather than a fixed token budget.

1. Problem setting and architectural role

RaBitQCache addresses long-context inference in transformer LLMs, where the KV cache becomes the dominant systems bottleneck as sequence length grows. The central objective is not to remove attention computation entirely, but to avoid full retrieval of cached keys and values by estimating attention weights cheaply enough to decide which tokens merit full-precision access.

The method is organized around hybrid attention. Cached tokens are represented by a compact index derived from rotated binary quantization, while the actual full-precision $K,V$ tensors remain available for the subset selected during decoding. In the decode path, the system first computes a proxy score over the indexed cache, then performs adaptive Top-p selection, and finally fetches full-precision $K,V$ for the selected set together with a local window of recent tokens. This division is important: the quantized representation is used for retrieval, not as a full replacement for final attention computation (Li et al., 30 Jun 2026).

A common misconception is to treat RaBitQCache as merely another fixed-budget sparse attention method. The paper instead states that fixed $k$ can under-select or over-select when attention mass concentrates unevenly across heads and layers, and motivates Top-p as the mechanism for dynamically adjusting the token budget. This suggests that the framework is intended to track variation in attention sparsity rather than enforce a constant retrieval cardinality.

2. Rotated binary quantization and score approximation

The quantization pipeline begins with re-centering and normalization. For each Key $k$ and Query $q$, centroids $C_k, C_q$ are computed over the prefill phase, and centered unit vectors are formed as

$$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$

A random orthogonal rotation $P \in \mathbb{R}^{D \times D}$ is drawn once at model initialization from the Haar distribution, with $PP^T = I$. Keys are rotated as

$k' = P^T k_c.$

The 1-bit quantization operator then maps the rotated key to a binary code:

$K,V$0

where $K,V$1 if $K,V$2 and $K,V$3 otherwise. The associated codeword is reconstructed as

$K,V$4

and the paper states the equivalent optimization form

$K,V$5

Queries are treated asymmetrically. After rotation, $K,V$6, each dimension is uniformly quantized into 4-bit integers:

$K,V$7

with reconstruction $K,V$8.

This asymmetry enables binary-to-INT4 inner-product estimation. The paper writes

$K,V$9

Accordingly, a single custom GEMV kernel on GPU fuses popcount and INT4 multiply-accumulate, and the paper reports this yields more than $k$0 speedup over naïve INT4$k$1FP compute (Li et al., 30 Jun 2026).

The use of centroids is not an incidental preprocessing step. An ablation reported later shows that omitting centroid re-centering drops LongBench average generation score from $k$2 to $k$3, which the paper describes as validating its theoretical role.

3. Proxy-score theory and estimator guarantees

RaBitQCache reduces the attention estimation problem to the centered, normalized inner product. For one head, the full attention score is written as

$k$4

Only $k$5 varies across cached keys; the remaining terms are precomputed.

To estimate that varying term, the paper defines the rotated binary code $k$6 and a correction factor

$k$7

With $k$8, the proxy score is

$k$9

The stated theorem is

$k$0

The proof sketch decomposes $k$1 in a basis $k$2 so that $k$3 can be related to $k$4 plus orthogonal noise, and then uses randomness in $k$5 to make the noise term zero-mean.

The paper also gives a JL-style tail bound. With high probability over $k$6,

$k$7

and more precisely

$k$8

The authors explicitly connect this bound to retrieval policy design: it allows the proxy score to be trusted for magnitude, not just ranking, which is what enables Top-p retrieval rather than only Top-k (Li et al., 30 Jun 2026).

This theoretical framing distinguishes RaBitQCache from proxy-score methods described in the abstract as computationally expensive and biased. The claim is not merely that the estimator is useful empirically, but that its use in sparse attention is justified by unbiasedness and concentration.

4. Adaptive Top-p retrieval and decode integration

RaBitQCache uses Top-p rather than fixed Top-k retrieval. The paper defines Top-p as choosing the minimal set $k$9 such that

$q$0

The stated rationale is that fixed $q$1 can under-select or over-select when attention mass is distributed unevenly across heads and layers.

The Top-p kernel is formulated as an $q$2-time, $q$3-extra-memory algorithm that avoids sorting. It scans the scores to build partial sums while using a ternary-search-style thresholding procedure. The paper emphasizes that the absence of sorting avoids the $q$4 cost of explicit ranking.

In decode, the method is integrated as a five-stage procedure:

1. Rotate and INT4-quantize the query to obtain $q$5.
2. Run the binary-INT4 GEMV kernel to compute $q$6.
3. Normalize to $q$7.
4. Run TopP$q$8 to obtain $q$9.
5. Fetch full-precision $C_k, C_q$0 for $C_k, C_q$1 local window and apply hybrid attention.

The significance of this design is that adaptivity happens at the retrieval stage rather than through hand-tuned, layerwise budgets. A plausible implication is that the method can respond to heterogeneous sparsity patterns without requiring a separate budget schedule for each head or context length, though the paper states this only indirectly through the Top-p motivation and empirical token-count variation.

5. Hardware-aware system design

The systems component of RaBitQCache is built to keep index construction and retrieval overhead below the latency savings from sparse attention. In prefill, the main CUDA stream runs dense attention with $C_k, C_q$2 cost, while a low-priority stream concurrently computes $C_k, C_q$3, the sign codes, and $C_k, C_q$4 in $C_k, C_q$5. The paper states that index building is hidden behind the $C_k, C_q$6 prefill cost, producing zero visible prefill overhead and quantifying the overhead as less than $C_k, C_q$7 (Li et al., 30 Jun 2026).

During decode, the system uses lazy KV-cache updates. New tokens are buffered in a small full-precision local window $C_k, C_q$8 and are not quantized immediately. Instead, they are batch-quantized once the local window is full. Attention over $C_k, C_q$9 is used to preserve near-zero delay on the most recent context while offloading index updates.

The kernel-level implementation contains several explicit bandwidth and throughput optimizations:

Component	Design choice	Stated effect
Binary storage	32 bits mapped to one 32-bit word	$$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$0 memory bandwidth reduction
Query handling	Shared-memory tiling for INT4 query	Broadcast $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$1 once per block
Low-level arithmetic	Vectorized bit-extract plus popcount with 128-bit loads and loop unrolling	Higher throughput
Retrieval kernel	Custom fused Top-p kernel	Avoids sorts via parallel ternary search

The implementation stack reported in the paper consists of vLLM v0.10.2, FlashInfer for kernels, and LMCache for memory management, evaluated on NVIDIA H100/Hopper GPUs. These details place RaBitQCache within an LLM-serving context rather than a standalone algorithmic prototype.

6. Empirical behavior, baselines, and relation to RaBitQ

The empirical evaluation spans LongBench, RULER, and GSM8K, with Longchat-7B-32k, LLaMA-3.1-8B, and LLaMA-3.1-70B across contexts from $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$2K to $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$3K, with LLaMA-3.1-70B evaluated up to $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$4K. On LLaMA-3.1-8B, LongBench average generation scores are reported as follows: Full $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$5 gives $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$6 with recall $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$7; Oracle $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$8 gives $$q_c = \frac{q - C_q}{\|q - C_q\|_2}, \qquad k_c = \frac{k - C_k}{\|k - C_k\|_2}.$$9 with recall $P \in \mathbb{R}^{D \times D}$0; RaBitQCache with $P \in \mathbb{R}^{D \times D}$1 and $P \in \mathbb{R}^{D \times D}$2 tokens gives $P \in \mathbb{R}^{D \times D}$3 with recall $P \in \mathbb{R}^{D \times D}$4; Quest-4096 with $P \in \mathbb{R}^{D \times D}$5 gives $P \in \mathbb{R}^{D \times D}$6 with recall $P \in \mathbb{R}^{D \times D}$7; SparQ-$P \in \mathbb{R}^{D \times D}$8 gives $P \in \mathbb{R}^{D \times D}$9 with recall $PP^T = I$0; and DS-$PP^T = I$1 gives $PP^T = I$2 with recall $PP^T = I$3. On LLaMA-3.1-70B, LongBench average is $PP^T = I$4 for Full, $PP^T = I$5 for RaBitQCache with $PP^T = I$6, $PP^T = I$7 for Quest-1024, and $PP^T = I$8 for SparQ-$PP^T = I$9 (Li et al., 30 Jun 2026).

On RULER at $k' = P^T k_c.$0K-$k' = P^T k_c.$1K for LLaMA-8B, the paper states that RaBitQCache matches or exceeds Full at $k' = P^T k_c.$2K-$k' = P^T k_c.$3K while spending approximately $k' = P^T k_c.$4 tokens adaptively. On GSM8K, RaBitQCache reaches accuracy $k' = P^T k_c.$5 versus Full $k' = P^T k_c.$6, with recall $k' = P^T k_c.$7 versus $k' = P^T k_c.$8.

Latency and throughput measurements are equally central. The reported prefill time-to-first-token overhead is less than $k' = P^T k_c.$9 versus FlashAttention-2, decode token-by-token speedup is $K,V$00 at $K,V$01K and $K,V$02 at $K,V$03K tokens, and end-to-end speedup is $K,V$04 versus full-precision LLM serving. For memory, the paper gives FP16 KVCache size as $K,V$05 bytes, while the RaBitQCache index requires

$K,V$06

It also states that full $K,V$07 can be offloaded to host if desired.

The main ablations reinforce three design choices. First, threshold sensitivity over $K,V$08 yields a smooth trade-off between generation quality and token count. Second, removing centroid re-centering reduces LongBench average from $K,V$09 to $K,V$10. Third, the custom INT4$K,V$11Binary kernel gives $K,V$12-$K,V$13 speedup over naïve TEINT4 GEMV.

RaBitQCache is explicitly related to the earlier RaBitQ quantization method, which introduced randomized quantization of $K,V$14-dimensional vectors into $K,V$15-bit strings with an unbiased estimator and an $K,V$16 high-probability error bound for approximate nearest neighbor search (Gao et al., 2024). The connection is methodological rather than application-identical: RaBitQ was developed for ANN in high-dimensional Euclidean space, with bitwise-popcount and SIMD-based implementations, whereas RaBitQCache adapts the rotated binary quantization idea to KV-cache retrieval in long-context LLM inference. This suggests a transfer of quantization theory from ANN indexing to sparse attention, with the proxy-score guarantee preserved in a different systems setting.