Block-GTQ: RoPE-Aware KV-Cache Quantization

• Block-GTQ is a RoPE-aware key-cache quantization method that allocates bits based on the energy of two-dimensional frequency blocks.
• It exploits a block-wise rate-allocation strategy using an analytically motivated error proxy and a rigorously optimal greedy integer allocator.
• Empirical results demonstrate significant KV-cache compression and near-fp16 performance on long-context retrieval and reasoning tasks.

Block-GTQ is a RoPE-aware bit-allocation method for key-cache quantization in long-context transformer inference. It is designed for the setting in which KV-cache memory footprint and memory bandwidth dominate decoding latency and feasibility, and it departs from flat-vector key quantization by exploiting the fact that, under Rotary Positional Embeddings (RoPE), a key’s contribution to a future attention logit decomposes into a position-dependent sum over two-dimensional frequency blocks. In this formulation, key-cache compression becomes a block-wise rate-allocation problem: RoPE blocks with larger query-key energy are assigned more bits, while lower-impact blocks are assigned fewer bits. The method is built on TurboQuant-MSE (TQ-MSE), uses label-free calibration, and couples an analytically motivated error proxy with a greedy integer allocator that is proved optimal for the stated objective (Liang et al., 23 Jun 2026).

1. Problem setting and RoPE block structure

Block-GTQ is motivated by autoregressive decoding, where every new token streams cached keys and values for all previous tokens. In that regime, quantizing the KV cache reduces both memory footprint and HBM traffic, but the perturbation mechanisms for keys and values differ: values affect the post-softmax mixture, whereas keys perturb the pre-softmax logits seen by future queries. The method therefore focuses its non-uniform allocation on keys, while values are quantized uniformly with TQ-MSE in the reported deployment configurations (Liang et al., 23 Jun 2026).

The central structural observation is that RoPE changes the natural unit of key quantization. Let the head dimension be $d_h$ and define $L=d_h/2$. A query and key head are partitioned into $L$ two-dimensional RoPE blocks indexed by $m\in\{0,\dots,L-1\}$, each associated with frequency $\theta_m$. For query position $p_q$, key position $p_k$, and relative offset $\Delta p=p_q-p_k$, the RoPE-transformed query-key dot product decomposes as

$$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$

This decomposition is block-diagonal in the RoPE basis: each logit is a sum of independent two-dimensional block contributions modulated by $\cos(\theta_m\Delta p)$ and $L=d_h/2$0 (Liang et al., 23 Jun 2026).

The same block structure governs quantization error. If $L=d_h/2$1 is the decoded key and $L=d_h/2$2 is the error in block $L=d_h/2$3, then the induced logit perturbation satisfies

$L=d_h/2$4

because each $L=d_h/2$5 is a $L=d_h/2$6 rotation. There are no cross-block terms in this bound, and RoPE preserves each block’s $L=d_h/2$7 norm. This is the basis for treating key quantization as a block-wise allocation problem rather than a coordinate-wise or flat-vector problem (Liang et al., 23 Jun 2026).

2. Sensitivity scores and the rate-allocation objective

The ideal block sensitivity in Block-GTQ is the expected squared query-key energy

$L=d_h/2$8

The paper states that this quantity upper-bounds the expected squared logit error when combined with the block’s squared reconstruction error, but it also notes that direct estimation of $L=d_h/2$9 requires paired $L$0–$L$1 products and is noisy on short calibration. The method therefore uses a label-free AM–GM surrogate

$L$2

In grouped-query attention (GQA), for layer $L$3, KV head $L$4, RoPE block $L$5, query-head group $L$6, and calibration length $L$7, the instantiated score is

$L$8

The score is RoPE-invariant and depends only on marginal second moments of pre-RoPE queries and keys (Liang et al., 23 Jun 2026).

The local quantizer reused by Block-GTQ is TQ-MSE. For a nonzero vector $L$9, TQ-MSE normalizes by its radius, applies a shared orthogonal rotation, scalar-quantizes each rotated coordinate at $m\in\{0,\dots,L-1\}$0 bits, and rescales by the original radius. Its mean-squared reconstruction error obeys

$m\in\{0,\dots,L-1\}$1

with positive constant $m\in\{0,\dots,L-1\}$2. Under the paper’s block-error accounting, this leads to the weighted proxy

$m\in\{0,\dots,L-1\}$3

to be minimized over integer block bit-widths $m\in\{0,\dots,L-1\}$4 subject to a head-level budget (Liang et al., 23 Jun 2026).

The resulting optimization problem is

$m\in\{0,\dots,L-1\}$5

where $m\in\{0,\dots,L-1\}$6 is typically $m\in\{0,\dots,L-1\}$7 for average bit budget $m\in\{0,\dots,L-1\}$8. The concrete interpretation is that RoPE blocks with larger sensitivity scores should receive more bits because their contribution to future logits is larger (Liang et al., 23 Jun 2026).

3. Greedy allocation and calibration pipeline

Block-GTQ assigns bits by marginal gain. Increasing block $m\in\{0,\dots,L-1\}$9 from $\theta_m$0 to $\theta_m$1 reduces the objective by

$\theta_m$2

These gains are positive and decay geometrically with each additional bit on the same block. The paper proves that a simple greedy rule is exactly optimal for the stated integer program: repeatedly allocate the next bit to the block with the largest current marginal gain, subject only to the per-block upper bound. The proof is by exchange argument over “prefix-constrained” geometric gain sequences (Liang et al., 23 Jun 2026).

The implementation pipeline is layer-wise and head-wise. First, pre-RoPE queries and keys are collected on a short unlabeled calibration prefix, with the paper giving $\theta_m$3 WikiText-2 tokens as an example. Second, each head is split into $\theta_m$4 RoPE blocks and the scores $\theta_m$5 are computed. Third, the allocator chooses $\theta_m$6, $\theta_m$7, and total budget $\theta_m$8, initializes all blocks at $\theta_m$9, and greedily distributes the remaining budget. The algorithmic description uses a max-heap keyed by $p_q$0, which the paper notes gives the same ordering as the analytic marginal gain up to a constant. The resulting complexity is

$p_q$1

which is negligible for typical head sizes such as $p_q$2, where $p_q$3 (Liang et al., 23 Jun 2026).

A worked example in the paper considers one head with $p_q$4, hence $p_q$5, block energies $p_q$6, average budget $p_q$7, and bounds $p_q$8, $p_q$9. Starting from $p_k$0, the greedy updates allocate the four extra bits in the order $p_k$1, $p_k$2, $p_k$3, $p_k$4, yielding the final allocation $p_k$5. This illustrates the stated geometric trade-off: the highest-energy block receives repeated upgrades until its marginal gain falls below that of a lower-energy block still at low precision (Liang et al., 23 Jun 2026).

4. Quantizer materialization and packed-cache serving

After bit allocation, Block-GTQ groups blocks with the same assigned bit-width into “same-rate” segments. Post-RoPE key coordinates in each segment are concatenated and encoded with one TQ-MSE encoder at that segment’s rate. Values are not part of the RoPE logit decomposition and are therefore encoded with uniform TQ-MSE at the selected V budget. The reported experiments separate K-only diagnostics, where values remain in fp16 to isolate key effects, from deployment settings such as K3V3, K3V2, and K2V2, where both keys and values are quantized (Liang et al., 23 Jun 2026).

The serving path stores mixed-rate key segments and a uniform-rate packed value stream directly in HBM. The paper specifies nibble containers for segments at $p_k$6 bits, byte containers for higher-bit groups, per-segment codebooks, and per-token norms. The fused attention kernel then streams tiles of packed K/V, unpacks the segments, dequantizes through a small fp16 codebook that fits in L1, rescales by per-segment key norms and per-token value norms, computes $p_k$7 and $p_k$8 as fp16-input, fp32-accumulate tensor-core GEMMs with a fully fp32 online softmax, and never materializes a full fp16 KV cache in HBM; dequantized tiles are consumed immediately from registers or shared memory (Liang et al., 23 Jun 2026).

This systems design turns the mixed-rate allocation into concrete bandwidth and memory savings. At K3V3, the paper reports about $p_k$9 bytes per token per KV head for codes plus norms, compared with $\Delta p=p_q-p_k$0 bytes for an fp16 K/V pair, corresponding to approximately $\Delta p=p_q-p_k$1 KV-cache compression in practice. The text attributes the gap from the ideal $\Delta p=p_q-p_k$2 code-only rate to nibble or byte alignment and to per-segment norms and metadata. Calibration is entirely offline and amortized; after warmup, each layer-head allocation is fixed, no fp16 recent-key buffer is required, and a small Q-side rotation from TQ-MSE can be folded into $\Delta p=p_q-p_k$3 weights offline (Liang et al., 23 Jun 2026).

5. Empirical behavior and benchmark results

The paper evaluates Block-GTQ on a ten-model diagnostic panel and on long-context retrieval, understanding, and reasoning tasks. Under K-only quantization at $\Delta p=p_q-p_k$4 and $\Delta p=p_q-p_k$5 b/dim, Block-GTQ cuts mean per-layer RoPE-logit MAE by $\Delta p=p_q-p_k$6–$\Delta p=p_q-p_k$7 relative to uniform TQ-MSE and wins all $\Delta p=p_q-p_k$8 layer comparisons at each budget. Without an fp16 recent-key buffer, it also achieves the lowest mean softmax KL against fp16 and the highest top-10 attended-token overlap at every evaluated budget. The largest gains are reported at higher key budgets, where the allocator can route extra bits to the highest-energy RoPE blocks (Liang et al., 23 Jun 2026).

On long-context retrieval and understanding, the reported gains are similarly large. For Llama-3.1-8B-Instruct on the six-task NIAH benchmark, Block-GTQ at K2V2 raises the average from $\Delta p=p_q-p_k$9 with uniform TQ-MSE to $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$0, and at K3V3 or K3V2 it remains close to the fp16 ceiling of $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$1. On LongBench-EN with the same model, the average rises from $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$2 under uniform TQ-MSE to $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$3 at K2V2, while Block-GTQ remains closest to fp16 at all listed budgets, with $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$4 at K3V3/K3V2/K2V2 versus $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$5 for fp16. For Qwen2.5-7B-Instruct on NIAH, the paper reports that uniform TQ-MSE collapses to $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$6 at all tested budgets, whereas Block-GTQ attains $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$7 at K3V3/K3V2/K2V2 against $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$8 for fp16 (Liang et al., 23 Jun 2026).

The long-reasoning results highlight the same pattern. On AIME 2024/2025 with DeepSeek-R1-Distill-Qwen-7B, and without an fp16 recent-key buffer, Block-GTQ at K3V2 scores $$q'\!\cdot k' = \sum_{m=0}^{L-1} \Big[ (q_{2m}k_{2m}+q_{2m+1}k_{2m+1})\cos(\theta_m\Delta p) + (q_{2m+1}k_{2m}-q_{2m}k_{2m+1})\sin(\theta_m\Delta p) \Big].$$9, close to fp16’s $\cos(\theta_m\Delta p)$0, while uniform TQ-MSE collapses to $\cos(\theta_m\Delta p)$1. For DeepSeek-R1-Distill-Llama-8B, the corresponding numbers are $\cos(\theta_m\Delta p)$2 for Block-GTQ and $\cos(\theta_m\Delta p)$3 for fp16. The paper notes that PM-KVQ can lead on this backbone through joint K+V layer-wise allocation, which marks a boundary of the present K-focused design (Liang et al., 23 Jun 2026).

The systems benchmarks report both efficiency and feasibility gains. On a single H800 GPU with Qwen2.5-3B-Instruct at K3V3, the packed-cache path runs $\cos(\theta_m\Delta p)$4 faster than fp16 FlashAttention-2 at $\cos(\theta_m\Delta p)$5K context and reduces peak memory from $\cos(\theta_m\Delta p)$6 GB to $\cos(\theta_m\Delta p)$7 GB. At $\cos(\theta_m\Delta p)$8K and $\cos(\theta_m\Delta p)$9K context, fp16 is reported as OOM, while Block-GTQ remains feasible with peak memory $L=d_h/2$00 GB and $L=d_h/2$01 GB. Uniform TQ-MSE is about $L=d_h/2$02 faster at $L=d_h/2$03K and reaches about $L=d_h/2$04 KV compression, but the paper states that its perplexity collapses, whereas Block-GTQ preserves fp16-comparable quality and is therefore the deployable operating point (Liang et al., 23 Jun 2026).

6. Limitations, comparisons, and scope of the term

Block-GTQ allocates bits only on keys; values use uniform TQ-MSE. The paper explicitly lists this as a limitation, together with the assumption that per-block MSE decays as $L=d_h/2$05 and the requirement that short calibration capture the typical RoPE energy profile. Distribution shift can perturb the ranking of block sensitivities, especially at very low key budgets. The calibration analysis reports that with $L=d_h/2$06 tokens, allocation is stable at K3V3, with block Hamming distances across seeds of approximately $L=d_h/2$07–$L=d_h/2$08 and objective regret at most $L=d_h/2$09–$L=d_h/2$10, whereas K2V2 is more sensitive and the same ranking noise costs about $L=d_h/2$11 more in the rate objective. The paper therefore treats a V-side allocator, joint K+V optimization, dynamic per-request allocation, denser packing, and multi-GPU or batched serving for packed caches as future directions (Liang et al., 23 Jun 2026).

The method is compared primarily against uniform TQ-MSE and KIVI. Against uniform TQ-MSE, the reported conclusion is consistent across RoPE-logit MAE, attention KL, top-10 attended-token overlap, and downstream long-context tasks. Against KIVI-ScaleOnly, the paper states that per-channel quantization can be competitive when RoPE occupies only half of the key dimensions, as in partial-rotary or MLA settings, because Block-GTQ then has fewer RoPE blocks to differentiate. On fully RoPE’d GQA backbones, the paper reports that Block-GTQ usually wins. A plausible implication is that the method’s advantage is largest precisely when the inference stack exposes a strong RoPE block hierarchy (Liang et al., 23 Jun 2026).

The name should not be conflated with unrelated block-based constructions in other areas. The supplied literature also uses block-oriented formulations for quantitative group testing under noiseless, Gaussian, and Z-channel models (Li et al., 16 Jan 2026), for block floating point and scaled block floating point numerical formats (Soloveychik et al., 2022), for LDPC-based and spatially coupled quantitative group testing designs (Mashauri et al., 2024), and for block components of generalized quaternion group codes (Willenborg, 2 Jan 2025). Those topics are distinct from the method introduced as “Block-GTQ” in the KV-cache quantization setting.