XQuant-CL: Memory-Efficient LLM Inference

• XQuant-CL is a memory-efficient technique that rematerializes quantized layer activations to reconstruct key and value tensors on-the-fly during LLM inference.
• It leverages cross-layer delta compression by storing only the differences between successive activations, significantly reducing both storage and memory bandwidth.
• Empirical evaluations show up to 12.5× memory savings with negligible perplexity degradation, outperforming existing KV caching quantization schemes.

XQuant-CL is a memory-efficient technique for accelerating LLM inference by rematerializing key-value (KV) tensors from quantized layer input activations, further exploiting cross-layer similarity through delta compression of successive activations. It achieves up to 12.5× reduction in memory usage with negligible (≤0.1) perplexity degradation and surpasses leading KV caching quantization approaches.

1. Rematerialization via Quantized Layer Inputs

The foundational principle of XQuant-CL is to store quantized input activations $X$ of each transformer layer rather than the conventional approach of caching the separate key ($K$) and value ($V$) tensors. During inference, when attention computation requires the $K$ and $V$ representations, they are reconstructed ("rematerialized") on-the-fly by multiplying the cached and quantized $X$ tensor by the corresponding learned projection matrices ($W_K$ and $W_V$): $[K \mid V] \approx \hat{X} \cdot [W_K \mid W_V]$ This process avoids direct quantization of the potentially higher dynamic range $K$/$V$ tensors and effectively trades additional matrix multiplications for a substantial reduction of memory access and storage.

The approach leverages increasing compute-to-memory ratios of modern hardware, enabling inference workloads to withstand additional compute overhead while minimizing the performance penalty incurred by memory constraints (Tomar et al., 14 Aug 2025).

2. Cross-Layer Similarity and Delta Compression

Transformers employ residual connections, resulting in high correlation of $X$ activations across adjacent layers. XQuant-CL capitalizes on this by storing only the quantized deltas ($\Delta \hat{X}_j$) between successive layer activations rather than the full $X$ tensor for each layer. The activation for layer $i$ is reconstructed as: $$\hat{X}_{(i)} = X_0 + \sum_{j=1}^{i} \Delta \hat{X}_j$$ where $X_0$ is the initial activation and each $\Delta \hat{X}_j$ is quantized at extremely low bit-width (as low as 2-3 bits per element).

Because the deltas have much smaller variance than the absolute activations—owing to the incremental nature of residual updates—quantization error is significantly mitigated, permitting more aggressive compression without substantial degradation in accuracy.

3. Memory Footprint and Compression Ratio

The XQuant-CL strategy results in immediate memory savings by converting the standard practice of caching two separate matrices ($K$ and $V$) per token per layer into a single (quantized) $X$ tensor. In addition, the delta-based cross-layer compression multiplies the effect:

• With 3-bit quantization, XQuant-CL attains up to $10\times$ memory savings relative to an FP16 KV cache, with only $0.01$ perplexity degradation.
• In 2-bit quantization scenarios, up to $12.5\times$ reduction is reported, at the cost of approximately $0.1$ perplexity degradation.

This approach empirically outperforms prior KV caching quantization schemes (e.g., KVQuant, KIVI*) across a range of LLM architectures, yielding lower error for equivalent memory budgets (Tomar et al., 14 Aug 2025).

Quantization Precision	Memory Savings (vs FP16 KV Cache)	Typical Perplexity Degradation
3 bits	$10\times$	0.01
2 bits	$12.5\times$	0.1

4. Perplexity and Model Fidelity

Despite the aggressive compression, XQuant-CL maintains near-FP16 accuracy for modern LLM inference. Systematic evaluations demonstrate:

• Perplexity increases of less than $0.1$ compared to the FP16 baseline, even at ultra-low bitwidths.
• The delta quantization mechanism effectively preserves predictive power, as the essential token-context relationships encoded in attention calculations still receive high-fidelity input vectors upon rematerialization.

A plausible implication is that the error introduced by quantizing the deltas is largely absorbed by the model's inherent tolerance to minor deviations in its internal representations, given their constrained dynamic range.

5. Technical Details: Quantization and SVD for GQA Models

For grouped query attention (GQA) architectures, the key and value projection matrices ($W_K$, $W_V$) typically have large output dimensionality. The XQuant framework factorizes these projections via singular value decomposition (SVD): $W_K = U_K \Sigma_K B_K^T$ By down-projecting $X$ with $U_K$, the resulting activations $X U_K$ can be quantized with per-channel precision. Notably, the first channel (largest singular value) often features outlier behavior and is retained at full precision (FP16), while the other channels are aggressively quantized. This hybrid scheme further reduces memory requirements and preserves accuracy through targeted precision allocation.

6. Comparative Evaluation Against Other Quantization Schemes

XQuant-CL distinguishes itself from contemporaneous methods such as CLAQ (Wang et al., 27 May 2024), QuantX (Mazher et al., 12 May 2025), and other KV cache quantization approaches along several dimensions:

• Uniform Quantization Simplicity: XQuant-CL generally utilizes uniform quantization per delta, differentiating it from more complex centroid selection or outlier-handling schemes.
• Rematerialization Efficiency: Caching only quantized $X$ (and/or cross-layer deltas) reduces both storage and memory bandwidth.
• No Need for Layer-Specific Calibration: Unlike CLAQ, which adopts K-means centroids and adaptive precision, XQuant-CL's compression leverages intrinsic architectural properties (cross-layer similarity), reducing engineering complexity.
• Performance Superiority at Low Bits: In evaluations, XQuant-CL achieves lower perplexity at similar or smaller memory footprint than established state-of-the-art approaches, even when using simple uniform quantization.

7. Implications for Scalable LLM Inference

XQuant-CL directly addresses the growing disparity between compute and memory bandwidth in LLM deployments, which is exacerbated by large context lengths and deep transformer stacks. By compressing KV caches through quantized and cross-layer delta-based mechanisms, it enables:

• Efficient inference for long-context LLMs on GPUs and edge devices with fixed memory constraints.
• Trade-off of increased (but hardware-amortized) matrix multiplications for reduced memory movement, boosting overall throughput.
• Deployment of models with larger context windows and deeper architectures without memory-related bottlenecks.

This suggests that XQuant-CL is particularly advantageous for practical settings demanding minimized latency and maximum context length support.

8. Limitations and Forward Directions

While XQuant-CL's methodology exhibits strong empirical benefits, current implementations report results using uniform quantization and delta compression. It remains plausible that combining per-channel or hybrid centroids adopted in frameworks like CLAQ (Wang et al., 27 May 2024) and QuantX (Mazher et al., 12 May 2025) could further boost accuracy, especially for architectures with more heterogeneous activation statistics.

Potential limitations include:

• Error accumulation in scenarios where successive activation deltas are non-trivial.
• Hardware constraints might bound the throughput of rematerialization steps on platforms where matrix multiplication is not sufficiently accelerated.

Continued research will clarify integration points and the best practices for combining cross-layer compression with adaptive quantization techniques.

---

In summary, XQuant-CL employs quantization and cross-layer delta accumulation of transformer input activations for rematerialized KV cache construction, yielding order-of-magnitude memory reduction with minimal accuracy loss. Its design is well-adapted to contemporary hardware and provides a scalable solution for LLM inference under strict memory budgets (Tomar et al., 14 Aug 2025).