XQuant: Efficient LLM Inference

• XQuant is an algorithmic framework that mitigates memory bottlenecks in LLM inference by leveraging low-bit quantization of activations and on-the-fly KV rematerialization.
• It employs uniform quantization and cross-layer delta compression, achieving up to 10x memory savings with minimal perplexity degradation.
• The method trades extra computation for reduced memory usage, aligning inference performance with modern hardware trends and large-scale deployments.

XQuant is an algorithmic framework that addresses the memory bottleneck during LLM inference by leveraging low-bit quantization of layer input activations (denoted X), together with on-the-fly rematerialization of the Key and Value (KV) caches required for transformer self-attention. The method is designed with the goal of aligning inference efficiency with modern hardware trends, where compute resources increasingly outpace both memory capacity and bandwidth. XQuant achieves substantial reductions in memory consumption while maintaining near–FP16 accuracy and exploits intrinsic cross-layer similarities to enable even more aggressive compression.

1. Algorithmic Strategy: Quantized Caching and Rematerialization

XQuant fundamentally departs from standard transformer inference, which materializes, quantizes, and stores the full sequence of Key and Value tensors (KV cache) for each layer. These are high-dimensional and memory intensive, especially in low-latency scenarios or during long context inference.

Instead, XQuant stores a quantized version of the post-layer normalization input activations for each layer, $\hat{X}_i = Q(X_i)$, with $Q$ a uniform low-bit quantizer (typically 2, 3, or 4 bits per value). This halves the memory footprint immediately since traditionally both KV caches are retained. During inference, whenever the current Key or Value tensor is needed for self-attention, it is computed as: $K = X \cdot W_K, \qquad V = X \cdot W_V$ using the quantized input $\hat{X}_i$. The rematerialization process trades increased computation (matrix multiplication with $W_K$, $W_V$ per step) for a drastic reduction in DRAM reads and cache pressure.

2. Quantization and Cross-Layer Delta Compression

The quantization operator $Q(\cdot)$ is implemented as a uniform quantizer, with configurations ranging from 2 to 4 bits. In the default mode, the full activation $X_i$ is quantized layerwise and cached. In the enhanced XQuant-CL mode (“CL” for cross-layer), the method exploits an empirical observation: after layer normalization, successive $X$ activations are highly similar across layers. Accordingly, only the difference (delta) $\Delta X_i = X_i - X_{i-1}$ is quantized: $\widehat{\Delta X}_i = Q(\Delta X_i)$ with accumulation during inference: $$\hat{X}_i = X_0 + \sum_{j=1}^i \widehat{\Delta X}_j$$ Because $\Delta X_i$ is of smaller amplitude than $X_i$ itself, this approach supports more aggressive quantization and thus amplifies memory savings.

3. Empirical Results: Memory Savings and Accuracy Trade-offs

Extensive evaluation on a range of models (including Llama-3.1-8B and Mistral-7B with grouped query attention) demonstrates that XQuant achieves up to $7.7\times$ memory reduction relative to FP16 baselines for activation caching, with perplexity degradation less than $0.1$. The XQuant-CL variant, leveraging delta compression, pushes savings further up to $10\times$ with just $0.01$ perplexity loss, and $12.5\times$ with only $0.1$ perplexity loss.

Performance metrics were measured by calculating the memory footprint of activation and/or KV caches and recording test perplexity on methods using WikiText-2 and C4 datasets. When compared with state-of-the-art KV cache quantization schemes (e.g., KVQuant, KIVI*), XQuant delivers improved memory savings and comparable or superior accuracy—achieved with simple uniform quantization and without complex calibration.

4. Rematerialization Computation and Hardware Considerations

The computational overhead incurred by rematerializing $K$ and $V$ at inference time is balanced against reductions in memory access and bandwidth. Since LLM inference is typically memory-bound on modern accelerators, XQuant’s approach matches the trend of abundant compute resources versus stagnant memory bandwidth. For further efficiency with grouped-query attention (GQA), XQuant employs an offline SVD to project $X$ into a lower-dimensional latent space, which reduces both computation and quantization error.

This compute-for-memory tradeoff is particularly favorable on current and anticipated accelerators (GPUs, custom hardware) that deliver ample FLOPs per byte of DRAM transfer, making XQuant pragmatic for large-scale deployments and edge inference.

5. Cross-Layer Similarity and Delta Accumulation: Detailed Mechanism

The key empirical property exploited in XQuant-CL is that post-normalization layer input embeddings $X_i$ exhibit high cosine similarity across adjacent layers. Quantizing only the delta $\Delta X_i$ per layer allows:

• Significantly reduced quantization dynamic range (enabling more aggressive bitwidth reduction).
• Simpler accumulator logic during evaluation: the current input is always reconstructible as a running sum from a small initial vector $X_0$ plus quantized deltas. This selective quantization does require careful handling in early layers (where $X_1$ and $X_0$ are less similar), but the method achieves well below $0.1$ perplexity degradation in practice across a broad suite of models.

6. Comparative Analysis and Limitations

When compared to other cutting-edge KV cache quantization methods, XQuant’s main strengths are memory efficiency, retention of near–FP16 accuracy, and conceptual simplicity—owing to uniform quantization and on-the-fly rematerialization rather than customized non-uniform quantization or calibration. A possible limitation, relevant for hardware without compute overprovisioning, is the increased per-token matrix multiplication overhead. Additionally, per-layer configuration (bitwidth selection and delta handling) must be tuned for models where activation distribution varies more widely.

7. Implications and Outlook

XQuant’s compute-for-memory paradigm, and its cross-layer extension, represent a shift in LLM inference design where memory operations are explicitly traded for arithmetic operations, rationalized by modern hardware trends. For entities deploying LLM-based systems at scale, this approach enables operation within strict memory or bandwidth envelopes that were previously unattainable, all while maintaining negligible accuracy loss. As compute/memory imbalance continues to widen, the core strategies of XQuant are likely to assume increasing importance in scalable, efficient transformer inference pipelines.

For XQuant's specific technical details, relevant loss curves, and allocation figures, as well as further experimental exploration on downstream tasks, see (Tomar et al., 14 Aug 2025).