SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

Abstract: KV-cache memory is a major bottleneck in real-world LLM serving, where systems must simultaneously support latency-sensitive small-batch requests and high-throughput concurrent workloads. Although many KV-cache compression methods improve offline accuracy or compression ratio, they often violate practical serving constraints such as paged memory layouts, regular memory access, and fused attention execution, limiting their effectiveness in deployment. In this work, we identify the minimal set of 4-bit KV-cache quantization methods that remain viable under these constraints. Our central finding is that a simple design--token-wise INT4 quantization with block-diagonal Hadamard rotation--consistently achieves the best accuracy-efficiency trade-off. Across multiple models and benchmarks, this approach recovers nearly all of the accuracy lost by naive INT4, while more complex methods such as vector quantization and Hessian-aware quantization provide only marginal additional gains once serving compatibility is taken into account. To make this practical, we implement a fused rotation-quantization kernel that integrates directly into paged KV-cache layouts and introduces zero measurable end-to-end overhead, matching plain INT4 throughput across concurrency levels. Our results show that effective KV-cache compression is fundamentally a systems co-design problem: under real serving constraints, lightweight block-diagonal Hadamard rotation is a viable method that delivers near-lossless accuracy without sacrificing serving efficiency.

• The paper presents a system-aware INT4 KV-cache quantization method using block-diagonal Hadamard rotation to mitigate channel-wise outliers.
• It achieves 4× compression while maintaining throughput and nearly matching BF16 accuracy in real-world LLM serving environments.
• Empirical evaluations across varied benchmarks confirm that the integrated rotation approach eliminates extra data movement and preserves serving efficiency.

SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

Motivation and Problem Definition

Modern LLM inference workloads increasingly demand ultra-long context windows, driving exponential growth in key-value (KV) cache memory requirements. In production environments, KV cache size is often the primary throughput constraint, limiting concurrent request capacity and operational efficiency. Prevailing compression techniques—ranging from token eviction, channel-quantized methods, vector quantization, to low-rank projections—show promising offline accuracy but typically violate fundamental serving constraints, such as paged memory architectures, uniform access patterns, and fused attention kernels. As a result, these techniques yield limited practical gains and nontrivial serving stack integration complexity.

Figure 1: Overview of the system-aware INT4 KV-cache quantization framework and rotate-quantize pipeline, conforming to memory and kernel constraints in LLM serving.

System Constraints and Quantization Design Space

Practical KV cache compression must align with two primary constraints: paged memory layouts and kernel compatibility for fused decoding attention operations. Paged memory layouts, as used in vLLM and TensorRT-LLM, require uniform precision within memory blocks and prohibit fragmented or mixed-precision caches, rendering token eviction and channel-wise scaling largely infeasible. FlashAttention kernels demand contiguous, coalesced access and prohibit data-dependent lookups or strided memory access, invalidating most codebook-based vector quantization methods.

These serving realities essentially restrict feasible compression to token-wise quantization schemes. The main challenge then becomes maximizing accuracy while retaining system-level throughput and latency guarantees.

Quantization Approaches and Rotation Intuition

Token-wise asymmetric INT4 quantization is highly system-friendly, offering $4\times$ compression of KV-cache with preserved tensor layout. However, naive INT4 quantization is severely impacted by channel-wise range heterogeneity, especially due to activation outliers in keys. This results in high quantization error and catastrophic accuracy collapse (all zero scores in benchmark evaluations).

Block-diagonal Hadamard rotation is introduced as a pre-quantization orthonormal transform to mitigate channel-wise outliers. By redistributing energy across head dimensions and smoothing variance, rotation suppresses outlier-induced errors and largely restores accuracy. Applying the rotation in block-diagonal form enables efficient kernel fusion and parallelism, allowing selection of block size for system-optimal trade-offs between quantization quality and compute throughput.

Vector quantization via offline k-means is empirically evaluated, showing incremental improvement as codebook size increases, but saturating below the best block-diagonal rotation performance. Hessian-aware quantization, using layerwise query statistics to learn orthogonal eigenvectors, is implemented but introduces calibration complexity and only marginal accuracy gains relative to rotation alone.

Empirical Evaluation: Accuracy and Efficiency

Comprehensive evaluation on Qwen3-4B, Qwen3-8B, Qwen3-32B, and GLM-4.7-FP8 across five reasoning/coding benchmarks (GPQA, HumanEval, LiveCodeBench, AIME25, MATH500) demonstrates several strong numerical results:

• Naive INT4 quantization catastrophically collapses accuracy (all metrics zero on Qwen3), rendering it unusable.
• Block-diagonal rotation (BDR128) recovers nearly all lost accuracy, with mean scores just 1–3 points below BF16.
• For more robust models (GLM-4.7-FP8), even naive INT4 remains competitive, but BDR yields marginal further gains.
• More complex quantization (vector quantization with up to 2048 centroids, Hessian-aware variants) provides only small additional benefit but incurs significant calibration and deployment overhead.

Serving Throughput and System-Level Performance

Kernel microbenchmarks reveal that plain INT4 achieves the highest isolated kernel bandwidth; fused block-diagonal rotation introduces minor arithmetic overhead but eliminates extra KV-cache data movement by integrating the rotation directly into the decode kernel. In end-to-end serving benchmarks—across varying batch sizes, concurrency levels, and model scales—INT4+BDR matches the throughput of plain INT4 and consistently outperforms BF16.

Figure 2: System-level throughput ($\mathrm{TPS}_\mathrm{sys}$) comparison demonstrates substantial gains of INT4+BDR over BF16 under long-context and high-concurrency loads.

Figure 3: Qwen3-8B per-GPU throughput vs. batch size; complex methods (e.g., Kitty) lag behind BF16 and INT4+BDR, which excels at large batch sizes.

These results substantiate a critical claim: block-diagonal Hadamard rotation enables near-lossless INT4 KV-cache accuracy while fully preserving serving efficiency, with zero measurable end-to-end throughput penalty.

Ablation and Scaling Analysis

Rotation block size, target (keys-only vs. keys-and-values), and vector quantization codebook size are ablated. Rotating keys only is sufficient; larger block size (up to 128) is necessary for quantization-sensitive models but shows diminishing returns above that threshold. Vector quantization codebooks saturate at $C=256$; larger sizes do not yield further improvements and are impractical due to calibration and memory cost.

Systems Implications and Future Directions

The evidence highlights that KV-cache compression in LLMs is fundamentally a systems co-design problem. Fused block-diagonal rotation is inherently compatible with paged layouts, attention kernel fusion, and large-scale batching—critical for production workloads. More complex quantization methods can undermine these system optimizations and are not justified by minor accuracy improvements in deployment scenarios.

Looking forward, continued increases in model context size and concurrency will further intensify KV-cache constraints. Lightweight, system-aware quantization (e.g., INT4+block-diagonal Hadamard) will be central to scalable, efficient LLM inference stacks. Future research may explore dynamic block size adaption, hardware-specific rotation kernels, and generalization to multimodal transformers.

Conclusion

This work establishes that token-wise INT4 quantization, augmented with block-diagonal Hadamard rotation, constitutes the minimal viable solution for serving-compatible, high-efficiency, high-accuracy KV-cache compression. The method is robust across model scales, benchmarks, and hardware configurations, and introduces zero serving overhead. The findings demonstrate that effective KV-cache quantization is anchored in system co-design, not purely quantization heuristics, and will be foundational for next-generation LLM serving architectures.