Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach

Abstract: As LLMs scale, memory bandwidth for key-value caches and retrieval-augmented generation systems becomes a critical bottleneck. While 1-bit quantization addresses this constraint, recent TurboQuant relies on dense random rotation matrices to condition the vector distribution before quantization. This projection demands millions of floating-point multiplications per embedding, making it difficult to deploy on constrained edge silicon. We introduce Fast-TurboQuant, a multiplier-free projection architecture that replaces the dense matrix with a structured fast Johnson-Lindenstrauss transform. By applying a Rademacher phase inversion followed by a fast Walsh-Hadamard transform (FWHT), the method leverages sub-Gaussian concentration to satisfy the prerequisites of scalar Lloyd-Max quantization without Gaussian projections. This substitution reduces the arithmetic complexity to only additions, eliminating hardware multipliers. Evaluation on DBpedia OpenAI-3 Large embeddings demonstrates a 19.7 times algorithmic speedup under sequential execution. Furthermore, the dimension expansion due to the FWHT zero-padding reduces the mean squared error and improves Recall@10.

• The paper introduces a multiplier-free quantization method that replaces dense Gaussian rotations with an efficient fast Johnson-Lindenstrauss transform (FJLT).
• A structured FWHT-Rademacher pipeline achieves a 19.7× speedup alongside improved MSE and Recall@10 metrics versus traditional TurboQuant.
• This method significantly reduces hardware and memory costs, enabling practical deployment in edge devices and retrieval-augmented generation systems.

Fast-TurboQuant: Multiplier-Free Online Vector Quantization for Key-Value Cache Compression

Motivation and Context

As LLMs and retrieval-augmented generation (RAG) systems advance, memory bandwidth and silicon efficiency become limiting constraints, especially in key-value cache and embedding retrieval pipelines. Conventional approaches like TurboQuant achieve low-distortion 1-bit quantization by projecting vectors through dense Gaussian rotation matrices, enforcing coordinate distributions favorable for scalar quantization. However, the hardware cost—millions of floating-point multiplications per inference token—renders TurboQuant impractical for edge deployments with tight arithmetic logic unit (ALU) and memory budgets. Fast-TurboQuant addresses this gap by replacing the dense matrix rotation with a fast Johnson-Lindenstrauss transform (FJLT), enabling efficient, multiplier-free embedding quantization.

Architectural Innovations

Fast-TurboQuant distinguishes itself through the structured use of the FJLT, implemented as a sequence of operations: Rademacher phase inversion followed by a fast Walsh-Hadamard transform (FWHT). This composition enables sub-Gaussian concentration required for scalar quantization, without requiring Gaussian random projections. The Rademacher diagonal alters signs at the bit level, executable through lightweight XOR operations; the Hadamard transform mixes the vector via an in-place butterfly network that only uses additions and subtractions. The normalization required for scalar Lloyd-Max quantization is handled offline, obviating the need for runtime multiplications.

Zero-padding the input to the nearest power-of-two increases dimension from $d$ (e.g., 1536) to $N$ (e.g., 2048), both facilitating FWHT and expanding the bit capacity available to quantization codebooks. This architectural decision not only enhances hardware efficiency but also improves mean-squared error (MSE) and Recall@10 metrics in practical retrieval tasks.

Hardware and Computational Analysis

The shift from dense matrix multiplication ($\mathcal{O}(d^2)$ operations) to FJLT-based projection ($\mathcal{O}(N \log_2 N)$ additions/subtractions, zero multiplications) yields substantial computational savings. Memory requirements for TurboQuant are dominated by storage of the rotation matrix, which can reach tens of MBs on high-dimensional embeddings, incurring bandwidth penalties during streaming inference. Fast-TurboQuant circumvents this by requiring only a single 1-bit Rademacher vector and fixed butterfly network routing—total storage under 1 KB for typical embedding dimensions.

Empirical Evaluation

Benchmarks employing DBpedia entities with OpenAI-3-Large 1536-d embeddings compare TurboQuant against Fast-TurboQuant. Both systems quantize projected embeddings to 1-bit using the sign operation, followed by appropriate scaling. Evaluation metrics include runtime, MSE, and Recall@10 for retrieval accuracy.

Under sequential (non-vectorized) execution, Fast-TurboQuant yields 147.08 ms projection time versus 2899.54 ms for TurboQuant—a 19.7x algorithmic speedup. MSE improves from $1.025\times10^{-3}$ (TurboQuant) to $9.09\times10^{-4}$ (Fast-TurboQuant), and Recall@10 improves from 0.7122 to 0.7390. These improvements stem from expanded bit budget per coordinate due to dimension padding and intrinsic properties of the FWHT-Rademacher pipeline.

Fast-TurboQuant preserves the statistical integrity of inner product estimation: empirical error distributions remain centered at zero, but the padded FJLT projection narrows the error distribution, improving precision and recall in similarity search.

Figure 1: Empirical distribution of inner product estimation errors on DBpedia OpenAI-3 embeddings. The padded FJLT projection ($N=2048$) narrows the error distribution relative to the baseline projection ($d=1536$).

Theoretical Foundation and Sub-Gaussian Concentration

The substitution of FJLT for dense Gaussian projection is theoretically justified. The randomized Hadamard transform, augmented with Rademacher phase inversion, produces coordinates with sub-Gaussian tails, matching the requirements for optimal scalar quantization. Formal proofs establish that the projected vectors approach asymptotic normality, making the quantization performance comparable with dense random rotation pipelines. The hardware realization aligns precisely with theoretical constraints, ensuring that practical compression does not deviate from the statistical demands of quantized retrieval.

Implications and Future Directions

Fast-TurboQuant enables low-distortion similarity-preserving quantization for online, streaming inference in resource-constrained edge devices, making long-context LLM deployment feasible without dense matrix storage or floating-point arithmetic bottlenecks. The architecture demonstrates that structured transforms—especially FWHT with randomized sign inversion—can replace dense projections in quantization-critical retrieval workloads.

Practically, this innovation opens the door for real-time LLM inference and RAG systems in cost-sensitive environments (mobile, IoT, FPGA, and low-power CPUs). Theoretical implications include broader adoption of transform-domain quantization in neural compression and approximate nearest neighbor retrieval.

Future research may explore adaptive codebook design responsive to the expanded sketch dimension, hybrid quantization strategies across power-of-two padded vectors, and efficient hardware fusion with bitwise operations. Analytical extension to multi-bit quantization, block-wise transforms, and integration into high-speed vector search engines is likely.

Conclusion

Fast-TurboQuant represents a multiplier-free quantization architecture, substituting dense random rotations with FJLT for scalable deployment in edge and bandwidth-constrained environments. Empirical results validate speedup, precision improvement, and semantic integrity preservation in LLM vector retrieval. Theoretical guarantees on statistical concentration, combined with practical hardware efficiency, position Fast-TurboQuant as a sound solution for large-scale embedding compression and retrieval (2606.21448).