1-Bit Quantized JL Transform

• The paper introduces a 1-bit Quantized JL transform that leverages a Gaussian JL projection followed by sign-bit quantization to create compact binary sketches preserving inner product structure.
• It provides unbiased inner-product estimators with controlled variance and distortion, achieving near-optimal performance with minimal storage via a two-stage process.
• Practical implementations demonstrate significant KV cache compression and accelerated inference through GPU-optimized batch processing and hybrid quantization strategies.

A 1-bit Quantized Johnson–Lindenstrauss (QJL) transform is a data-oblivious compression technique that creates ultra-compact binary sketches of high-dimensional vectors while preserving geometric structure for inner product estimation. QJL leverages a random Gaussian projection (Johnson–Lindenstrauss transform) followed by sign-bit (1-bit) quantization, allowing unbiased and low-variance estimation of inner products and $\ell_2$ norms. It achieves near-optimal distortion at minimal storage and computational cost, with particular efficacy in compressing neural network Key-Value (KV) caches and enabling efficient large-scale search and inference (Zandieh et al., 2024, Zandieh et al., 28 Apr 2025).

1. Mathematical Foundations and Construction

QJL comprises a two-stage process: first, a Johnson–Lindenstrauss (JL) projection matrix reduces a $d$-dimensional real vector $x \in \mathbb{R}^d$ to a sketch of dimension $m$ via random projection $S \in \mathbb{R}^{m \times d}$ with i.i.d.\ Gaussian entries $S_{ij} \sim \mathcal{N}(0,1)$. Second, each entry of the projected vector $z = Sx$ is quantized to $\tilde x = \mathrm{sign}(z) \in \{-1, +1\}^m$, and the norm $\nu(x) = \|x\|_2$ is separately stored in reduced precision.

The core sketch is thus:

• $\tilde x = H_S(x) = \mathrm{sign}(Sx)$,
• Storage per vector: $m$ bits for signs, plus $b$ bits (e.g.~16) for $\|x\|_2$.

For downstream usage where inner products with a query $q \in \mathbb{R}^d$ must be estimated, QJL provides an asymmetric estimator:

$$\widehat{\langle q, k \rangle} = \frac{\sqrt{\pi/2}}{m} \cdot \nu(k) \langle S q, H_S(k) \rangle,$$

where $k$ is a compressed vector and $S$ is the (shared, fixed) projection matrix (Zandieh et al., 2024, Zandieh et al., 28 Apr 2025).

2. Theoretical Guarantees: Unbiasedness and Distortion

QJL provides unbiased estimators for bilinear forms:

• For any $q, k \in \mathbb{R}^d$, $$\mathbb{E}_S[\widehat{\langle q, k \rangle}] = \langle q, k \rangle$$.
• The distortion bound satisfies (with high probability)

$$|\widehat{\langle q, k \rangle} - \langle q, k \rangle| \le \varepsilon\,\|q\|_2\,\|k\|_2,$$

for $m = O(\varepsilon^{-2} \log n)$, where $n$ is the total number of stored keys.

Variance is controlled by both projection dimension and vector norm:

• For self-reconstruction (e.g., in TurboQuant's second stage), $$\operatorname{Var}[\langle y, \hat r \rangle] \le (\pi/(2d)) \|y\|_2^2 \|r\|_2^2$$ for each $y\in\mathbb{R}^d$ when $d=m$ (Zandieh et al., 28 Apr 2025).
• This property extends to all queries $q$ by union bound; relative error scales as $O(1/\sqrt{m})$.

The unbiasedness arises from the rotational invariance of Gaussian projections, with the expectation of sign-product coinciding with the true inner-product up to a known constant.

3. Algorithmic Workflow and Implementation

The construction and application of the QJL transform can be summarized as follows:

Stage	Operation	Computational Cost
Sketch Construction	$z = Sx$, $\tilde x = \mathrm{sign}(z)$, $\nu(x)$	$O(dm)$ mul-add, $O(m)$ sign extraction
Query Processing (Inner Product)	$u = S q$; $$\widehat{\langle q, k \rangle} = (\sqrt{\pi/2}/m) \nu(k) \langle u, \tilde k \rangle$$	$O(dm)$ for $u$, $O(m)$ per key

CUDA implementations use memory packing for sign vectors and parallel reductions for batch processing. Shared memory caches $S q$ for amortized cost, resulting in high throughput: e.g., for $d = 4096, m = 256$, projection latency $\sim 0.2$ms, per-key estimation $\sim 0.1\,\mu$s/key (Zandieh et al., 2024).

Integration into inference pipelines typically involves:

1. Fix $S$ (with PRNG seed for reproducibility).
2. For each new vector $x$, store $(\tilde x, \nu(x))$.
3. At query time, compute $S q$ once and reuse across all comparisons.
4. Proceed with inner product estimation, softmax, and value-weighted aggregation (e.g., in transformer attention) (Zandieh et al., 2024, Zandieh et al., 28 Apr 2025).

4. Applications in Neural Network Quantization

QJL's primary application is the quantization of key/value caches in LLMs and attention-based networks:

• For KV cache compression, QJL enables more than $5\times$ reduction in memory (3 bits vs 16 bits) while maintaining or improving accuracy on long-context benchmarks (Zandieh et al., 2024).
• On datasets such as LongBench and models including longchat-7B, Llama-2, and Llama-3, the QJL sketch preserves F1 and accuracy within 0.1% of full-precision baseline at 3-bit quantization.
• QJL enables 2$\times$ acceleration of token generation over exact computation for long sequences, while traditional methods (e.g., KVQuant) may be $2\times$ slower due to added memory and access overhead (Zandieh et al., 2024).

TurboQuant incorporates QJL as a second-stage residual quantizer, following an MSE-optimal quantization step. MSE-optimal quantizers introduce bias in inner product estimation; the 1-bit QJL stage corrects for this by quantizing the residual to produce an unbiased estimator with provably bounded variance. The overall distortion rate matches the best achievable up to a small constant (factor $\approx 2.7$), achieving $O(1/4^b)$ convergence with bits-per-vector $b$ (Zandieh et al., 28 Apr 2025).

5. Comparative Memory and Computational Efficiency

QJL eliminates the per-block memory overheads typical in traditional blockwise or per-channel quantization schemes, which require storing scales and zero-points. For typical block sizes (e.g., $B=32$), per-entry overhead in classic quantization is $\sim 0.5$ bits for $b_f=8$, whereas QJL's per-entry overhead is $b_f/m$, vanishing as $m$ grows with $n$ ($m = O(\log n/\varepsilon^2)$). This directly contributes to the observed $5.3\times$ reduction in KV cache memory footprint on practical workloads (Zandieh et al., 2024).

Computation is equally efficient: for $m\ll d$, QJL decouples query and storage complexity, focusing most cost on a single JL projection per query, making it scalable for large inference batches or retrieval tasks.

6. Practical Considerations and Integration

• Random Seed Management: For consistent reconstruction, $S$ must be fixed and shared, typically by storing a PRNG seed rather than $S$ itself.
• Projection Dimension $m$: Chosen according to target error $\varepsilon$, largest number of stored vectors $n$, and application-specific tolerance. $m \approx c\, \varepsilon^{-2} \log n$ balances accuracy and memory.
• CUDA Implementation: Bit-packing and shared-memory dot products enable GPU-parallel batch processing with low overhead.
• Hybrid Quantization: For layers or channels with large dynamic ranges or outlier norms, it is practical to quantize only the top-$r$ channels with higher precision (e.g., 6 bits), while applying QJL to the remainder (Zandieh et al., 2024).
• Pipeline Position: Within TurboQuant, QJL is used only for the residual of the initial MSE-optimal quantization, yielding unbiasedness in the final estimated inner product (Zandieh et al., 28 Apr 2025).

7. Empirical Results and Benchmarks

Experimental analysis confirms the theoretical guarantees:

• Unbiasedness: Inner-product estimate histograms (TurboQuant) display zero bias across all bit widths. The variance closely matches the predicted $(\pi/2d)$ scaling.
• Quality Preservation: Benchmarks (NarrativeQA, Qasper, 2WikiMultiQA) demonstrate no loss—and occasionally improvement—in F1 compared to full-precision baseline at 3-bit quantization (Zandieh et al., 2024).
• Speedup: Prompt encoding and quantization incurs only $\sim$5% overhead compared to no-quantization baseline, while overall inference for long context is $2\times$ faster than exact computation due to lower memory traffic and simplified arithmetic.

8. Summary Table: Key Properties of QJL

Property	Description	Reference
Sketch type	1-bit sign of random Gaussian projection, with stored $\ell_2$ norm	(Zandieh et al., 2024)
Estimation property	Unbiased inner-product, distortion $\varepsilon$ with $m=O(\varepsilon^{-2} \log n)$	(Zandieh et al., 2024, Zandieh et al., 28 Apr 2025)
Memory overhead	None (vanishing per-entry as $n$ grows)	(Zandieh et al., 2024)
Dual-stage use (TurboQuant)	1-bit QJL on residual after $(b-1)$-bit MSE quantizer; unbiased, variance-bounded	(Zandieh et al., 28 Apr 2025)

In summary, the 1-bit Quantized JL transform offers a mathematically principled, hardware-friendly strategy for compressing vectors while preserving critical linear measurements, eliminating memory overheads of traditional quantization, and enabling accurate and efficient inference at scale (Zandieh et al., 2024, Zandieh et al., 28 Apr 2025).