---
title: Block-wise Triplet Quantization
url: https://www.emergentmind.com/topics/block-wise-triplet-quantization
type: topic
---

# Block-wise Triplet Quantization

Block-wise triplet quantization refers to a joint vector quantization technique for compressing high-dimensional key vectors, principally in attention-based sequence models, that operates by dividing the rotated and normalized feature space into contiguous three-dimensional blocks (“triplets”). Each triplet is then jointly quantized by parametrizing its direction using an octahedral map and its norm as a scalar, optimizing for squared error under data-oblivious distributions derived from random orthogonal rotation. This method achieves higher fidelity than per-coordinate scalar quantization, particularly in low bitwidth regimes, as established in the OCTOPUS codec [2605.21226].

## 1. Rotation Preconditioning and Marginalization

The initial step is to transform the original key vector $k \in \mathbb{R}^d$ by separating its Euclidean norm and normalizing it to unit length, $y = \|k\|_2$, $\tilde{u} = k/y \in S^{d-1}$. A randomized, structured orthogonal rotation is constructed as $R = H\,\mathrm{diag}(s)$, where $H$ is the normalized Walsh–Hadamard matrix and $s \in \{\pm1\}^d$ is a sign-flip vector sampled per attention head. This rotation ensures that $R^\top R=I$ and can be evaluated in $O(d\log d)$ time. The rotated coordinates $u = R\tilde{u}$ inherit known symmetric-Beta marginal distributions:
$$
f(u_j) \propto (1-u_j^2)^{(d-3)/2}, \quad u_j \in [-1,1].
$$
This preconditioning both homogenizes the variance across coordinates and induces analytically tractable marginals necessary for subsequent quantization.

## 2. Octahedral Parametrization of Triplets

The vector $u$ is partitioned into $n_{\mathrm{tri}} = d/3$ contiguous triplets, $t_i = (u_{3i}, u_{3i+1}, u_{3i+2}) \in \mathbb{R}^3$. Each triplet is further decomposed into its Euclidean norm $p_i = \|t_i\|_2$ and its unit direction $n_i = t_i / p_i \in S^2$. The unit direction is mapped to a square $[-1,1]^2$ via a piecewise-linear, equal-area octahedral projection:
- Compute $\ell = |x|+|y|+|z|$ and $(P_x,P_y,P_z) = (x,y,z)/\ell$ for $n_i = (x,y,z)$.
- If $P_z \geq 0$, $(\xi,\eta) = (P_x,P_y)$; else, “fold” the lower hemisphere using $(\xi,\eta) = (\operatorname{sign}(P_x)[1 - |P_y|],\, \operatorname{sign}(P_y)[1 - |P_x|])$.
The inverse map reconstructs $n$ from $(\xi,\eta)$ by “unfolding” and normalizing. This parametrization enables efficient and uniform quantization of $S^2$ directions with minimal distortion.

## 3. Joint Lloyd–Max Quantization and Bit Allocation

OCTOPUS employs separate 1D Lloyd–Max quantizers for the two octahedral direction parameters and the triplet norm. Codebooks are defined as:
- $C_{\rm dir} \subset [-1,1]$ (shared for $\xi$ and $\eta$), trained on their marginal,
- $C_{\rm nrm} \subset [0,1]$ for $p$, trained against $f_p(r) = \frac{2 r^2 (1 - r^2)^{(d-5)/2}}{B(\frac{3}{2},\frac{d-3}{2})}$.

Quantization indices are determined by the mid-point boundaries of sequential centroids. For a triplet, the squared error distortion approximates as:
$$
\| t_i - \hat{t}_i \|_2^2 \approx 2 (p_i - \hat{p}_i)^2 + 2 p_i^2 \| n_i - \hat{n}_i \|_2^2.
$$

Bit allocation between direction and norm is optimized under an MSE criterion. For $b_{\rm dir}$ bits per direction coordinate and $b_{\rm nrm}$ bits for the norm:
$$
B_{\rm tri} = 2 b_{\rm dir} + b_{\rm nrm},
$$
and the optimal bit-gap is given by
$$
b_{\rm dir} - b_{\rm nrm} = \log_4 \left( \frac{6\,C_n\,\sigma_n^2}{2\,d\,C_p\,\sigma_p^2} \right),
$$
where $\sigma_n^2, \sigma_p^2$ are the direction and norm variances, respectively. Empirically, for $d=128$, $b_{\rm dir}=b+1$, $b_{\rm nrm}=b-1$ is optimal, giving a strictly non-uniform, dimension-dependent bit allocation.

## 4. Quantization Error Analysis

Expected total MSE for $n_{\rm tri}=d/3$ triplets is:
$$
\mathrm{MSE}_{\rm total} = \frac{d}{3} \left[ 2 C_p \sigma_p^2 4^{-b_{\rm nrm}} + \frac{6}{d} C_n \sigma_n^2 4^{-b_{\rm dir}} \right].
$$
This result follows directly from the Panter–Dite high-rate quantization theory, leveraging the rotation-sphere prior induced by $R$. The error per triplet is bounded by the sum of norm quantization error and direction quantization error, each weighted appropriately.

## 5. Encoder/Decoder Workflow and Fused Implementation

The encoding process proceeds per-key as follows:
1. Compute $y = \|k\|_2$, $\tilde{u} = k/y$, $u = R \tilde{u}$.
2. For each triplet $t_i$: compute $(p_i, n_i)$, map $n_i$ to $(\xi_i, \eta_i)$, quantize $\xi_i, \eta_i$ to indices in $C_{\rm dir}$, and $p_i$ to index in $C_{\rm nrm}$. Optionally, a $3\times3$ local search over index neighbors refines the indices.
3. Pack all direction and norm indices, plus an fp32 value for $y$.

The decoder (implemented as a fused split-K flash kernel) performs bit-unpacking, centroid lookup, octahedral inverse, triplet reconstruction, inverse Walsh–Hadamard transform, and remultiplies by $y$ (delayed until inside the attention dot-product). The full uncompressed key is never materialized; all operations remain in registers, minimizing memory bandwidth.

A fused Triton kernel implementation reconstructs keys on the fly with no increased decode-time bandwidth or latency compared to per-coordinate dequantization, as only the packed bitstreams and small centroid tables are required.

## 6. Empirical Results and Comparative Performance

On synthetic Gaussian keys ($d=128$), at 4 bits per coordinate, the per-vector MSE with block-wise triplet quantization is $1.3\times$ lower than per-coordinate TurboQuant and $2.4\times$ lower than PolarQuant at 2 bits. The addition of a 1-bit QJL residual (OCTOPUS-QJL) yields a $3\times$ lower inner-product error than TurboQuant-QJL.

For long-context LLMs (Qwen2.5-7B on WikiText-2/C4), at 4 bits per coordinate, the perplexity (PPL) gap to fp16 is +2.7% for block-wise triplet quantization, compared to +3.1/4.4/8.0% for TurboQuant-MSE/PolarQuant/TurboQuant-QJL. At 2 bits, the PPL gap is +34.7% (vs. +63/187/772% for baselines), and retrieval recall ($\approx 0.8$) remains robust, whereas competitors fail.

For autoregressive video (Wan-1.3B DiT), at 2 bits per coordinate, worst-case LPIPS is approximately $0.82$ with block-wise triplet quantization, compared to $1.0$ for TurboQuant-QJL (indicative of visual noise). In next-scale autoregressive audio, OCTOPUS achieves LSD $\approx 6.75\,\mathrm{dB}$ and SNR $\approx +1 \,\mathrm{dB}$ at 2 bits, while baselines degrade to LSD $> 12\,\mathrm{dB}$ and negative SNR.

Block-wise triplet quantization strictly outperforms prior per-coordinate codecs (TurboQuant, PolarQuant) in the extreme-compression regime (2–3 bits per coordinate), retaining a (smaller) lead at 4 bits. The addition of QJL further improves unbiased inner-product estimation at a modest bit overhead.

## 7. Summary and Implications

Block-wise triplet quantization, as formalized in OCTOPUS, generalizes rotation–Lloyd–Max codecs by partitioning the compressed feature space into three-dimensional blocks and encoding their norm and direction jointly via octahedral mapping. This achieves a dimension-dependent, strictly non-uniform bit allocation ($b+1$ bits for each direction parameter, $b-1$ for the norm), which is found to be MSE-optimal and data-oblivious. The fused implementation strategy, notably using Triton kernels, provides high computational efficiency with negligible extra peak memory and achieves $3$–$5\times$ higher compression ratios than baselines. Across modalities and tasks, block-wise triplet quantization is strictly superior in low-bit regimes and never worse than existing rotation-based quantizers at high bit widths [2605.21226].

Source: https://www.emergentmind.com/topics/block-wise-triplet-quantization