---
title: Product Quantization Overview
url: https://www.emergentmind.com/topics/product-quantization
type: topic
---

# Product Quantization Overview

Product quantization (PQ) is a dimensionality reduction and vector quantization technique that decomposes high-dimensional vectors into multiple low-dimensional subspaces, assigns independent codebooks to each subspace, and quantizes the resulting sub-vectors independently. This design enables extremely compact representations and efficient approximate nearest neighbor (ANN) search in large-scale retrieval, deep learning, and compression systems. PQ’s partitioned quantization strategy, with exponentially growing codebook capacity at modest memory cost, yields fast lookup-based distance computation while controlling quantization distortion, which is critical for both accuracy and efficiency across a broad range of machine learning and signal processing applications.

## 1. Mathematical Formulation and Core Principles

Given a $D$-dimensional feature vector $x\in\mathbb{R}^D$, PQ partitions $x$ into $M$ non-overlapping sub-vectors $x = [x^{(1)}; \ldots; x^{(M)}]$ with $x^{(m)} \in \mathbb{R}^{D/M}$ (assume $D$ divisible by $M$). For each subspace $m$, an independent codebook $C^{(m)} = \{c^{(m)}_1, \ldots, c^{(m)}_K\}$ ($c^{(m)}_k\in\mathbb{R}^{D/M}$, $K$ codewords) is trained via $k$-means. The quantization of $x$ is

\[
Q(x) = [c^{(1)}_{e_1(x)}, \ldots, c^{(M)}_{e_M(x)}], \quad e_m(x) = \arg\min_{1\leq k\leq K} \| x^{(m)} - c_k^{(m)} \|_2^2
\]

The code for $x$ is the tuple of $M$ centroid indices $(e_1(x),\ldots,e_M(x))$, using $M \log_2 K$ bits. The reconstruction error is

\[
\|x - Q(x)\|_2^2 = \sum_{m=1}^M \| x^{(m)} - c^{(m)}_{e_m(x)} \|_2^2
\]

During retrieval, the squared Euclidean distance between a query $q$ (not quantized) and a database point is approximated by summing $M$ precomputed lookup values (“asymmetric distance computation”):

\[
\|q - Q(x)\|^2 \approx \sum_{m=1}^M \| q^{(m)} - c^{(m)}_{e_m(x)} \|^2
\]

This “ADC” enables $\mathcal{O}(M)$ table lookups per database vector [1404.1831], [1812.09162].

## 2. Algorithmic Procedure and Extensions

### 2.1 Standard PQ Pipeline

- **Subvector Partitioning:** Split data into $M$ blocks.
- **Codebook Training:** For each subspace, run $k$-means clustering.
- **Encoding:** For vector $x$, assign each subvector to its nearest centroid.
- **Storage:** Store $M$ indices per vector, plus codebooks.
- **Querying:** For query $q$, compute $M \times K$ distances, then for each stored code sum the $M$ relevant values.

### 2.2 Enhancements and Variants

| Extension      | Description                                              |
|----------------|---------------------------------------------------------|
| OPQ            | Rotates space for “optimal” subspace partition [1404.1831] |
| Bilayer PQ     | Adds PQ at both coarse (indexing) and fine (compression) layers for massive databases [1404.1831]          |
| Sparse PQ      | Allows each subvector to use a sparse (L>1 codewords) linear combination, reducing distortion [1603.04614]  |
| Online PQ      | Supports dynamically updating codebooks as data streams, using per-subspace incremental mean updates [1711.10775] |
| Projective PQ  | Assigns a scaling factor (possibly quantized) per block to improve MIPS and dot-product search [2112.02179] |

Modern implementations such as Quicker ADC accelerate the lookup-and-accumulate loop using SIMD and bit-split lookups [1812.09162].

## 3. Applications in Large-Scale Retrieval and Compression

PQ is foundational in billion-scale approximate nearest neighbor search for vision and multimedia:

- **ANN Image Retrieval:** PQ is used in state-of-the-art systems such as Multi-D-ADC and its fast/hierarchical variants (FBPQ, HBPQ) [1404.1831], combining an inverted multi-index structure for pruning and PQ for residual compression. Recall@1 is boosted by 10–17pp compared to non-hierarchical PQ with sublinear memory increase.
- **Embedding Compression/ASR:** In speech self-supervised learning, PQ and Random Product Quantization (RPQ) decorrelate sub-quantizers via random sampling, outperforming basic k-means in discrete WER/CER while also supporting efficient embedding fusion [2504.04721].
- **LLM KV Cache Compression:** MILLION applies PQ to LLM attention KV caches, leveraging outlier-robust subspace clustering, asynchronous quantization, and GPU-friendly lookup kernels—achieving $\sim2\times$ end-to-end speedup at 32K context versus fp16, with trivial perplexity loss [2504.03661].
- **Diffusion Model Weight Compression:** For extreme model compression, PQ enables sub-2-bit parameterization of diffusion models at up to $24\times$ reduction in model size; reinforcement via codebook pruning and end-to-end calibration recovers FID at low assignments [2411.12306].
- **DNN Hardware Acceleration:** Custom PQ accelerators replace MAC units with lookup-and-sum logic, delivering up to $3\times$ higher performance-per-area on FPGAs compared to dense MAC, and allowing for sub-6-bit precision in edge DNNs [2305.18334].

## 4. Deep and Supervised Product Quantization

Classical PQ is unsupervised and may not align with semantic or retrieval objectives. Recent advances integrate PQ into end-to-end differentiable frameworks:

- **Deep Product Quantization (DPQ):** Incorporates soft/hard codeword assignment differentiable via straight-through estimators; supports joint classification and retrieval losses. This yields significant mAP improvements (e.g., DPQ mAP@32bits on CIFAR-10: 0.831 vs deep PQ 0.733) [1906.06698], [1711.08589].
- **Generalized Product Quantization (GPQ):** Adds supervised N-pair metric loss and mini-max entropy regularization for labeled/unlabeled data in semi-supervised regimes, achieving ΔmAP $>$4pp over hashing/PQ baselines [2002.11281].
- **Matching-Oriented PQ (MoPQ):** Directly optimizes retrieval probability under a multinoulli-contrastive loss, rather than reconstruction, yielding larger Recall@K gains in ad-hoc retrieval [2104.07858].
- **Orthonormal PQ Network (OPQN):** Utilizes fixed orthonormal bases for codebooks and subspace-wise angular-loss, maximizing codeword separation and enabling superior face/image retrieval, especially for unseen classes [2107.00327].
- **Differentiable PQ for Embedding Compression:** Uses softmax (Gumbel/straight-through) relaxations to make the codebook assignment fully differentiable, enabling direct integration within language models and deep nets; compression ratios exceed $100\times$ with negligible loss [1908.09756].

## 5. Theoretical Analysis and Error Bounds

### 5.1 Quantization Error

The total quantization distortion $e_{\text{PQ}}(x)=\|x-Q(x)\|^2$ is minimized as the sum over subspaces. The curse of subspace correlation, codebook allocation, and code length all govern the reconstruction and retrieval performance [2504.04721]:

\[
\epsilon_\text{RPQ} = \left[\frac{1}{M} + (1 - \frac{1}{M})\rho\right] \epsilon_\text{kmeans}
\]

where $\rho$ (sub-quantizer correlation) controls the lower bound; in random PQ designs, lower $\rho$ via feature mixing yields improved error rates.

### 5.2 Approximation Guarantees (for MIPS)

When blocks are randomly permuted and codebooks are unbiased, the maximum error in inner product (and, by extension, in ranking) decays exponentially in the number of subspaces $K$ [1509.01469]. Projective clustering variants further minimize error along discriminative axes [2112.02179].

## 6. Practical Considerations, Implementation, and Acceleration

### 6.1 Fast ADC/Lookup Implementations

Modern hardware-aware pipelines exploit SIMD instructions (e.g., AVX-512) by packing subindices and employing split lookup tables, supporting irregular subquantizer bitwidth allocation for optimal memory and speed trade-offs [1812.09162]. Quicker ADC achieves up to $10\times$ faster exhaustive ANN than float PQ at similar recall rates.

### 6.2 Codebook Compression and Online Updates

PQ codebooks themselves can dominate memory in massive-scale regimes. Methods such as centroid importance scoring, codebook pruning and offlining, sliding-window incremental PQ, or per-subspace/layer quantization further enhance scalability [1711.10775], [2411.12306].

### 6.3 Hyperparameters

Typical settings:

| Parameter          | Usual Range                |
|--------------------|---------------------------|
| $M$ (subspaces)    | 4–32                      |
| $K$ (per-subspace) | 16–256 (for 4–8 bits)     |
| Code length        | $M\log_2 K$ bits          |
| Subspace dimension | $D/M$, typically 4–32     |

## 7. Geometrical and Task-Specific Extensions

- **Hyperbolic PQ (HiHPQ):** Embeds subspaces in Lorentz-model manifolds and employs fully differentiable “hyperbolic codebook attention,” yielding improved semantic preservation for hierarchical retrieval [2401.07212].
- **Supervised and Contrastive Losses:** PQ variants can blend supervised clustering, cross-entropy, N-pair and contrastive/self-supervised losses; these regularize codeword usage, sharpen assignments, and better align quantization with downstream retrieval or classification [1906.06698], [2002.11281], [1711.08589].
- **Weight/Activation Quantization:** In large models (LLMs, Diffusion), PQ is directly applied to weights or activations, replacing linear scaling with cluster-based representation robust to outliers and low-bit regimes (as in MILLION, achieving 4b KV-caches at $<0.5$ PPL loss) [2504.03661], [2411.12306].

## References

- "Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval" [1906.06698]
- "Improving Bilayer Product Quantization for Billion-Scale Approximate Nearest Neighbors in High Dimensions" [1404.1831]
- "Quicker ADC : Unlocking the hidden potential of Product Quantization with SIMD" [1812.09162]
- "Diffusion Product Quantization" [2411.12306]
- "MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization" [2504.03661]
- "End-to-End Supervised Product Quantization for Image Search and Retrieval" [1711.08589]
- "Generalized Product Quantization Network for Semi-supervised Image Retrieval" [2002.11281]
- "Random Product Quantization" [2504.04721]
- "Hierarchical Hyperbolic Product Quantization for Unsupervised Image Retrieval" [2401.07212]
- "Matching-oriented Product Quantization For Ad-hoc Retrieval" [2104.07858]
- "Projective Clustering Product Quantization" [2112.02179]
- "Scalable Image Retrieval by Sparse Product Quantization" [1603.04614]
- "Online Product Quantization" [1711.10775]
- "Differentiable Product Quantization for End-to-End Embedding Compression" [1908.09756]
- "Orthonormal Product Quantization Network for Scalable Face Image Retrieval" [2107.00327]
- "Quantization based Fast Inner Product Search" [1509.01469]
- "PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration" [2305.18334]

Product quantization remains a central technique in scalable vector compression and fast retrieval systems, with continuing advances in its supervised, geometric, and hardware-optimized forms.

Source: https://www.emergentmind.com/topics/product-quantization