Product Quantization (PQ)

• Product Quantization (PQ) is a compositional vector quantization technique that partitions high-dimensional vectors into independent subspaces and quantizes each subvector.
• It enables billion-scale nearest neighbor search and large-scale clustering by reducing storage needs and query time while leveraging lookup tables for fast distance computations.
• Extensions like PQTable, deep PQ training, and variants such as OPQ and AQ improve trade-offs between quantization error and computational efficiency, spurring ongoing research.

Product Quantization (PQ) is a compositional vector quantization technique designed to enable highly memory-efficient representation and fast approximate distance computations for high-dimensional vectors. By decomposing vectors into subspaces and independently quantizing each sub-vector, PQ achieves a radical reduction in storage requirements and query-time complexity, making it the method of choice for billion-scale nearest neighbor search, large-scale clustering, and hardware-accelerated inference. The PQ framework serves as the foundation for a family of increasingly sophisticated compressed-code-based algorithms across machine learning, information retrieval, and hardware system design.

1. Mathematical Formulation and Core Principles

Product Quantization operates by partitioning a $D$-dimensional vector $x \in \mathbb{R}^D$ into $M$ disjoint sub-vectors, each of dimension $d = D/M$: $$x = [x^{(1)}; x^{(2)}; \ldots; x^{(M)}], \quad x^{(m)} \in \mathbb{R}^{d}.$$ For each subspace $m=1,\dots,M$, PQ learns a local codebook $\mathcal{C}^m = \{c^m_1, \ldots, c^m_L\}$ of $L$ codewords via $k$-means clustering. The encoding of $x$ is an $M$-tuple of codeword indices: $$q(x) = [\,\bar{x}^1,\,\ldots,\,\bar{x}^M\,]^T, \quad \bar{x}^m = \arg\min_{1\leq \ell \leq L} \| x^{(m)} - c^m_\ell \|_2^2.$$ The PQ code length is $B = M \log_2 L$ bits per vector (commonly $L=256 \to$ 8 bits per subvector). For $N$ vectors, storage is $N \times B$ bits for the codes plus $M \times L \times d$ floats for the codebooks (Matsui et al., 2017, Martinez et al., 2014).

Approximate distance between two PQ-coded vectors $\bar{x}, \bar{y}$ uses precomputed symmetric tables: $$d_{\mathrm{SD}}^2(\bar{x},\bar{y}) = \sum_{m=1}^M \| c^m_{\bar{x}^m} - c^m_{\bar{y}^m} \|_2^2.$$ For query-to-database search, asymmetric distance computation (ADC) employs lookup tables of $\| q^m - c^m_k \|^2$ for query $q^m$ and each codeword $k$ per subspace, yielding per-item cost $O(M)$ (Matsui et al., 2017, Matsui et al., 2017).

2. Training, Encoding, and Theoretical Properties

PQ training comprises independent $k$-means clustering on each subspace, yielding linear total complexity $O(n L D I)$ for $n$ samples and $I$ iterations. Encoding a new vector similarly reduces to $M$ nearest-neighbor searches, $O(M L d) = O(L D)$ per vector (Martinez et al., 2014, Matsui et al., 2017).

By imposing strict orthogonality (block independence) between codebooks, PQ admits highly parallelizable training, storage, and fast query computation. The implicit full codebook size is $L^M$, exponential in $M$, yet storage and distance computation scale only linearly with $M$, not $D$. Empirically, PQ incurs higher quantization error than codebook-dependent compositional quantization (e.g., Additive Quantization), but this error may be reduced by increasing $M$ or $L$ (Martinez et al., 2014).

3. Efficient Large-Scale Clustering and Search with PQ

PQ is widely adopted for billion-scale approximate nearest neighbor (ANN) search and clustering in memory-restricted regimes. In PQk-means clustering (Matsui et al., 2017), both assignment and centroid update steps are performed directly in the PQ code domain. Assignment is computed via lookup of symmetric distances, supporting hash-table–based acceleration (PQTable), which can offer $10^2$–$10^5\times$ speedup for large $K$, and the update employs fast histogram-based voting (sparse voting) per subspace.

The PQTable search structure (Matsui et al., 2017) replaces linear ADC scan ($O(N)$) with hash-table lookups, yielding sublinear query time, scalable for code lengths up to 128 bits and database sizes up to $N=10^9$ ($\sim$5.5 GB for $B=32$). For optimal table count $T$, the subcode length $B/T$ is chosen to approximately match $\log_2 N$ (Matsui et al., 2017).

4. Extensions, Variants, and Theoretical Innovations

Multiple advances have generalized PQ to address quantization error, codebook structure, and application-specific challenges:

• Projective Clustering Product Quantization (PCPQ) introduces scalar projection within each block, giving each section a richer representational capacity, and quantizing the scalars to $s$ levels (Q-PCPQ). This expands the effective codebook size to $(L s)^M$ with negligible cost increase per query (Krishnan et al., 2021).
• Stacked and Additive Quantization: PQ fixes fully independent subcodebooks; Additive Quantization (AQ) removes independence entirely but makes encoding NP-hard; Stacked Quantizers (SQ) offer hierarchically dependent codebooks to balance expressivity and efficiency (Martinez et al., 2014). AQ/SQ achieve lower distortion than PQ but at far higher encoding cost.
• Deep and End-to-End PQ Training: Recent approaches (e.g., MoPQ, DPQ) combine PQ with neural feature encoders and task-aligned, differentiable objectives (e.g., Multinoulli Contrastive Loss, or progressive quantization blocks under deep supervision), yielding significant end-to-end gains in retrieval and image search (Xiao et al., 2021, Gao et al., 2019).

Algorithm	Codebook Structure	Encoding Time Per Vector	Achievable Distortion
PQ	Fully independent	$O(L D)$	Highest (typ. 0.12–0.15)
AQ	Fully dependent	NP-hard ($O(M^3 b L D)$)	Lowest (typ. 0.10)
SQ	Hierarchically stacked	$O(M L D)$	$\leq$ AQ error

[Data: (Martinez et al., 2014)]

5. Applications in Information Retrieval, Clustering, and Hardware Acceleration

PQ is a central technology for:

• Billion-scale clustering: PQk-means achieves near–k-means accuracy and 100$\times$ memory reduction, enabling clustering of $10^9$ vectors with $K=10^5$ in $<14$ h on a single multi-core machine, using only 32 GB RAM (Matsui et al., 2017).
• ANN search: PQTable and bilayer PQ structures (FBPQ, HBPQ) perform sublinear search with tight recall/runtime trade-offs, outperforming classical inverted indices, especially on high-dimensional feature descriptors (Babenko et al., 2014, Matsui et al., 2017).
• End-to-end dense retrieval: Architectures such as JPQ and MoPQ co-train the encoder and quantizer to maximize ranking accuracy under extreme compression; for example, JPQ achieves 30$\times$ index size reduction and 10$\times$ CPU speedup while matching brute-force retrieval accuracy (Zhan et al., 2021, Xiao et al., 2021).
• Online and streaming data: Online PQ supports incremental codebook updates, including sliding-window forgetting and budget-constrained partial updates, with provable loss bounds and negligible deviation from offline PQ recall (Xu et al., 2017).
• DNN hardware acceleration: PQ facilitates multiply-free inference on edge/FPGA with performance/area gains up to $3.1\times$ relative to standard accelerators at sub-1% accuracy loss, via LUT-based multiply-accumulate replacement (AbouElhamayed et al., 2023, Ran et al., 2022).

6. Limitations, Trade-offs, and Practical Considerations

PQ’s independence assumption can limit adaptability to complex or highly correlated data: quantization error plateaus as $M$ increases due to blockwise rigidity, and performance is sensitive to partition strategy (axis-aligned splits vs. learned rotations, as in OPQ) (Martinez et al., 2014, Matsui et al., 2017). Increasing code length ($B$) improves recall at the cost of memory/latency; empirical results highlight code-length–accuracy trade-offs, with 32 bits often delivering strong compression and 64+ bits yielding further accuracy gains.

For large-scale clustering, PQk-means is typically slower per iteration than Bk-means but delivers consistently 10–30% lower clustering error at comparable or lower memory occupation (Matsui et al., 2017). For neural retrieval, decoupled reconstruction-loss minimization in PQ can fail to guarantee improved ranking, motivating task-specific joint objectives (Xiao et al., 2021).

7. Recent Advances and Open Research Problems

Recent work has introduced:

• Random Product Quantization (RPQ): Employing randomized subspace selection for each sub-quantizer to reduce inter-quantizer correlation and achieve lower quantization error bounds—a decrease to $\rho\,\epsilon_{\text{kms}}$ as $M\to\infty$, where $\rho$ is the mean overlap correlation coefficient (Li et al., 7 Apr 2025).
• Fuzzy Norm-Explicit PQ: Utilizing interval type-2 fuzzy codebooks and norm-based integration, achieving up to +6% recall over standard PQ on recommendation datasets with only $<2\%$ additional computation (Jamalifard et al., 2024).
• Routing-guided PQ for Graph-Based ANNS: Integrating differentiable PQ with routing and neighborhood features from proximity graphs, optimizing codebooks end-to-end for efficient disk or in-memory search with 1.7–4.2$\times$ query-per-second gains at fixed recall (Yue et al., 2023).

Open questions include: jointly optimizing codebook rotation and per-point assignment (as in OPQ/PCPQ), GPU-based assignment and update algorithms for further speedup, and theoretical convergence rates for clustering in PQ-code space (Krishnan et al., 2021, Matsui et al., 2017).

---

References:

• (Matsui et al., 2017) PQk-means: Billion-scale Clustering for Product-quantized Codes
• (Martinez et al., 2014) Stacked Quantizers for Compositional Vector Compression
• (Matsui et al., 2017) PQTable: Non-exhaustive Fast Search for Product-quantized Codes using Hash Tables
• (Krishnan et al., 2021) Projective Clustering Product Quantization
• (Xu et al., 2017) Online Product Quantization
• (Xiao et al., 2021) Matching-oriented Product Quantization For Ad-hoc Retrieval
• (AbouElhamayed et al., 2023) PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration
• (Li et al., 7 Apr 2025) Bridging the Gap between Continuous and Informative Discrete Representations by Random Product Quantization
• (Babenko et al., 2014) Improving Bilayer Product Quantization for Billion-Scale Approximate Nearest Neighbors in High Dimensions
• (Yue et al., 2023) Routing-Guided Learned Product Quantization for Graph-Based Approximate Nearest Neighbor Search
• (Jamalifard et al., 2024) Fuzzy Norm-Explicit Product Quantization for Recommender Systems
• (Gao et al., 2019) Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
• (Ran et al., 2022) PECAN: A Product-Quantized Content Addressable Memory Network
• (Zhan et al., 2021) Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance