Stacked Quantizers: Hierarchical Vector Quantization

• Stacked Quantizers (SQ) are a hierarchical compositional vector quantization method that uses a coarse-to-fine encoding strategy to achieve low reconstruction error and efficient training.
• The methodology involves sequential k-means initialization followed by residual-based refinement, enabling scalable training and strong performance in image retrieval and classification.
• Empirical evaluations show that SQ nearly matches fully dependent quantization methods like AQ while being significantly faster, making it ideal for large-scale applications.

Stacked Quantizers (SQ) are a hierarchical approach to compositional vector quantization designed to achieve low reconstruction error comparable to fully dependent quantization schemes, while retaining computational efficiency that approaches methods based on codebook independence. SQ introduces a coarse-to-fine structure in subcodebooks, enabling efficient deterministic encoding, scalable training, and strong empirical performance across multiple descriptor types and benchmarks, particularly in image retrieval, nearest neighbour search, and classification with compressed features (Martinez et al., 2014).

1. Problem Setting and Background

Vector quantization seeks to encode high-dimensional data $X \subset \mathbb{R}^d$ using a compact codebook $C \in \mathbb{R}^{d \times k}$ and one-hot codes $b \in \{0,1\}^k$ (with $\|b\|_0 = \|b\|_1 = 1$) minimizing average reconstruction error: $$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$ Compositional quantization generalizes traditional schemes by approximating each vector $x \in \mathbb{R}^d$ with a sum over $m$ codes from $m$ smaller subcodebooks: $$x \approx \sum_{i=1}^m C_i b_i, \quad b_i \in \{0,1\}^h, \; \|b_i\|_0 = 1$$ resulting in $h^m$ representable clusters. Key approaches within this framework include Product Quantization (PQ) and Additive Quantization (AQ). PQ imposes strict orthogonality on subcodebooks, yielding efficient encoding at the cost of representational power. AQ removes independence constraints for improved reconstruction fidelity but renders encoding NP-hard, necessitating beam search or other heuristics. SQ establishes a middle ground by exploiting a hierarchical, residual-based structure.

2. Hierarchical Structure and Encoding Procedure

Stacked Quantizers construct $C \in \mathbb{R}^{d \times k}$0 subcodebooks $C \in \mathbb{R}^{d \times k}$1 arranged in a hierarchy. Encoding proceeds in a greedy, coarse-to-fine manner:

1. Assign $C \in \mathbb{R}^{d \times k}$2 (select codeword from $C \in \mathbb{R}^{d \times k}$3 to minimize residual).
2. Compute residual $C \in \mathbb{R}^{d \times k}$4.
3. Assign $C \in \mathbb{R}^{d \times k}$5; update residual $C \in \mathbb{R}^{d \times k}$6.
4. Repeat through $C \in \mathbb{R}^{d \times k}$7, with the final residual $C \in \mathbb{R}^{d \times k}$8 representing global quantization error.

Because each encoding step only requires search over $C \in \mathbb{R}^{d \times k}$9 centroids in $b \in \{0,1\}^k$0, per-vector encoding complexity is $b \in \{0,1\}^k$1. This is only a constant factor above PQ but several orders of magnitude faster than AQ's beam search, which requires $b \in \{0,1\}^k$2 per encoding.

3. Codebook Training and Refinement Strategy

Training for Stacked Quantizers comprises two phases:

• Initialization (Sequential $b \in \{0,1\}^k$3-means):
  • Apply standard $b \in \{0,1\}^k$4-means clustering to $b \in \{0,1\}^k$5 to form $b \in \{0,1\}^k$6 and corresponding codes $b \in \{0,1\}^k$7.
  • Compute residuals $b \in \{0,1\}^k$8.
  • Iteratively, apply $b \in \{0,1\}^k$9-means to each subsequent residual to yield $\|b\|_0 = \|b\|_1 = 1$0, $\|b\|_0 = \|b\|_1 = 1$1, ..., $\|b\|_0 = \|b\|_1 = 1$2 and their codes, each time recomputing residuals.
  • Total complexity is $\|b\|_0 = \|b\|_1 = 1$3 for $\|b\|_0 = \|b\|_1 = 1$4 iterations per level.
• Hierarchical Refinement (Coordinate Descent):
  • For each codebook $\|b\|_0 = \|b\|_1 = 1$5:
  • Remove $\|b\|_0 = \|b\|_1 = 1$6's contribution, recompute residuals.
  • Reassign $\|b\|_0 = \|b\|_1 = 1$7 using greedy encoding on these updated residuals.
  • Update codebook $\|b\|_0 = \|b\|_1 = 1$8 via a single $\|b\|_0 = \|b\|_1 = 1$9-means pass.
  • Each refinement pass operates in $$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$0, substantially below AQ’s encoding cost.

This scheme maintains the hierarchical structure and allows regularized, top-down codebook improvement. SQ’s codebooks are typically better initialized and refined than AQ, which can result in competitive or superior empirical results despite the hierarchical constraint.

4. Complexity Comparison

A comparison of computational complexities across compositional quantization methods is presented below:

Method	Per-vector Encoding	Training on $$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$1 Samples
PQ	$$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$2	$$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$3
AQ	$$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$4	$$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$5
SQ	$$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$6	$$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$7

Here, $$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$8 (for AQ) is the beam width, $$\min_{C,B}\;\frac{1}{n}\sum_{j=1}^n\Big\lVert x_j - C\,b_j\Big\rVert_2^2$$9 and $x \in \mathbb{R}^d$0 are the number of refinement iterations, and $x \in \mathbb{R}^d$1 refers to $x \in \mathbb{R}^d$2-means iterations. SQ achieves encoding and training costs within a modest constant factor of PQ, while avoiding the intractability of AQ for large-scale datasets.

5. Empirical Evaluation and Results

SQ was evaluated on three million-scale datasets: SIFT1M (128-D hand-crafted), GIST1M (960-D hand-crafted), and ConvNet1M-128 (128-D deep CNN features). For code lengths of 16, 32, 64, and 128 bits ($x \in \mathbb{R}^d$3 per subcodebook), key findings include:

• Quantization Error: SQ matches or improves AQ’s error on SIFT1M and GIST1M, and achieves up to $x \in \mathbb{R}^d$4 lower error than AQ on ConvNet1M-128 at longer code lengths.
• Approximate Nearest Neighbours (Recall@$x \in \mathbb{R}^d$5): At 32 bits, SQ provides the highest recall across $x \in \mathbb{R}^d$6 for SIFT1M and GIST1M; on ConvNet1M-128, SQ remains competitive with AQ and outperforms PQ/OPQ, particularly as code length increases.
• Classification with Compressed Features: On ILSVRC-2012 deep feature compression, both SQ and AQ exhibit more graceful efficacy degradation than PQ/OPQ as code length shrinks; for example, at 32 bits, PQ/OPQ top-5 error can exceed $x \in \mathbb{R}^d$7, while SQ/AQ remain around $x \in \mathbb{R}^d$8.
• Running Time (ConvNet1M-128, 8 codebooks = 64 bits):
  • Training: PQ/OPQ (100 $x \in \mathbb{R}^d$9-means iters): $m$0 min; SQ (init + 100 refinements): $m$1 min; AQ/APQ (beam search): $m$2 h.
  • Database Encoding: PQ/OPQ: $m$3 s; SQ: $m$4 s; AQ/APQ: $m$5 h.

These results substantiate SQ’s ability to deliver strong quantization fidelity at near-PQ computational efficiency.

6. Practical Considerations and Limitations

Stacked Quantizers offer state-of-the-art quantization error and search accuracy with scalable, deterministic, and easily parallelizable greedy encoding. Training, particularly refinement, is more costly than for PQ, but remains an offline process and maintains feasibility for large datasets (encoding $m$6 million vectors in under one minute). Although the hierarchical structure theoretically may omit certain cross-subcodebook dependencies fully modeled by AQ, in practice, SQ frequently matches or surpasses AQ performance due to better initialization and refinement regimes.

SQ integrates with asymmetric distance computation and supports deployment within inverted-index or multi-index architectures for sublinear approximate nearest neighbour search.

A plausible implication is that SQ’s hierarchical design strikes a practical balance: it achieves near-optimal trade-offs between computational scalability and quantization error, making it suitable for high-performance, large-scale visual search systems and learning scenarios requiring compressed representations (Martinez et al., 2014).