Generalized Product Quantization (GPQ)

• Generalized Product Quantization (GPQ) is a semi-supervised image retrieval framework that integrates product quantization with deep feature learning, metric learning, and entropy-based regularization.
• It embeds soft-assignment quantization within a deep neural network to jointly optimize codebooks and feature extraction using both labeled and unlabeled data.
• GPQ enhances retrieval performance by achieving 4–5% mAP improvements and robust generalization on known and unseen categories across large-scale datasets.

Generalized Product Quantization (GPQ) constitutes a semi-supervised framework for image retrieval that integrates product quantization, deep feature learning, metric learning, and entropy-based regularization into a unified end-to-end approach. Designed to exploit both labeled and unlabeled data, GPQ addresses limitations of traditional vector quantization and deep hashing methods, which require substantial annotated data for optimal performance. By coupling quantization and representation learning within a deep neural network, GPQ achieves superior retrieval effectiveness under both known and unseen category protocols, demonstrating robust generalization in large-scale settings (Jang et al., 2020).

1. From Classical Product Quantization to GPQ

Classical product quantization (PQ) decomposes a high-dimensional feature vector $\mathbf{x} \in \mathbb{R}^D$ into $M$ sub-vectors, $\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$, where each $\mathbf{x}_m \in \mathbb{R}^d$ with $d = D/M$. Each subspace $m$ has an associated codebook $$\mathbf{Z}_m = \{\mathbf{z}_{m1}, ..., \mathbf{z}_{mK}\}$$ containing $K$ codewords. Quantization maps each $\mathbf{x}_m$ to its nearest codeword, yielding a compact binary index $\mathbf{b} \in \{0,1\}^{M\log_2 K}$, with the objective of minimizing reconstruction error: $M$0 where $M$1.

Generalized Product Quantization extends PQ in three fundamental ways:

• The quantization operation is embedded into the deep network $M$2, allowing joint optimization of codebooks and feature extraction.
• Metric learning is introduced through supervised loss, preserving semantic proximity for labeled data.
• Unlabeled data are leveraged via an entropy-based minimax regularizer, enabling semi-supervised learning and improved generalization.

The overall quantization mapping becomes: $M$3 where $M$4 is the image input, $M$5 is the deep feature extractor, and soft PQ refers to soft-assignment quantization.

2. Objective Functions and Regularization

GPQ employs a composite loss defined over mini-batches with both labeled and unlabeled data:

• Soft-assignment quantization: Each sub-vector is softly assigned to codewords: $M$6 Concatenation across subspaces yields $M$7.
• N-pair Product Quantization Loss ($M$8): Preserves semantic similarity for labeled data using a cross-entropy loss on cosine similarities between original features and quantized representations.

$M$9

where $\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$0 and $\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$1 are softmaxed cosine similarity and label matrices, respectively.

• Cosine-based classification loss ($\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$2): Maintains feature discriminability via sub-prototype weights $\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$3: $\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$4 where $\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$5.
• Subspace Entropy Minimax loss ($\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$6): For unlabeled samples, maximizes entropy of predicted class distributions (w.r.t.\ classifier weights) while feature extraction minimizes it, encouraging clustering:

$\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$7

• Total loss:

$\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$8

where gradient reversal ensures minimax dynamics on the entropy term.

This design enforces both quantization fidelity and task-relevant semantic structure, while exploiting unlabeled data for increased generalization capacity.

3. Network Architecture and Quantization Components

GPQ comprises three principal modules:

• Feature extractor ($\mathbf{x} = [\mathbf{x}_1, ..., \mathbf{x}_M]$9): A CNN variant (modified VGG or CNN-F) processes input images to $\mathbf{x}_m \in \mathbb{R}^d$0-dimensional feature vectors.
• Product Quantization Table ($\mathbf{x}_m \in \mathbb{R}^d$1): Each subspace $\mathbf{x}_m \in \mathbb{R}^d$2 has a learnable codebook with $\mathbf{x}_m \in \mathbb{R}^d$3 codewords of dimension $\mathbf{x}_m \in \mathbb{R}^d$4.
• Classifier ($\mathbf{x}_m \in \mathbb{R}^d$5): Each subspace includes a weight matrix $\mathbf{x}_m \in \mathbb{R}^d$6 for class prototypes.

After feature extraction, intra-normalization is applied to each sub-vector. Soft assignment computes $\mathbf{x}_m \in \mathbb{R}^d$7 for each subspace, whose concatenation forms the quantized representation $\mathbf{x}_m \in \mathbb{R}^d$8.

During retrieval, hard assignment maps each $\mathbf{x}_m \in \mathbb{R}^d$9 to its nearest codeword in $d = D/M$0, storing corresponding indices. Asymmetric distance computation is supported by precomputing query-to-codeword similarities and aggregating over subspaces.

4. Training Procedure and Hyperparameters

Each mini-batch contains equal numbers of labeled and unlabeled images. The algorithm performs:

• Forward passes to obtain $d = D/M$1.
• Sub-vectorization, intra-normalization, and soft quantization by subspaces.
• Computation of logits for classification and entropy losses.
• Evaluation of all loss terms ($d = D/M$2, $d = D/M$3, $d = D/M$4).
• Adam optimization with initial learning rate $d = D/M$5, $d = D/M$6, and exponential decay schedules update all parameters.

A gradient reversal layer is applied to $d = D/M$7 so that feature extractor and classifier modules are optimized in a minimax fashion.

Typical hyperparameters include:

• Number of subspaces $d = D/M$8, constrained by desired code length ($d = D/M$9, $m$0, $m$1, $m$2 bits).
• Codewords per subspace $m$3, with $m$4.
• Soft-assignment scaling $m$5, classification scaling $m$6.
• Loss weights $m$7, batch size $m$8.
• Equal split between labeled and unlabeled samples per batch.

5. Experimental Protocols and Results

Two primary retrieval protocols are employed:

• Protocol 1 (known categories): Training on a labeled subset with retrieval on the remaining (unlabeled) subset.
• Protocol 2 (unseen categories): Training on 75% of classes, with retrieval on the 25% held-out classes.

Benchmarks include CIFAR-10 (60,000 images, 10 classes) and NUS-WIDE (169,643 images, 21 concepts). The evaluation metric is mean Average Precision (mAP) at various code lengths.

GPQ achieves 4–5 percentage point improvements in mAP over semi-supervised hashing (SSDH, BGDH, SSGAH), deep PQ methods (PQN, DTQ, DQN), and classic unsupervised PQ/binary hashing, especially at low code lengths. Under domain shift (unseen classes), GPQ maintains a 2–3% mAP margin over all baselines, indicating resilience and generalization due to entropy regularization.

6. Ablation Studies and Architectural Insights

Compact ablation studies reveal:

• Use of a deeper backbone (modified VGG) outperforms shallow architectures (CNN-F).
• Removing classification or entropy losses (GPQ-H) diminishes mAP by 5–7%.
• Replacing the N-pair PQ loss with a traditional triplet loss (GPQ-T) results in suboptimal clustering and lower retrieval effectiveness, confirming the importance of the N-pair design.
• Hyperparameters $m$9 and $$\mathbf{Z}_m = \{\mathbf{z}_{m1}, ..., \mathbf{z}_{mK}\}$$0 are optimal near $$\mathbf{Z}_m = \{\mathbf{z}_{m1}, ..., \mathbf{z}_{mK}\}$$1.
• t-SNE analysis of representations reveals that full GPQ yields well-separated clusters compared to ablated variants, demonstrating the benefit of the combined losses and quantization strategy.

7. Context, Significance, and Implications

By embedding learnable product quantization in a semi-supervised deep network and leveraging both supervised and entropy-driven unsupervised losses, GPQ advances the retrieval performance under challenging annotation-scarce conditions. Its ability to generalize to novel categories is empirically validated, indicating utility in open-world retrieval scenarios. The architecture, loss design, and exploitation of unlabeled data offer a model for future extensions of quantization and coding-based retrieval in high-dimensional embedding spaces (Jang et al., 2020).