---
title: 'SPDQ: Shared Predictive Deep Quantization'
url: https://www.emergentmind.com/topics/shared-predictive-deep-quantization-spdq
type: topic
---

# SPDQ: Shared Predictive Deep Quantization

Shared Predictive Deep Quantization (SPDQ) is a supervised deep quantization approach designed for cross-modal similarity search, particularly for retrieving semantically related items across heterogeneous modalities such as images and text. SPDQ introduces a dual-stream neural network architecture that explicitly decomposes feature representations into shared and modality-private subspaces. These latent representations are embedded into a reproducing-kernel Hilbert space and regularized for cross-modal alignment through Maximum Mean Discrepancy (MMD). In the shared subspace, SPDQ employs label-aligned additive quantization to produce compact, semantics-preserving codes suitable for efficient large-scale retrieval. Joint optimization integrates representation learning and supervised quantizer training, yielding significantly reduced quantization error and enhanced retrieval performance relative to prior state-of-the-art methods [1904.07488].

## 1. Dual-Stream Network Architecture: Shared and Private Subspaces

SPDQ operates on paired cross-modal datasets, consisting of $N$ image–text pairs with $K$ semantic classes. Each modality is processed by a distinct, yet architecturally parallel, deep network branch:

- The **image branch** (e.g., AlexNet up to fully connected layers) bifurcates into:
  - **Shared-space unit:** A fully-connected layer with $\tanh$ activation yielding $s_i \in \mathbb{R}^{d_s}$.
  - **Image-private unit:** A fully-connected layer with $\tanh$ activation yielding $r_i \in \mathbb{R}^{d_r}$.

- The **text branch** (three-layer MLP) mirrors this structure:
  - **Shared-space unit:** Output $s_t \in \mathbb{R}^{d_s}$.
  - **Text-private unit:** Output $r_t \in \mathbb{R}^{d_r}$.

For each data pair, embeddings $(s_i, r_i)$ and $(s_t, r_t)$ are obtained for images and texts, respectively, leading to global representations $S_i \in \mathbb{R}^{d_s \times N}$, $R_i \in \mathbb{R}^{d_r \times N}$ (images) and $S_t$, $R_t$ (texts). This explicit decomposition enables SPDQ to model modality-invariant semantics in the shared space while capturing modality-specific information in the private spaces.

## 2. RKHS Embedding and Cross-Modal Alignment with Maximum Mean Discrepancy

To enforce semantic alignment of the shared representations and separation of the private representations, SPDQ employs the Maximum Mean Discrepancy (MMD) criterion within a reproducing-kernel Hilbert space (RKHS). Using a composite multi-kernel $k = \sum_u \beta_u k_u$ (typically a convex combination of Gaussians), the squared MMD between distributions $p$ and $q$ is:

$$
d_k^2(p, q) = \left\| \mathbb{E}_{x \sim p}[\phi(x)] - \mathbb{E}_{y \sim q}[\phi(y)] \right\|_{\mathcal{H}_k}^2.
$$

For each class $c$, the loss terms encourage the means of shared representations $s_i^{(c)}$ and $s_t^{(c)}$ to coincide (semantic invariance), while maximizing the discrepancy between private representations $r_i^{(c)}$ and $r_t^{(c)}$ (modality separation):

$$
\mathcal{L}_1 = \sum_{c=1}^K \Bigl[ \|\mathbb{E}[\phi(s_i^{(c)})] - \mathbb{E}[\phi(s_t^{(c)})] \|_{\mathcal{H}_k}^2
- \| \mathbb{E}[\phi(r_i^{(c)})] - \mathbb{E}[\phi(r_t^{(c)})] \|_{\mathcal{H}_k}^2 \Bigr].
$$

Discriminability is further enforced by concatenating $h_i = [s_i; r_i]$, $h_t = [s_t; r_t]$ and applying a $K$-way sigmoid cross-entropy loss over labels $L$, yielding

$$
\mathcal{L}_2 = \mathcal{L}_c(h_i, L) + \mathcal{L}_c(h_t, L).
$$

## 3. Supervised Additive Quantization with Label Alignment

After extracting shared representations, SPDQ introduces a quantization module with class-based label alignment:

- **Projection matrices:** $C_i, C_t \in \mathbb{R}^{d_\ell \times d_s}$ (orthonormal: $C_i C_i^\top = I$).
- **Class prototype codebook:** $Z \in \mathbb{R}^{d_\ell \times K}$.
- **Label-aligned representation:** $Z L \in \mathbb{R}^{d_\ell \times N}$.

Alignment loss enforces congruence between projected shared features and their class prototypes:

$$
\mathcal{L}_3 = \| C_i S_i - ZL \|_F^2 + \| C_t S_t - ZL \|_F^2,
\quad \text{subject to } C_i C_i^\top = I, \; C_t C_t^\top = I.
$$

Additive quantization is applied over $ZL$ via $M$ sub-dictionaries $D = [D_1, ..., D_M]$, each of dimension $d_\ell \times K$, and associated code assignments $B$. The quantization loss is

$$
\mathcal{L}_4 = \|ZL - DB \|_F^2.
$$

The label alignment and quantization losses are jointly weighted:

$$
\mathcal{O}_q = \mathcal{L}_3 + \beta \mathcal{L}_4,
$$

with hyperparameter $\beta > 0$.

## 4. End-to-End Objective and Optimization

The combined objective function is

$$
\mathcal{O} = \mathcal{L}_1 + \alpha\mathcal{L}_2 + \lambda(\mathcal{L}_3 + \beta\mathcal{L}_4), \quad
\text{subject to } C_i C_i^\top = I, \; C_t C_t^\top = I,
$$

with $\alpha$ controlling classification guidance and $\lambda$ balancing between representation and quantization.

Optimization alternates between two blocks:

- **A. Deep network update:** Parameters are updated via mini-batch SGD, backpropagating through the RKHS-MMD alignment, classification, and quantizer alignment losses.
- **B. Quantization module update:** Given frozen network outputs $(S_i, S_t)$, solve the subproblems:
  1. $C_i$, $C_t$ through orthogonal Procrustes (using SVD).
  2. $Z$ in closed form.
  3. $D$ via least-squares.
  4. $B$ by Iterated Conditional Modes (ICM) search with exhaustive enumeration over subcodes.

This iterative scheme continues until convergence.

### Training Pipeline Pseudocode

```
Input:  X_i, X_t, L, code length M·log₂K, α, β, λ, batch size B.
Initialize: network weights, C_i, C_t, Z, D, B.

repeat
    // Deep network update (A)
    Sample minibatch of B image–text–label triples
    Forward: CNN/MLP → {s_i, s_t, r_i, r_t}
    Compute losses: ℒ₁, ℒ₂, ℒ₃, ℒ₄
    𝒪_batch = ℒ₁ + αℒ₂ + λ(ℒ₃ + βℒ₄)
    Backpropagate, SGD-step on weights

    // Quantizer update (B)
    Compute S_i, S_t on all N points
    1) C_i ← OrthogonalProcrustes(S_i, ZL)
    2) C_t ← OrthogonalProcrustes(S_t, ZL)
    3) Z  ← [(C_i S_i + C_t S_t)+β D B] Lᵀ [(2+β)(L Lᵀ)]⁻¹
    4) D  ← (Z L Bᵀ)(B Bᵀ)⁻¹
    5) For n=1…N, update b_n by ICM over M sub-dictionaries
until convergence

Output: final network, C_i, C_t, Z, D, B.
```

At query time, an image (or text) is embedded and projected; retrieval relies on efficient Asymmetric Quantizer Distance (AQD) scoring in the shared subspace.

## 5. Retrieval Process

During inference, only the shared subspace is used. For a query image $x_i^{(q)}$:

1. Pass through CNN to obtain $s_i^{(q)}$.
2. Project: $C_i s_i^{(q)} \in \mathbb{R}^{d_\ell}$.
3. Retrieve nearest text codes encoded as $D b_d$ by AQD:

$$
AQD(x_i^{(q)}, x_t^{(d)}) = (C_i s_i^{(q)})^\top (D b_d)
$$

Scoring is accelerated via precomputed lookup tables of inner products for fast large-scale search.

## 6. Empirical Performance and Comparative Analysis

Experiments on FLICKR25K and NUS-WIDE demonstrate that SPDQ surpasses previous deep quantization methods, including CDQ, by 3–5 mean average precision (MAP) points, with pronounced gains in low-bit regimes (e.g., 16/32 bits). Top-$N$ precision metrics are also significantly improved for small $N$, indicating effective alignment and quantization.

The following table summarizes principal innovations and their empirical consequences:

| SPDQ Design Feature       | Mechanism                                               | Empirical Outcome              |
|--------------------------|--------------------------------------------------------|-------------------------------|
| Shared/Private Decomposition | Dual subspace network heads                           | Tighter inter-modal alignment  |
| MK-MMD Loss              | RKHS distribution matching; modality separation        | Lower retrieval error          |
| Label Alignment          | Collapse classes to prototypes before quantization      | Reduced intra-class variance   |
| Supervised Additive Quantization | Joint codebook learning in shared subspace         | Improved MAP, precision        |

SPDQ’s explicit RKHS-based subspace modeling, strong label alignment, and integrated quantizer learning collectively underpin both its theoretical robustness and leading empirical results [1904.07488].

## 7. Methodological Significance

Shared Predictive Deep Quantization exemplifies a cohesive integration of several advanced concepts in cross-modal retrieval:

- Subspace decomposition enhances semantic fidelity across modalities by explicitly modeling shared and modality-specific factors.
- RKHS-based MMD alignment is more flexible than Euclidean similarity constraints, as it matches entire distributions within each semantic class.
- Label alignment ensures robust representation collapse, directly optimizing for retrieval by minimizing intra-class spread before quantization.
- End-to-end learning ensures that feature extraction and quantizer adaptation are synergistically optimized, contrasting favorably with two-stage or decoupled frameworks.

A plausible implication is that deep quantization approaches employing multi-granular alignment (distributional, label-aligned, and quantization error-aware) are likely to exhibit superior performance in large-scale, heterogeneous cross-modal retrieval settings.

Source: https://www.emergentmind.com/topics/shared-predictive-deep-quantization-spdq