---
title: Supervised Encoding Quantizers (SEQ)
url: https://www.emergentmind.com/topics/supervised-encoding-quantizers-seq
type: topic
---

# Supervised Encoding Quantizers (SEQ)

Supervised Encoding Quantizers (SEQ) constitute a supervised learning framework that combines discrete representation learning with interpretable clustering and semigenerative capabilities. SEQ departs from the classical paradigm, where features are directly mapped to label probabilities, by explicitly clustering encoded features and leveraging quantization to yield both interpretable graph-structured representations and controllable style interpolation. Discrete cluster assignments correspond to "styles" or sub-classes, providing semantic structure and transparency to the latent space. The approach was introduced and developed in "Supervised Encoding for Discrete Representation Learning" [1910.11067].

## 1. Model Architecture and Key Components

SEQ comprises three core modules: encoder, quantizer, and decoder, supported optionally by a classification head.

- **Encoder $f_\phi$:** A feed-forward or convolutional neural network mapping each input $x \in \mathbb{R}^D$ to an embedding $z = f_\phi(x) \in \mathbb{R}^d$. Typical architectures include MLPs of depth 2 or 4 ("LAE-2", "LAE-4") or a convolutional frontend plus MLP ("CAE-4"). Initial encoder training uses a conventional cross-entropy objective with a softmax classification layer $W$ attached.

- **Quantizer $Q_C$:** Once encoder parameters $\phi$ are fixed, all embeddings $\{z_i\}$ are clustered using $k$-means, yielding cluster centers $C = \{c_1, \ldots, c_K\}$. Quantization is performed by hard-assignment:
  $$
  Q_C(z) = \arg\min_{k=1\cdots K} \|z - c_k\|_2^2
  $$
  The cluster assignment can be represented by the one-hot vector $e_k$.

- **Decoder $g_\theta$:** A symmetric network (MLP or deconv+MLP) that reconstructs the input from latent embeddings or cluster centers: $\hat{x} = g_\theta(z)$. Decoder training minimizes mean squared error (MSE) between $x$ and $\hat{x}$.

- **Style-mixing via Convex Combination:** Embeddings $z_{i_1}, \ldots, z_{i_m}$ from different clusters (or the same) can be linearly interpolated as $z_\alpha = \sum_j \alpha_j z_{i_j}$ with $\alpha_j \geq 0, \sum_j \alpha_j = 1$, enabling smooth traversal and "style transfer" in latent space.

## 2. Training Workflow and Losses

SEQ employs a three-stage training protocol, with the option for joint optimization.

1. **Encoder Pre-training:** Encoder $f_\phi$ (and $W$) are optimized via the standard classification loss:
   $$
   \ell_{\text{cls}}^0(\phi,W) = -\sum_i y_i^T \log \mathrm{softmax}(W z_i)
   $$
   where $z_i = f_\phi(x_i)$.

2. **Quantizer Fitting:** Fix $\phi$, compute $\{z_i\}$. Run $k$-means to minimize:
   $$
   \ell_{\text{quant}}(\phi, C) = \sum_{i=1}^N \min_k \|f_\phi(x_i) - c_k\|_2^2
   $$
   Assign each $z_i$ to its nearest cluster $c_{k_i}$.

3. **Decoder Training:** With $\phi, C$ fixed, train $g_\theta$ by minimizing reconstruction loss:
   $$
   \ell_{\text{rec}}(\theta) = \sum_{i=1}^N \|x_i - g_\theta(\mathrm{sg}[f_\phi(x_i)])\|_2^2
   $$
   or, for strictly quantized codes, using $c_{Q_C(f_\phi(x_i))}$.

Optionally, a classification head $W$ can be trained on the one-hot quantized codes with a loss:
$$
\ell_{\text{cls}}(\phi,W) = - \sum_{i=1}^N y_i^T \log \mathrm{softmax}(W\, e_{Q_C(f_\phi(x_i))})
$$

A joint end-to-end objective is possible:
$$
\ell_{\text{total}} = \ell_{\text{cls}}(\phi,W) + \lambda\, \ell_{\text{quant}}(\phi,C) + \mu\, \ell_{\text{rec}}(\phi,C,\theta)
$$
with hyperparameters $\lambda, \mu$ controlling trade-offs. In practice, the original scheme sets $\lambda, \mu \rightarrow \infty$ and optimizes each stage sequentially.

### SEQ Training Pipeline (Pseudocode)

```
Input: data {x_i,y_i}_{i=1}^N, target cluster count K, encoder f_φ, decoder g_θ.
1. Pre-train encoder:
   Initialize φ,W randomly.
   Repeat until convergence:
     z_i = f_φ(x_i)
     p_i = softmax(W z_i)
     φ,W ← SGD minimizing ℓ_cls^0 = –∑_i y_i^T log p_i
   Discard W; keep φ.
2. k-means quantization:
   Compute embeddings {z_i = f_φ(x_i)}.
   Run k-means to find centers {c_k}_{k=1}^K minimizing ∑_i min_k ‖z_i–c_k‖².
   Assign each z_i to cluster k_i = argmin_k ‖z_i–c_k‖.
   Define cluster‐labels ŷ_i = majority_vote({y_j | k_j = k_i}).
3. (Optional) Train classification head W on quantized codes:
   Minimize –∑_i ŷ_i^T log softmax(W e_{k_i}).
4. Train decoder g_θ:
   Freeze φ (and C).
   Repeat until convergence:
     z_i = f_φ(x_i)
     θ ← SGD minimizing ℓ_rec = ∑_i ‖x_i – g_θ( z_i )‖²
Output: φ, C, ŷ (cluster → label map), θ.
```

## 3. Clustering Structure and Interpretability

SEQ’s quantizer induces a discrete cluster graph in latent space where each codebook center $c_k$ forms a node. Edges may be constructed between nodes whose centers are within a threshold: $\|c_k - c_\ell\|_2 < \tau$. Due to the supervised nature of training, each cluster node predominantly contains samples of a single class, and often, distinct clusters correspond to sub-styles or sub-classes within a label.

Cluster quality and alignment with semantic labels are assessed using clustering purity
$$
\text{Purity} = \frac{1}{N} \sum_{k=1}^K \max_{\ell=1..L} | \{ i : Q_C(f_\phi(x_i)) = k\,\, \text{and}\,\, y_i = \ell \} |
$$
as well as metrics such as normalized mutual information (NMI) and adjusted Rand index (ARI). In the reference work, purity is reported as the percentage of samples whose cluster’s majority-vote label matches the true label.

Empirically, clusters reveal interpretable "styles": for example, the digit '1' clusters into thin-upright, slanted, and fat-base, and fashion "bags" cluster by handle type.

## 4. Decoder and Style Interpolation

The decoder $g_\theta$ enables not only reconstruction but also style interpolation:

- **Within-cluster interpolation:** For embeddings $\{z_1, z_2, z_3\}$ within the same cluster, convex combinations
  $$
  z_\alpha = \sum_{j=1}^3 \alpha_j z_j
  $$
  decoded to $\hat{x} = g_\theta(z_\alpha)$ yield images retaining the same "style" due to local convexity.

- **Between-cluster interpolation:** Given $z_a$ (cluster A) and $z_b$ (cluster B) from the same class, interpolations $z_\lambda = \lambda z_a + (1-\lambda)z_b$ produce a smooth morph between distinct styles.

- **Interpolation assessment:** The quality of interpolated reconstructions may be quantified by
  $$
  \ell_{\text{interp}} = \|g_\theta(z_\alpha) - \hat{x}\|^2
  $$
  or visual inspection. Outputs exhibit sharp, semantically meaningful morphs, without the blurring typical in VAEs or GANs.

## 5. Experimental Results

Experiments were conducted on MNIST and Fashion-MNIST datasets, comparing SEQ to DEC, IDEC, DCEC, and CAE-$\ell_2$ baselines.

| Dataset         | DEC   | IDEC  | DCEC  | CAE-ℓ₂ | SEQ (k-means)       |
|-----------------|-------|-------|-------|--------|---------------------|
| MNIST Purity    | 86.55 | 88.06 | 88.97 | 95.11  | 99.74 (±0.046)      |

Classification accuracy (test set) increases with cluster count $K$: on MNIST, moving $K$ from 10 to 120 raises accuracy from ~92% to ~99%, and on Fashion-MNIST from ~84% to ~91.8%. The encoder’s softmax head provides a performance upper bound (~99.4% MNIST, ~92.2% Fashion-MNIST). Deeper networks (LAE-4 vs. LAE-2) and convolutional encoders (CAE-4) further enhance results. SEQ demonstrates monotonic accuracy improvement up to $K \approx 120$ on MNIST.

Qualitative evaluations reveal that clusters correspond to visually and semantically distinct sub-styles within classes; interpolated samples remain sharp and do not display VAE-like or GAN-like blurring artifacts.

## 6. Advantages, Limitations, and Sensitivity

SEQ provides several benefits over classical supervised classifiers:

- **Interpretability:** Clusters correspond to discrete, visually meaningful style modes.
- **Low Complexity Classification:** Post-training, prediction reduces to nearest-centroid lookup.
- **Fine-grained Generative Control:** The architecture enables controlled style generation and morphing via convex interpolation.

Limitations include reliance on two-stage training—the quantizer is not inherently end-to-end differentiable. The choice of cluster count $K$ is critical and impacts trade-offs between computational cost and cluster purity. There is no explicit regularization on inter-cluster distances, and, in a joint scheme, improper weighting can lead to collapsed or poorly separated clusters.

Ablation studies indicate that deeper and convolutional encoders yield higher accuracy and more meaningful style clusters, with cluster count $K$ selected to maintain quantization accuracy $P_Q$ above the encoder’s baseline $P_E$ minus a small tolerance $\epsilon$.

## 7. Broader Implications and Open Questions

A plausible implication is that SEQ bridges the gap between interpretable symbolic representations and high-accuracy deep representations, especially in domains where style diversity or subclass semantics matter. Open questions concern seamless end-to-end training of quantizable representations (potentially via soft or straight-through assignment) and transferable adaptation of SEQ to other modalities and hierarchical clustering tasks. The need for principled choice of $K$ and robust regularization of inter-cluster structure remains a central issue in extending the framework [1910.11067].

Source: https://www.emergentmind.com/topics/supervised-encoding-quantizers-seq