SOM-VAE: Discrete Topographic Autoencoder

• SOM-VAE are models that combine variational autoencoding with self-organizing maps to produce discrete, topologically structured latent representations.
• They use an encoder, a SOM-structured codebook for vector quantization, and a decoder alongside composite losses for reconstruction, commitment, and neighborhood smoothness.
• Applications include interpretable time series and medical monitoring, though the fixed grid topology may limit flexibility in non-grid domains.

SOM-VAE (Self-Organizing Map Variational Autoencoder) denotes a family of models that integrate vector quantization-based autoencoding with a self-organizing map to yield discrete representations with topological structure. These architectures, introduced in two distinct research lines—SOM-VAE for interpretable time series representation learning (Fortuin et al., 2018) and VQ-VAEs with Kohonen-style codebook learning (Irie et al., 2023)—use the SOM paradigm to impart topographic smoothness and interpretability to neural discrete latent variables.

1. Model Architecture and Codebook Quantization

SOM-VAE combines the expressive power of variational autoencoders with discrete codebooks that are structured as self-organizing maps. The typical dataflow consists of:

• Encoder: Transforms input $x \in \mathbb{R}^{d_{in}}$ to $m$-dimensional latent vectors via a neural network $f_\theta(x)$.
• SOM Quantization: Each latent vector $z_e = f_\theta(x)$ is assigned to its closest codebook entry $e_\nu$ in a set $$\mathcal{E} = \{e_1, \dots, e_K\} \subset \mathbb{R}^m$$, yielding $$z_q = \operatorname{argmin}_{e \in \mathcal{E}} \|z_e - e\|^2$$.
• Decoder: The quantized representation $z_q$ is mapped back to the data space by $g_\phi(z)$; some variants decode both $z_e$ and $z_q$ for improved gradient flow (Fortuin et al., 2018).
• Topographic Codebook: Unlike standard vector quantization, the codebook entries are arranged on a fixed 1D or 2D grid. During learning, not only the winner code $e_\nu$ but also its grid neighbors are updated, enforcing local similarity among code indices (Irie et al., 2023).

In time series applications, the sequence of discrete code indices is modeled as a Markov chain with a learned transition matrix, giving rise to SOM-VAE-prob (Fortuin et al., 2018).

2. Objective Functions and Learning Dynamics

SOM-VAE architectures employ a composite loss tailored to both reconstruction fidelity and the imposition of a topographic structure:

• Reconstruction Loss: $$\mathcal{L}_{\mathrm{rec}}(x) = \|x - \hat{x}_q\|^2 + \|x - \hat{x}_e\|^2$$, where $\hat{x}_q = g_\phi(z_q)$ and $\hat{x}_e = g_\phi(z_e)$. The additional path via $z_e$ helps with non-differentiability of the quantizer (Fortuin et al., 2018).
• Commitment Loss: $\alpha\|z_e - z_q\|^2$ encourages encoder outputs to remain close to their assigned code.
• SOM Loss: $$\beta \sum_{\tilde{e}\in N(z_q)}\|\tilde{e} - \mathrm{sg}[z_e]\|^2$$ penalizes codebook entries in the local neighborhood $N(z_q)$ for deviating from the local data point. Here, $\mathrm{sg}$ denotes stop-gradient (Fortuin et al., 2018).
• Codebook Update Loss (VQ-VAE variant): The codebook can also be updated by minimizing $$\mathcal{L}_{\mathrm{VQ}} = (1/N)\sum_{i=1}^N\|e_{k_i^*} - \mathrm{sg}(z_i)\|_2^2$$ when using gradient-based updates (Irie et al., 2023).

For temporal data, the full loss further integrates:

$$\mathcal{L}(x^{t-1},x^t) = \mathcal{L}_{\mathrm{SOM\text{-}VAE}}(x^t) + \gamma \left[ -\log P_M(z_q(x^{t-1}) \rightarrow z_q(x^t)) \right] + \tau \mathbb{E}_{j\sim P_M(i\to\cdot)} \|e_j - z_e(x^t)\|^2$$

This enriches the representation with probabilistic and smooth dynamics (Fortuin et al., 2018).

3. Codebook Update Mechanisms: Kohonen vs. EMA-VQ

SOM-VAE implements codebook adaptation using a variant of Kohonen's learning rule:

• Per-sample Kohonen update: For encoder output $z$, find BMU $b$ and update each $e_i$ via

$$e_i \leftarrow e_i + \alpha(t)\, h_{i,b}(t)\,(z - e_i)$$

where $\alpha(t)$ is a learning rate (typically $\beta \approx 0.01$), and $h_{i,b}(t)$ is a neighborhood kernel (hard or Gaussian, often with a shrinking radius) over the SOM grid (Irie et al., 2023).

• Batch/EMA-KSOM update: At each batch, maintain running means $M_k^{(t)}$ and counts $N_k^{(t)}$ weighted by spatial kernel $h_{j,k}^{(t-1)}$ to update

$e_k^{(t)} = \frac{M_k^{(t)}}{N_k^{(t)}}$

EMA-KSOM generalizes EMA-VQ, with the latter recovered by restricting $h_{j,k} = \delta_{j,k}$ (Irie et al., 2023).

The gradient-based update in (Fortuin et al., 2018) performs a similar operation via backpropagation, exploiting the differentiable SOM loss to move codebook entries for all local neighbors towards $z_e$ in proportion to their squared distance.

4. Emergent Properties: Topography, Cluster Quality, and Robustness

SOM-VAE representations exhibit several emergent qualities:

• Topographic organization: The update of neighbor nodes induces local similarity in the codebook, resulting in a grid where similar code indices produce similar decoded content. Perturbation studies show that shifting indices by $\pm1$ results in plausible reconstructions for KSOM-trained models, whereas EMA-VQ models collapse to noise (Irie et al., 2023).
• Improved codebook usage: SOM-VAE increases codebook perplexity (measured as $p_{ppl} = \exp\left(-\sum_k p(k)\log p(k)\right)$), indicating more uniform use of the available codes (Irie et al., 2023).
• Clustering performance: SOM-VAE outperforms $k$-means, classic SOM, and VQ-VAE benchmarks on static clustering metrics (NMI $\approx 0.59$ on MNIST/Fashion-MNIST with $K=16$) (Fortuin et al., 2018).
• Interpretability: Visualizing codebook entries as decoded images reveals smooth manifolds; in medical time series, coloring SOM cells by outcome (e.g., APACHE score) exposes low- and high-risk blocks and enables path-level trajectory analysis (Fortuin et al., 2018).

Robustness is also enhanced, as the SOM-based update is insensitive to initializations and supports stability even with batch-empty clusters or alternate initial cluster masses (Irie et al., 2023).

5. Experimental Evaluation and Computational Aspects

Experiments confirm the algorithmic benefits and practical footprint of SOM-VAE:

• Datasets: Evaluations span CIFAR-10, ImageNet, CelebA-HQ+AFHQ, MNIST/Fashion-MNIST, chaotic Lorenz attractor, and eICU medical time series (Irie et al., 2023, Fortuin et al., 2018).
• Compression and reconstruction: Reconstruction losses are nearly identical to EMA-VQ or k-means across diverse domains; e.g., CIFAR-10: EMA-VQ $51.9 \times 10^{-3}$ vs. KSOM $51.8 \times 10^{-3}$ (Irie et al., 2023).
• Convergence: KSOM achieves $20\%$ of its final loss $5$–$15\%$ faster than optimized EMA-VQ (Irie et al., 2023); training converges in $50$–$100$ epochs on MNIST scale (Fortuin et al., 2018).
• Scalability: The parameter footprint is dominated by the $K \times m$ codebook and encoder/decoder modules; inference requires a $1$NN search (complexity $O(Km)$ per point). Real-time inference is practical for $K \lesssim 512$ and $m \sim 32$ (Fortuin et al., 2018).
• Downstream tasks: In the eICU experiment, SOM-VAE-prob (with Markov smoothing) significantly improves mutual information between cluster trajectory and future outcome labels (NMI: $0.0474$ at $6$h horizon, compared to $0.0407$ for SOM-VAE and $0.0411$ for k-means) (Fortuin et al., 2018).

6. Practical Applications and Limitations

SOM-VAE and its variants are applicable wherever interpretable, discrete representations of data—especially sequences—are advantageous:

• Medical monitoring: Models patient state evolution in discretized, interpretable SOM grid spaces, supporting human audits and risk assessment (Fortuin et al., 2018).
• Dynamical systems analysis: Extracts macro-state structure from time series, e.g., capturing attractor basin transitions in chaotic systems (Fortuin et al., 2018).
• General interpretable discrete representation: Visualization and labeling are more effective with emergent topographies than with unordered VQ or $k$-means maps.

Nonetheless, the SOM grid is fixed in topology and may be mismatched to domains with non-grid manifold structure. The workaround for non-differentiability, namely adding extra reconstruction paths and stop-gradient, is effective but not a fully principled continuous relaxation. High-order Markov or recurrent models may be required for capturing long-range dependencies (Fortuin et al., 2018).

KSOM as applied in VQ-VAE offers interpretability and early training speedup but little improvement in final reconstruction compared to well-tuned EMA-VQ. The method introduces an extra hyperparameter (the neighborhood schedule, $\tau$) and topography has limited effect on generative quality aside from offering greater transparency (Irie et al., 2023).

7. Extensions and Future Directions

Planned improvements to the SOM-VAE framework include:

• Learning the underlying graph structure rather than using a fixed grid for the codebook (Fortuin et al., 2018).
• Applying Gumbel-softmax or continuous relaxations for full differentiability of the quantization operation (Fortuin et al., 2018).
• Integrating hidden Markov models or Gaussian Process priors in latent space for improved temporal modeling and uncertainty quantification (Fortuin et al., 2018).
• Directly combining the approach with attention mechanisms to better capture long-range dependencies (Fortuin et al., 2018).
• Further exploration of downstream uses for topographically organized latent spaces, particularly in domains with interpretability constraints or human-in-the-loop requirements.

SOM-VAE and its Kohonen-updated VQ-VAE variants thus represent a confluence of deep generative modeling, vector quantization, and unsupervised topological organization, offering practical advances in robustness, interpretability, and temporal modeling (Irie et al., 2023, Fortuin et al., 2018).