---
title: GMM-Anchored JEPA for Speech SSL
url: https://www.emergentmind.com/topics/gmm-anchored-jepa
type: topic
---

# GMM-Anchored JEPA for Speech SSL

Searching arXiv for the specified paper and closely related JEPA/GMM speech SSL work.
GMM-Anchored JEPA is a self-supervised speech representation learning method that stabilizes Joint Embedding Predictive Architecture training by fitting a Gaussian Mixture Model once on log-mel spectrograms and using its frozen soft posterior assignments as auxiliary cluster targets throughout training. Its defining mechanism is a decaying supervision schedule in which GMM regularization dominates early training before gradually yielding to the JEPA objective, while a nonzero residual weight is retained late in training to prevent drift. The method was introduced in "Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architectures" [2602.09040].

## 1. Problem setting and motivation

GMM-Anchored JEPA was proposed for a specific failure mode of speech JEPA training: representation collapse without explicit grounding. In the formulation described by the paper, JEPA predicts latent representations rather than reconstructing raw inputs, and the student encoder and EMA teacher can drift together without any stable acoustic reference. The paper argues that the JEPA objective alone can admit trivial or degenerate solutions and that, in speech, this appears as representation collapse in which the model stops encoding meaningful phonetic distinctions [2602.09040].

The empirical manifestation reported for ungrounded training is severe. Under the paper’s experimental conditions, **Pure JEPA** gave essentially random downstream behavior, including **ASR WER = 100%**, very poor phoneme and speaker probes, low cluster entropy, and unstable assignments. This is presented not merely as an optimization issue but as evidence that JEPA, when used in speech without an external anchor, lacks a fixed connection to acoustic structure [2602.09040].

The proposed remedy is to supply such grounding externally. Instead of repeatedly reclustering learned internal representations, the method clusters **log-mel spectrograms** once, freezes the resulting GMM, and uses its soft posterior assignments as a persistent acoustic scaffold. This yields an objective in which predictive learning and fixed acoustic supervision coexist, with the latter especially influential during early training.

## 2. Mathematical formulation and anchoring mechanism

The anchor in GMM-Anchored JEPA is a diagonal-covariance GMM fit once on log-mel frame vectors. Given a log-mel frame vector $\mathbf{m}$, the GMM defines soft assignments over $K$ clusters through the posterior

$$
q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.
$$

The paper also gives the diagonal-covariance form as

$$
q_k(\mathbf{m}) = \mathrm{softmax}_k\left(\log \pi_k - \frac{D}{2}\log(2\pi) - \frac{1}{2}\sum_{d=1}^{D}\left[\log \sigma_{k,d}^2 + \frac{(m_d - \mu_{k,d})^2}{\sigma_{k,d}^2}\right]\right).
$$

These targets are explicitly **soft posteriors**, not hard labels, so a frame near a boundary can carry nontrivial mass on multiple clusters. The paper presents this softness as important because acoustic and phonetic boundaries are often ambiguous [2602.09040].

The JEPA component consists of a student encoder $f_\phi$, an EMA teacher $f_{\phi'}$, and a predictor $h_\psi$. The student processes augmented audio, the teacher processes clean audio, and the predictor infers masked teacher latents. The JEPA loss is

$$
\mathcal{L}_{\text{JEPA}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \left\| h_\psi(\tilde{z}_{\text{student},t}) - z_{\text{teacher},t} \right\|^2,
$$

where $\mathcal{M}$ denotes masked time positions and $\tilde{z}_{\text{student}}$ is the student latent sequence with masked spans replaced by a learned mask token.

The encoder also includes a cluster head that produces probabilities $p_{\text{cluster},t}$ over the same $K$ clusters. These are trained against the frozen GMM posteriors with a KL term,

$$
\mathcal{L}_{\text{cluster}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \mathrm{KL}(q_{\text{gmm},t} \,\|\, p_{\text{cluster},t}).
$$

The total objective is

$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{JEPA}} + \lambda(t)\,\mathcal{L}_{\text{cluster}},
$$

with a linear decay schedule

$$
\lambda(t): \lambda_{\text{start}} = 1.0 \;\to\; \lambda_{\text{end}} = 0.01.
$$

The paper is explicit that $\lambda$ is **not** decayed to zero. Early in training, $\lambda \approx 1$ grounds the representation in acoustic clusters and prevents early collapse; later, $\lambda \to 0.01$ makes the JEPA objective dominant while retaining residual GMM anchoring to prevent drift [2602.09040].

## 3. Training pipeline, model variants, and operational characteristics

The training recipe uses a single, frozen GMM fit on stable input features rather than on evolving internal representations. The paper justifies **log-mel spectrograms** as a clustering substrate because they are stable, low-dimensional, acoustically meaningful, and unchanged during training. This differs conceptually from HuBERT- and WavLM-style pipelines, which derive targets from the model’s own intermediate representations and therefore require iterative re-clustering [2602.09040].

Pretraining uses about **50,000 hours** of speech from the **LibriLight large subset** and **English Granary**. Two model variants are reported. **GMM-JEPA** uses a Conformer-based encoder with about **31.1M parameters**, and **GMM-JEPA-T** uses a Transformer-based encoder with about **41.3M parameters**. Both employ the same GMM supervision scheme. The reported baseline settings include **Pure JEPA** with $\lambda = 0$ and a **WavLM-style baseline** using **k-means** clustering on log-mel features in a WavLM-like objective [2602.09040].

The masking strategy is block-wise masking with contiguous spans of **10–25 frames** and a masking ratio of **40–65\%**. Optimization uses **AdamW**. The EMA teacher update is

$$
\phi' \leftarrow \tau \phi' + (1-\tau)\phi, \qquad \tau = 0.996.
$$

The learning-rate schedule uses warmup and decay, and training is reported on **8 × B200 GPUs**. Implementation details include **$K=1024$** GMM components, **80** mel bins, **k-means++** initialization for the GMM, GMM training by **mini-batch SGD** on **9.6B frames**, and a WavLM-style baseline using **20 Lloyd iterations** with the same $K=1024$. To avoid memory issues, posterior computation is chunked [2602.09040].

Operationally, the method’s principal simplification relative to HuBERT and WavLM is that the GMM is fit once and then frozen for the entirety of training. The paper presents this as both computationally cheaper and conceptually simpler than iterative re-clustering, while also emphasizing that the target space remains externally anchored and therefore cannot co-adapt and collapse with the encoder.

## 4. Empirical performance on downstream speech tasks

The paper evaluates frozen representations on automatic speech recognition, emotion recognition, and slot filling. For ASR, evaluation is on **LibriSpeech dev-clean** using a frozen encoder and a 2-layer BiLSTM CTC decoder with greedy decoding. The reported results are as follows [2602.09040].

| Model | WER (%) | CER (%) |
|---|---:|---:|
| Pure JEPA | 100.00 | 93.11 |
| WavLM-style | 33.22 | 11.28 |
| GMM-JEPA | 29.18 | 9.62 |
| GMM-JEPA-T | **28.68** | **9.44** |

The Transformer variant gives the best ASR result, and the paper states that relative to the WavLM-style baseline this is about a **14\% relative WER reduction**. The contrast with Pure JEPA is central to the paper’s claim that external grounding is necessary for stable speech JEPA pretraining.

Emotion recognition is evaluated on **IEMOCAP**, using 5-fold cross-validation with a linear classifier on pooled frozen embeddings. The reported average accuracies are **48.11** for Pure JEPA, **65.46** for the WavLM-style baseline, **67.30** for GMM-JEPA, and **67.76** for GMM-JEPA-T. The paper characterizes the improvement as modest but consistent across folds [2602.09040].

Slot filling is evaluated on **SNIPS** with speaker-disjoint splits. The reported results are **5.0 / 2.2** for Pure JEPA, **59.1 / 33.7** for the WavLM-style baseline, **64.7 / 36.0** for GMM-JEPA, and **59.2 / 32.8** for GMM-JEPA-T, where the paired metrics are **Type F1** and **Edit F1**. In this task, the Conformer-based GMM-JEPA outperforms the Transformer variant [2602.09040].

The abstract additionally summarizes the same comparison in terms of matched-compute gains over a WavLM-style baseline: **ASR (28.68\% vs. 33.22\% WER)**, **emotion recognition (67.76\% vs. 65.46\%)**, and **slot filling (64.7% vs. 59.1% F1)** on approximately **50k hours of speech**. These comparisons define the method’s empirical profile: stabilization of JEPA training accompanied by consistent downstream improvements over the matched baseline.

## 5. Cluster-space behavior, ablations, and qualitative analysis

A major analytical theme in the paper is that stabilization should be visible not only in downstream metrics but also in cluster-space usage. The paper defines cluster occupancy entropy as

$$
H = -\sum_{k=1}^{K} p_k \log p_k, \qquad
p_k = \frac{|\{t : c_t = k\}|}{T_{\text{total}}},
$$

and reports it normalized by $\log K$ to yield a percentage. It also reports the number of used clusters and defines an adjacent-consistency metric,

$$
\mathrm{Consistency} = \frac{1}{T-1} \sum_{t=1}^{T-1} \mathbf{1}[c_t = c_{t+1}],
$$

to quantify temporal smoothness of frame assignments [2602.09040].

The reported entropy findings are central. **Pure JEPA** achieves **45\%** entropy and **516** used clusters; the **WavLM-style** baseline achieves **31\%** entropy and **978** used clusters; **GMM-JEPA** achieves **85\%** entropy and **1007** used clusters; and **GMM-JEPA-T** reaches **98\%** entropy with **1013** used clusters. The paper highlights that the WavLM-style baseline uses many clusters at least once but still has low entropy because assignments are highly imbalanced. By contrast, GMM-Anchored JEPA distributes usage much more uniformly across the available cluster vocabulary [2602.09040].

The qualitative analyses align with these statistics. According to the paper’s UMAP and temporal cluster plots, **Pure JEPA** collapses into a small dense region with unstable flickering cluster assignments, the **WavLM-style** baseline spreads more but remains diffuse and overlapping, and **GMM-JEPA variants** form more separated and coherent regions. Temporal plots show more stable cluster spans and higher confidence for GMM-JEPA than for the baselines.

An ablation on late-stage supervision addresses whether the GMM term is merely a curriculum device. When $\lambda_{\text{end}} = 0$, compared with $\lambda_{\text{end}} = 0.01$, entropy drops from **84.7\%** to **57.7\%**, used clusters drop from **1011** to **530**, WER worsens from **29.2** to **40.9**, and SER drops from **67.3** to **63.8**; slot filling also degrades. The paper interprets this as evidence that the GMM anchor is not just a warm start: even a small residual weight is necessary to preserve useful structure [2602.09040].

## 6. Relation to HuBERT, WavLM, and subsequent GMM-anchored JEPA variants

The paper’s central comparison is with **HuBERT** and **WavLM**. In its description, HuBERT- and WavLM-style systems fit **k-means** on intermediate learned representations, use **hard assignments**, and require **iterative re-clustering**, so clustering and training are repeated. GMM-Anchored JEPA instead fits **one GMM only once**, on **log-mel spectrograms** rather than intermediate learned features, uses **soft assignments**, and keeps the clustering targets frozen throughout training. The key conceptual difference is that HuBERT and WavLM use the model’s own evolving representations as the source of targets, whereas GMM-Anchored JEPA uses an **external, fixed acoustic anchor** [2602.09040].

The paper also frames softness as a substantive difference. Hard $k$-means labels force each frame into exactly one bin, whereas GMM posteriors preserve uncertainty at boundaries. However, the authors explicitly note that they do **not** ablate soft versus hard GMM assignments, so the value of softness remains a hypothesis rather than a proven causal factor [2602.09040].

A later JEPA-based speech method, **S-JEPA**, preserves the broad idea of GMM-based soft anchoring but changes the role of the GMM substantially. S-JEPA is trained to match the soft posteriors of a GMM at masked positions via KL divergence; training proceeds in two phases, beginning with a fixed GMM over **39-dim MFCC + delta + delta-delta** features with **$K=100$**, then moving to an **online GMM** over encoder features with **$K=500$**, updated minibatch-by-minibatch from EMA encoder features. It also introduces adaptive, label-free layer selection based on effective rank, and the soft posterior becomes the **sole training target**, without offline re-clustering or teacher distillation [2606.19398]. This comparison suggests a methodological progression from a fixed frozen anchor toward an online, continuously updated target space, but the two formulations differ in objective structure and training pipeline.

The limitations reported for GMM-Anchored JEPA are specific. The authors do **not** compare directly to published HuBERT or WavLM checkpoints because architecture, data, and compute confounds would make the comparison unfair. They do **not** ablate soft versus hard GMM assignments. The decay schedule and final residual weight were chosen from preliminary experiments rather than exhaustive tuning. Experiments are only on **English speech**. The paper therefore presents the method as a way to stabilize JEPA, not as a claim of absolute state of the art [2602.09040].

Source: https://www.emergentmind.com/topics/gmm-anchored-jepa