GMM-Anchored JEPA for Speech SSL

• The paper introduces a method that fits a fixed Gaussian Mixture Model on log-mel spectrograms to provide soft posterior anchors, preventing JEPA representation collapse.
• The methodology employs a decaying GMM supervision schedule, where a high initial weight gradually shifts emphasis to the JEPA objective while retaining a residual anchor to stabilize learning.
• Empirical results demonstrate improved performance over pure JEPA and WavLM-style baselines in ASR, emotion recognition, and slot filling, with notable reductions in WER and enhanced cluster entropy.

Searching arXiv for the specified paper and closely related JEPA/GMM speech SSL work. GMM-Anchored JEPA is a self-supervised speech representation learning method that stabilizes Joint Embedding Predictive Architecture training by fitting a Gaussian Mixture Model once on log-mel spectrograms and using its frozen soft posterior assignments as auxiliary cluster targets throughout training. Its defining mechanism is a decaying supervision schedule in which GMM regularization dominates early training before gradually yielding to the JEPA objective, while a nonzero residual weight is retained late in training to prevent drift. The method was introduced in "Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architectures" (Ioannides et al., 30 Jan 2026).

1. Problem setting and motivation

GMM-Anchored JEPA was proposed for a specific failure mode of speech JEPA training: representation collapse without explicit grounding. In the formulation described by the paper, JEPA predicts latent representations rather than reconstructing raw inputs, and the student encoder and EMA teacher can drift together without any stable acoustic reference. The paper argues that the JEPA objective alone can admit trivial or degenerate solutions and that, in speech, this appears as representation collapse in which the model stops encoding meaningful phonetic distinctions (Ioannides et al., 30 Jan 2026).

The empirical manifestation reported for ungrounded training is severe. Under the paper’s experimental conditions, Pure JEPA gave essentially random downstream behavior, including ASR WER = 100%, very poor phoneme and speaker probes, low cluster entropy, and unstable assignments. This is presented not merely as an optimization issue but as evidence that JEPA, when used in speech without an external anchor, lacks a fixed connection to acoustic structure (Ioannides et al., 30 Jan 2026).

The proposed remedy is to supply such grounding externally. Instead of repeatedly reclustering learned internal representations, the method clusters log-mel spectrograms once, freezes the resulting GMM, and uses its soft posterior assignments as a persistent acoustic scaffold. This yields an objective in which predictive learning and fixed acoustic supervision coexist, with the latter especially influential during early training.

2. Mathematical formulation and anchoring mechanism

The anchor in GMM-Anchored JEPA is a diagonal-covariance GMM fit once on log-mel frame vectors. Given a log-mel frame vector $\mathbf{m}$, the GMM defines soft assignments over $K$ clusters through the posterior

$$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$

The paper also gives the diagonal-covariance form as

$$q_k(\mathbf{m}) = \mathrm{softmax}_k\left(\log \pi_k - \frac{D}{2}\log(2\pi) - \frac{1}{2}\sum_{d=1}^{D}\left[\log \sigma_{k,d}^2 + \frac{(m_d - \mu_{k,d})^2}{\sigma_{k,d}^2}\right]\right).$$

These targets are explicitly soft posteriors, not hard labels, so a frame near a boundary can carry nontrivial mass on multiple clusters. The paper presents this softness as important because acoustic and phonetic boundaries are often ambiguous (Ioannides et al., 30 Jan 2026).

The JEPA component consists of a student encoder $f_\phi$, an EMA teacher $f_{\phi'}$, and a predictor $h_\psi$. The student processes augmented audio, the teacher processes clean audio, and the predictor infers masked teacher latents. The JEPA loss is

$$\mathcal{L}_{\text{JEPA}} = \frac{1}{|\mathcal{M}|} \sum_{t \in \mathcal{M}} \left\| h_\psi(\tilde{z}_{\text{student},t}) - z_{\text{teacher},t} \right\|^2,$$

where $\mathcal{M}$ denotes masked time positions and $\tilde{z}_{\text{student}}$ is the student latent sequence with masked spans replaced by a learned mask token.

The encoder also includes a cluster head that produces probabilities $K$0 over the same $K$1 clusters. These are trained against the frozen GMM posteriors with a KL term,

$K$2

The total objective is

$K$3

with a linear decay schedule

$K$4

The paper is explicit that $K$5 is not decayed to zero. Early in training, $K$6 grounds the representation in acoustic clusters and prevents early collapse; later, $K$7 makes the JEPA objective dominant while retaining residual GMM anchoring to prevent drift (Ioannides et al., 30 Jan 2026).

3. Training pipeline, model variants, and operational characteristics

The training recipe uses a single, frozen GMM fit on stable input features rather than on evolving internal representations. The paper justifies log-mel spectrograms as a clustering substrate because they are stable, low-dimensional, acoustically meaningful, and unchanged during training. This differs conceptually from HuBERT- and WavLM-style pipelines, which derive targets from the model’s own intermediate representations and therefore require iterative re-clustering (Ioannides et al., 30 Jan 2026).

Pretraining uses about 50,000 hours of speech from the LibriLight large subset and English Granary. Two model variants are reported. GMM-JEPA uses a Conformer-based encoder with about 31.1M parameters, and GMM-JEPA-T uses a Transformer-based encoder with about 41.3M parameters. Both employ the same GMM supervision scheme. The reported baseline settings include Pure JEPA with $K$8 and a WavLM-style baseline using k-means clustering on log-mel features in a WavLM-like objective (Ioannides et al., 30 Jan 2026).

The masking strategy is block-wise masking with contiguous spans of 10–25 frames and a masking ratio of 40–65\%. Optimization uses AdamW. The EMA teacher update is

$K$9

The learning-rate schedule uses warmup and decay, and training is reported on 8 × B200 GPUs. Implementation details include $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$0 GMM components, 80 mel bins, k-means++ initialization for the GMM, GMM training by mini-batch SGD on 9.6B frames, and a WavLM-style baseline using 20 Lloyd iterations with the same $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$1. To avoid memory issues, posterior computation is chunked (Ioannides et al., 30 Jan 2026).

Operationally, the method’s principal simplification relative to HuBERT and WavLM is that the GMM is fit once and then frozen for the entirety of training. The paper presents this as both computationally cheaper and conceptually simpler than iterative re-clustering, while also emphasizing that the target space remains externally anchored and therefore cannot co-adapt and collapse with the encoder.

4. Empirical performance on downstream speech tasks

The paper evaluates frozen representations on automatic speech recognition, emotion recognition, and slot filling. For ASR, evaluation is on LibriSpeech dev-clean using a frozen encoder and a 2-layer BiLSTM CTC decoder with greedy decoding. The reported results are as follows (Ioannides et al., 30 Jan 2026).

Model	WER (%)	CER (%)
Pure JEPA	100.00	93.11
WavLM-style	33.22	11.28
GMM-JEPA	29.18	9.62
GMM-JEPA-T	28.68	9.44

The Transformer variant gives the best ASR result, and the paper states that relative to the WavLM-style baseline this is about a 14\% relative WER reduction. The contrast with Pure JEPA is central to the paper’s claim that external grounding is necessary for stable speech JEPA pretraining.

Emotion recognition is evaluated on IEMOCAP, using 5-fold cross-validation with a linear classifier on pooled frozen embeddings. The reported average accuracies are 48.11 for Pure JEPA, 65.46 for the WavLM-style baseline, 67.30 for GMM-JEPA, and 67.76 for GMM-JEPA-T. The paper characterizes the improvement as modest but consistent across folds (Ioannides et al., 30 Jan 2026).

Slot filling is evaluated on SNIPS with speaker-disjoint splits. The reported results are 5.0 / 2.2 for Pure JEPA, 59.1 / 33.7 for the WavLM-style baseline, 64.7 / 36.0 for GMM-JEPA, and 59.2 / 32.8 for GMM-JEPA-T, where the paired metrics are Type F1 and Edit F1. In this task, the Conformer-based GMM-JEPA outperforms the Transformer variant (Ioannides et al., 30 Jan 2026).

The abstract additionally summarizes the same comparison in terms of matched-compute gains over a WavLM-style baseline: ASR (28.68\% vs. 33.22\% WER), emotion recognition (67.76\% vs. 65.46\%), and slot filling (64.7% vs. 59.1% F1) on approximately 50k hours of speech. These comparisons define the method’s empirical profile: stabilization of JEPA training accompanied by consistent downstream improvements over the matched baseline.

5. Cluster-space behavior, ablations, and qualitative analysis

A major analytical theme in the paper is that stabilization should be visible not only in downstream metrics but also in cluster-space usage. The paper defines cluster occupancy entropy as

$$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$2

and reports it normalized by $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$3 to yield a percentage. It also reports the number of used clusters and defines an adjacent-consistency metric,

$$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$4

to quantify temporal smoothness of frame assignments (Ioannides et al., 30 Jan 2026).

The reported entropy findings are central. Pure JEPA achieves 45\% entropy and 516 used clusters; the WavLM-style baseline achieves 31\% entropy and 978 used clusters; GMM-JEPA achieves 85\% entropy and 1007 used clusters; and GMM-JEPA-T reaches 98\% entropy with 1013 used clusters. The paper highlights that the WavLM-style baseline uses many clusters at least once but still has low entropy because assignments are highly imbalanced. By contrast, GMM-Anchored JEPA distributes usage much more uniformly across the available cluster vocabulary (Ioannides et al., 30 Jan 2026).

The qualitative analyses align with these statistics. According to the paper’s UMAP and temporal cluster plots, Pure JEPA collapses into a small dense region with unstable flickering cluster assignments, the WavLM-style baseline spreads more but remains diffuse and overlapping, and GMM-JEPA variants form more separated and coherent regions. Temporal plots show more stable cluster spans and higher confidence for GMM-JEPA than for the baselines.

An ablation on late-stage supervision addresses whether the GMM term is merely a curriculum device. When $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$5, compared with $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$6, entropy drops from 84.7\% to 57.7\%, used clusters drop from 1011 to 530, WER worsens from 29.2 to 40.9, and SER drops from 67.3 to 63.8; slot filling also degrades. The paper interprets this as evidence that the GMM anchor is not just a warm start: even a small residual weight is necessary to preserve useful structure (Ioannides et al., 30 Jan 2026).

6. Relation to HuBERT, WavLM, and subsequent GMM-anchored JEPA variants

The paper’s central comparison is with HuBERT and WavLM. In its description, HuBERT- and WavLM-style systems fit k-means on intermediate learned representations, use hard assignments, and require iterative re-clustering, so clustering and training are repeated. GMM-Anchored JEPA instead fits one GMM only once, on log-mel spectrograms rather than intermediate learned features, uses soft assignments, and keeps the clustering targets frozen throughout training. The key conceptual difference is that HuBERT and WavLM use the model’s own evolving representations as the source of targets, whereas GMM-Anchored JEPA uses an external, fixed acoustic anchor (Ioannides et al., 30 Jan 2026).

The paper also frames softness as a substantive difference. Hard $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$7-means labels force each frame into exactly one bin, whereas GMM posteriors preserve uncertainty at boundaries. However, the authors explicitly note that they do not ablate soft versus hard GMM assignments, so the value of softness remains a hypothesis rather than a proven causal factor (Ioannides et al., 30 Jan 2026).

A later JEPA-based speech method, S-JEPA, preserves the broad idea of GMM-based soft anchoring but changes the role of the GMM substantially. S-JEPA is trained to match the soft posteriors of a GMM at masked positions via KL divergence; training proceeds in two phases, beginning with a fixed GMM over 39-dim MFCC + delta + delta-delta features with $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$8, then moving to an online GMM over encoder features with $$q_k(\mathbf{m}) = \frac{\pi_k \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_k, \boldsymbol{\sigma}_k^2)}{\sum_j \pi_j \mathcal{N}(\mathbf{m}; \boldsymbol{\mu}_j, \boldsymbol{\sigma}_j^2)}.$$9, updated minibatch-by-minibatch from EMA encoder features. It also introduces adaptive, label-free layer selection based on effective rank, and the soft posterior becomes the sole training target, without offline re-clustering or teacher distillation (Ioannides et al., 17 Jun 2026). This comparison suggests a methodological progression from a fixed frozen anchor toward an online, continuously updated target space, but the two formulations differ in objective structure and training pipeline.

The limitations reported for GMM-Anchored JEPA are specific. The authors do not compare directly to published HuBERT or WavLM checkpoints because architecture, data, and compute confounds would make the comparison unfair. They do not ablate soft versus hard GMM assignments. The decay schedule and final residual weight were chosen from preliminary experiments rather than exhaustive tuning. Experiments are only on English speech. The paper therefore presents the method as a way to stabilize JEPA, not as a claim of absolute state of the art (Ioannides et al., 30 Jan 2026).