S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning

Abstract: Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, removing both the offline re-cluster step and the hand-tuned choice of which transformer layer to cluster on. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, establishing a new Pareto frontier without offline re-clustering or teacher distillation. An analysis of the predictor's per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, providing direct empirical evidence that the soft-target objective preserves the acoustic ambiguity that hard targets would collapse. Code is available at https://github.com/gioannides/s-jepa.

• The paper introduces a novel SSL framework that uses soft clustering with online GMM adaptation to model acoustic uncertainty at speech boundaries.
• It employs a JEPA architecture with KL-divergence objectives and adaptive layer selection to eliminate iterative re-clustering.
• Empirical results on the SUPERB benchmark demonstrate its parameter efficiency and competitive performance in ASR and emotion recognition tasks.

S-JEPA: Soft Clustering Anchors for Self-Supervised Speech Representation Learning

Introduction

The "S-JEPA: Soft Clustering Anchors for Self-Supervised Speech Representation Learning" framework (2606.19398) presents a single-pass, mask-prediction-based speech self-supervised learning (SSL) architecture that utilizes soft clustering targets. In contrast to prevailing methods such as HuBERT and WavLM that rely on hard cluster assignment and offline iterative re-clustering pipelines, S-JEPA leverages online Gaussian Mixture Model (GMM) adaptation, adaptive layer selection via effective rank, and a KL-divergence training objective with soft GMM posteriors. This results in efficient parameter utilization and a streamlined training pipeline that simultaneously advances empirical performance and offers analytic interpretability of learned uncertainty at acoustic boundaries.

Methodology

JEPA Architecture with Soft Categorical Targets

S-JEPA employs a JEPA (Joint-Embedding Predictive Architecture) style encoder–predictor architecture with a cluster head, inspired by recent advances in masked predictive learning for speech and images. Unlike hard $k$-means assignments, S-JEPA supervises mask prediction with soft GMM posteriors, inducing uncertainty modeling at boundaries and removing the need for iterative, full-corpus offline cluster re-fitting. The setup involves two continuous training phases:

• Phase 1: A $K=100$ diagonal-covariance GMM is fit on 39-dimensional MFCC features and kept fixed, supplying posterior soft targets for early-stage masked prediction supervision.
• Phase 2: The GMM is updated online over EMA (exponential moving average) encoder outputs, with $K=500$ clusters. The input layer to the GMM is chosen adaptively according to maximized effective rank, supporting progressive abstraction and feature richness as training evolves.

The sole training signal is the KL divergence at masked positions between the softmax predictor head output and the GMM posterior:

$$\mathcal{L} = \frac{1}{|S|} \sum_{t \in S} \mathrm{KL}\left(q_t \,\|\, p_t\right)$$

The JEPA predictor, cluster head, and GMM are all discarded after pretraining; only encoder representations are retained for downstream use.

Figure 1: S-JEPA framework—an encoder–predictor pair matches soft GMM posteriors at masked positions, with a frozen Phase 1 GMM and online-adapted Phase 2 GMM.

Online GMM Updates and Adaptive Layer Selection

Unlike HuBERT/WavLM, S-JEPA eliminates the stop-restart iterative clustering step. The responsibility-weighted GMM updates proceed on each minibatch using no-grad features from the EMA encoder, with parameter adaptation decoupled from gradient flow. Layer selection for GMM anchoring is handled by evaluating the effective rank (entropy of normalized singular value distribution) at each encoder layer; the layer with maximal rank is chosen for GMM input, ensuring adaptation to representational shifts. This is empirically calibrated to track optimal downstream WER.

Figure 2: Effective rank across layers in Phase 1—Layer 2 is the most expressive and used for initial GMM input in Phase 2.

EMA Scheduling

Phase 2 includes a nonstationary EMA update schedule ($\alpha_{\text{fast}} = 0.999$, $\alpha_{\text{slow}} = 0.9999$), periodically switching the decay coefficient to balance target stability with adaptation responsiveness, supporting stable soft target estimation by the GMM as the encoder evolves.

Experimental Results

SUPERB Benchmark: Pareto Frontier and Empirical Comparison

S-JEPA is pretrained on ∼83k hours of English speech (LibriLight + Granary) and evaluated (encoder frozen) on the SUPERB suite, targeting ASR, emotion recognition (ER), and slot filling (SF). With only 51.8M parameters, S-JEPA achieves:

• WER (ASR, LibriSpeech test-clean): 12.10% (greedy CTC), 8.50% (4-gram LM)
• ER accuracy: 64.83%

These results establish a new Pareto frontier for SSL encoders below 90M parameters, dominating sub-90M baselines (DistilHuBERT, DeCoAR 2.0, APC variants) and matching HuBERT-Base's ER performance at only 55% of its parameter count.

Figure 3: S-JEPA advances the Pareto frontier on SUPERB below 90M parameters—parameter efficiency versus performance for ASR and ER tasks.

S-JEPA’s strong performance is achieved without any downstream teacher distillation or offline re-clustering, a key claim validated by: (i) direct single-pass training, and (ii) GMM parameterization and adaptation being fully decoupled from label annotation.

Probing Diagnostic Tasks

Linear and MLP probes on LibriSpeech indicate that S-JEPA’s representations are highly discriminative for speaker, gender, and chapter ID, with nonlinear probes especially benefiting phoneme classification, suggesting encoded phonetic content is both rich and nonlinear. The chapter ID task showcases S-JEPA's capacity to capture subtle recording and prosodic distinctions not explained by speaker identity alone.

Analysis of Learned Uncertainty

Per-Utterance and Aggregate Predictor Behavior

S-JEPA’s soft targets result in well-structured posterior distributions at masked positions. On held-out utterances, the predictor’s rank-1 posterior is near 1.0 in phonetically stable regions and sharply transitions to a 2-way competition with probabilities in the 0.3–0.45 range at word boundaries, directly reflecting acoustic ambiguity.

Figure 4: Prediction detail for one utterance—spectrogram, word alignments, cluster assignment, and top-3 cluster posteriors, illustrating sharp uncertainty rise at boundaries.

The aggregate distribution of per-frame predictor entropy (over 153k LibriSpeech frames) is distinctly bimodal: a confident regime below 0.3 bits and a prominent mode near 1 bit (matching a perfect two-way tie), with 36.6% of frames showing greater than 1 bit of uncertainty. This manifests direct and structured encoded ambiguity at boundaries—information that is lost with hard categorical labeling.

Figure 5: Per-frame cluster posteriors—rank-1: high in steady regions, sharp falls at word boundaries, with rank-2 rising, indicating interpretable uncertainty.

Implications and Future Directions

S-JEPA substantiates that KL-matching to soft GMM clusterings is not only analytically interpretable but also outperforms hard-label SSL formulations in compute-limited regimes ($<$90M parameters), all while reducing the complexity of the training pipeline.

• Analytic implications: The maintenance of structured soft uncertainty, particularly at acoustic transitions, supplies a direct metric of where representations encode ambiguity, enhancing interpretability compared to hard-cluster objectives.
• Practical implications: The single-pass, online-adapted learning trajectory minimizes engineering and computational burden. This will be crucial for low-resource regimes, languages lacking precise phoneme inventories, and multi-lingual or domain-agnostic SSL pretraining.
• Theoretical avenues: The use of adaptive GMM anchoring in latent space heuristically resembles mixture priors in variational methods and might inspire further work on mixture-based regularization or compositional latent structures in SSL.
• Open questions: The relationship between masking, GMM cluster cardinality, phase schedules, and downstream performance requires more controlled ablation. Expansion to tonal, code-switched, and non-speech audio modalities is a direct extension path.

Conclusion

S-JEPA introduces a new paradigm for SSL speech representation learning by leveraging continuous online adaptation of soft cluster anchors, KL-divergence training, and adaptive layer selection. It is the most parameter-efficient system evaluated on SUPERB to date, with robust empirical and analytic evidence that soft targets preserve and utilize structured ambiguity at acoustic boundaries—information systematically discarded by standard hard-cluster SSL objectives. The methodological modularity and observed performance gains position S-JEPA as a new technical baseline for efficient, direct, and interpretable speech pretraining in self-supervised settings.