Papers
Topics
Authors
Recent
Search
2000 character limit reached

GLaS-JEPA: Gaussian Latent Speech Framework

Updated 3 October 2026
  • GLaS-JEPA (*Gaussian Latent Speech Joint-Embedding Predictive Architecture*) is a self-supervised learning framework designed for speech processing that predicts continuous representations of speech with Gaussian regularization.
  • The framework utilizes SIGReg to prevent collapse and constraints its representations to approximate an isotropic Gaussian distribution.
  • A 57M-parameter GLaS-JEPA model pretrained on 960 hours of LibriSpeech data shows competitive performance on tasks like ASR and slot filling, with peak performances around 25.87% CER and 6.89% WER respectively.

GLaS-JEPA (Gaussian Latent Speech Joint-Embedding Predictive Architecture) is a speech self-supervised learning framework that predicts continuous representations produced by the same encoder from which the targets are generated. It removes contrastive negatives, discrete or clustered targets, separate EMA target encoders, and temporal prediction modules. Representation collapse is instead constrained with SIGReg (Sketched Isotropic Gaussian Regularization), which encourages the target representations to approximate an isotropic Gaussian distribution. A 57M-parameter GLaS-JEPA model pretrained on 960 hours of LibriSpeech achieves 6.89% WER on frozen-encoder SUPERB ASR and 25.87% CER on SUPERB slot filling (Botté et al., 29 Sep 2026).

1. Conceptual basis and objectives

GLaS-JEPA addresses the design of prediction targets in masked speech SSL. Conventional systems construct targets through contrastive discrimination, vector quantization, clustering, or a slowly updated teacher encoder. GLaS-JEPA instead uses continuous representations generated by the current encoder on an unmasked view of the same utterance.

The framework has two simultaneous objectives. The first is masked representation prediction: the encoder must infer the representation of masked acoustic content from surrounding context. The second is distributional regularization: SIGReg constrains the population of target representations so that a constant or low-rank solution is penalized.

This separation is central to the method. Direct prediction alone admits the degenerate solution

zb,t=c,z^b,t=c\mathbf z_{b,t}=\mathbf c,\qquad \widehat{\mathbf z}_{b,t}=\mathbf c

for every batch-time position, yielding zero prediction error without preserving speech information. GLaS-JEPA therefore does not claim that current-encoder prediction is sufficient by itself; its prediction objective is coupled to representation-space regularization.

The method’s conceptual training recipe is:

masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.

The approach is related to JEPA-style latent prediction but differs from systems that stabilize learning through an EMA target encoder. It also differs from S-JEPA, which predicts soft Gaussian-mixture-model assignment distributions and uses MFCC features and EMA-encoder features.

2. Architecture and masked prediction

The acoustic frontend converts waveforms into 80-bin log-Mel filterbank features using a 25 ms analysis window, a 5 ms frame hop, and temporal stride four. The resulting latent sequence has approximately 20 ms spacing.

The main encoder is an eight-layer Conformer with:

  • model width 576;
  • 8 attention heads;
  • feed-forward width 2,048;
  • convolution kernel size 31;
  • approximately 57M parameters, excluding the pretraining projector.

For each utterance, the frontend sequence is processed through two views.

The full, uncorrupted sequence is encoded as

y=fθ(h),\mathbf y=f_\theta(\mathbf h),

and a token-wise linear projector maps the encoder states into a 128-dimensional loss space:

z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).

These projected states constitute the prediction targets. They are not quantized, clustered, averaged with an EMA teacher, or transformed by a separate target encoder.

The second view is masked. A masking operator replaces contiguous spans of latent features with a learned mask token:

h~=m(h).\widetilde{\mathbf h}=m(\mathbf h).

The masked sequence is passed through the same encoder and projector:

z^=pα ⁣(fθ(m(h))).\widehat{\mathbf z}=p_\alpha\!\left(f_\theta(m(\mathbf h))\right).

The principal configuration masks 50% of positions in contiguous spans of 10 latent tokens, corresponding to approximately 200 ms per span. The projector operates independently at each time position and does not mix neighboring temporal representations. It is discarded after pretraining; downstream systems use the Conformer encoder representations.

Let M\mathcal M denote the masked batch-time indices. The prediction loss is

Lpred=1∣M∣∑(b,t)∈M∥z^b,t−sg⁡(zb,t)∥22,\mathcal L_{\mathrm{pred}} = \frac{1}{|\mathcal M|} \sum_{(b,t)\in\mathcal M} \left\| \widehat{\mathbf z}_{b,t} - \operatorname{sg}(\mathbf z_{b,t}) \right\|_2^2,

where sg⁡\operatorname{sg} denotes stop-gradient. The masked branch is therefore trained to infer the full-view representation, while the prediction loss does not directly update the target branch.

3. SIGReg and collapse prevention

SIGReg regularizes the full-view target representations. Its objective is to make their distribution resemble an isotropic standard Gaussian rather than a point mass or a low-rank distribution.

For a representation Z∈Rdp\mathbf Z\in\mathbb R^{d_p} with masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.0, a random unit direction masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.1 produces the scalar projection

masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.2

If masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.3 follows an isotropic standard Gaussian distribution, then every projection follows a one-dimensional standard normal distribution with characteristic function

masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.4

Given representations masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.5, the empirical characteristic function in direction masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.6 is

masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.7

SIGReg uses a finite random sketch that compares empirical characteristic functions of random projections with the characteristic function of a standard Gaussian. A generic form of the discrepancy is

masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.8

where masked speech⟶current-encoder representation prediction+SIGReg.\text{masked speech} \longrightarrow \text{current-encoder representation prediction} + \text{SIGReg}.9 is a nonnegative frequency weight. In implementation, the directions, frequencies, and integral are approximated rather than evaluated exhaustively.

The intended regularization encourages zero-centered, unit-scale marginal behavior, approximately Gaussian projected distributions, and non-collapsed variation. For an isotropic Gaussian representation, the empirical covariance should approximate the identity matrix:

y=fθ(h),\mathbf y=f_\theta(\mathbf h),0

Thus, SIGReg penalizes both collapsed directions and correlations between coordinates. Its statistical motivation is related to empirical-characteristic-function normality testing, including the Epps–Pulley formulation.

The regularizer is applied to the full-view targets:

y=fθ(h),\mathbf y=f_\theta(\mathbf h),1

The complete pretraining objective is

y=fθ(h),\mathbf y=f_\theta(\mathbf h),2

with y=fθ(h),\mathbf y=f_\theta(\mathbf h),3.

A collapsed representation produces projection characteristic functions resembling those of a point mass,

y=fθ(h),\mathbf y=f_\theta(\mathbf h),4

rather than the standard-Gaussian characteristic function. SIGReg consequently makes collapse statistically expensive. Once the full-view targets retain broad variation, the masked branch cannot minimize the prediction loss with a single constant output and must exploit temporal and phonetic context.

The paper reports that removing SIGReg causes effective rank to fall to approximately 1 within a few thousand updates, with phone accuracy around 16% and imbalanced class behavior. With SIGReg, the representation remains useful for downstream tasks.

4. Regularization populations and training configuration

Speech representations are temporally correlated, so the population used by SIGReg affects the estimated distribution.

In time-conditional SIGReg, the regularizer uses the y=fθ(h),\mathbf y=f_\theta(\mathbf h),5 representations at each time index:

y=fθ(h),\mathbf y=f_\theta(\mathbf h),6

and averages the penalty over time. This approximates regularization of y=fθ(h),\mathbf y=f_\theta(\mathbf h),7.

Marginal SIGReg instead treats tokens across utterances and positions as a single population, approximating y=fθ(h),\mathbf y=f_\theta(\mathbf h),8. Two marginal estimators were examined:

  • Shuffled Marginal: all y=fθ(h),\mathbf y=f_\theta(\mathbf h),9 tokens are shuffled and divided into z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).0 groups of size z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).1;
  • Full Marginal: a single SIGReg penalty is applied to all z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).2 tokens.

The main 57M-parameter model uses Full Marginal SIGReg. In a controlled four-layer ablation, Shuffled Marginal improved the 30M model relative to Time-conditional SIGReg from 13.99% to 12.24% ASR WER, from 57.92% to 58.60% emotion accuracy, and from 7.65% to 6.65% diarization DER.

Pretraining uses LibriSpeech’s 960 hours, crops between 2 and 15.6 seconds, approximately 4,000 seconds of audio per optimization step, and 220,000 updates. Optimization uses AdamW with a 40,000-update linear warmup, cosine decay after warmup, peak learning rate z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).3, weight decay z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).4, and SIGReg coefficient z=pα ⁣(fθ(h)).\mathbf z=p_\alpha\!\left(f_\theta(\mathbf h)\right).5.

5. Empirical evaluation

Evaluation uses frozen-encoder SUPERB tasks. The encoder is frozen, and task heads use a learned weighted sum of hidden layers. The reported tasks are ASR, speaker diarization, emotion recognition, and slot filling.

Model Parameters ASR WER Diarization DER Emotion accuracy Slot F1 Slot CER
wav2vec 2.0 Base 95M 6.43 6.08 63.43 88.30 24.77
HuBERT Base 95M 6.42 5.88 64.92 88.53 25.20
WavLM Base 95M 6.21 4.55 65.94 89.38 22.86
data2vec 2.0 Base 94M 4.81 6.50 66.66 89.67 22.09
S-JEPA 52M 12.10 — 64.83 83.05 33.17
GLaS-JEPA 57M 6.89 6.46 57.91 87.72 25.87

The model obtains 6.89% WER on ASR, 6.46% DER on diarization, 57.91% emotion-recognition accuracy, 87.72% slot-filling F1, and 25.87% slot-filling CER. Relative to the best listed non-distilled models below 90M parameters, the reported improvements are 43.1% in ASR WER and 22.0% in slot-filling CER.

The results do not establish universal superiority over larger models. data2vec 2.0 Base obtains 4.81% WER, and WavLM obtains 6.21% WER. GLaS-JEPA is particularly competitive in ASR and slot filling, whereas its emotion-recognition accuracy is substantially lower than those of WavLM, data2vec 2.0, and S-JEPA.

The comparisons are contextual rather than fully controlled: the baselines differ in architecture, parameter count, pretraining duration, dataset size, and training objective. S-JEPA, for example, uses approximately 83,000 hours of pretraining data compared with GLaS-JEPA’s 960 hours.

6. Ablations, interpretation, and limitations

The most decisive ablation removes SIGReg. Without it, direct current-encoder prediction collapses rapidly, supporting the conclusion that representation-space regularization is necessary for the simplified target construction.

The regularization-population ablation favors marginal formulations over time-conditional regularization in the reported controlled experiment. A plausible implication is that marginal populations expose SIGReg to greater variation across both utterances and temporal positions, although the paper does not establish this as a definitive causal explanation.

The training dynamics reveal task-specific trade-offs. Layerwise probes show that phone accuracy peaks around layer 6 for GLaS-JEPA and around layer 11 for WavLM, while speaker accuracy peaks around layer 3 for GLaS-JEPA and around layer 4 for WavLM. From approximately 140,000 to 220,000 updates, ASR WER improves from 7.52% to 6.89%, while emotion accuracy decreases from 59.20% to 57.91%. The paper interprets this pattern as evidence that continued optimization increasingly favors content information over some paralinguistic information, while noting that the results do not prove causality.

The reported limitations include:

  • Dependence on SIGReg: current-encoder prediction is unstable without strong distributional regularization.
  • Task imbalance: the model is stronger in ASR and slot filling than in emotion recognition.
  • Scale limitations: the results do not establish smooth scaling to models near or above 95M parameters.
  • Uncontrolled comparisons: baseline systems differ in data, architecture, and optimization.
  • Approximate Gaussianity: speech representations are temporally correlated and need not be IID Gaussian samples; SIGReg’s Gaussian target functions as a regularizing reference distribution.
  • Potential invariance trade-offs: isotropic and approximately independent representations may prevent collapse while discarding information useful for particular downstream tasks.
  • Limited masking analysis: the supplied results do not provide a complete numerical sweep over masking ratios.
  • Limited predictor analysis: the simple token-wise projector performs well, but the paper does not establish that it is universally optimal compared with deeper or temporally mixing predictors.

GLaS-JEPA’s principal methodological contribution is to relocate collapse prevention from target construction to representation geometry. Its targets are continuous current-encoder states, while SIGReg keeps those states distributed, nonconstant, and approximately isotropic. The reported experiments support the narrower conclusion that this combination can produce competitive frozen speech representations at 57M parameters using 960 hours of pretraining data. They do not establish that engineered targets are never beneficial, that isotropic Gaussian latents are intrinsically optimal for speech, or that GLaS-JEPA universally outperforms larger teacher-based systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to GLaS-JEPA.