DistilMOS: Enhanced MOS Prediction via Self-Distillation

• DistilMOS is a MOS prediction method that augments SSL models by also predicting discrete token IDs obtained through layer-wise k-means clustering.
• It mitigates catastrophic forgetting during fine-tuning by preserving the hierarchical structure of pretrained SSL representations for improved generalization.
• The approach enhances standard MOS regression with dense auxiliary supervision, yielding better in-domain and zero-shot correlation metrics.

Searching arXiv for the DistilMOS paper and a few directly relevant SSL backbone papers for citation support. arXiv search query: (Yang et al., 20 Jan 2026) DistilMOS DistilMOS is a mean opinion score (MOS) prediction method for speech synthesis and voice conversion evaluation that augments standard self-supervised learning (SSL)-based MOS models with a layer-wise self-distillation task. Instead of training only on utterance-level MOS regression, it also predicts discrete token IDs derived by clustering the hidden representations of each layer in a pretrained SSL backbone. These auxiliary token targets serve as self-distillation signals that preserve and exploit the pretrained model’s hierarchical internal knowledge, with the stated goal of improving both prediction accuracy and generalization under full fine-tuning (Yang et al., 20 Jan 2026).

1. Problem formulation and motivation

In text-to-speech (TTS) and voice conversion (VC) evaluation, subjective quality is typically quantified by MOS: listeners rate the naturalness of an utterance, and the utterance-level score is the average across listeners. Automatic MOS prediction seeks to map an utterance $x$ directly to its score $s$, avoiding repeated listening tests. In the formulation used for DistilMOS, the model predicts $\hat{s}$ and is trained with a regression loss such as

$\mathcal{L}_{\text{MOS}} = (\hat{s} - s)^2.$

Recent state-of-the-art MOS prediction models use large pretrained SSL speech encoders, including wav2vec 2.0, HuBERT, and WavLM (Baevski et al., 2020, Hsu et al., 2021, Chen et al., 2021).

The standard SSL-MOS workflow is to pass the raw waveform through a pretrained SSL encoder, extract frame-level hidden states from each Transformer layer, combine these layer-wise states, and train a downstream MOS head. Fine-tuning the SSL backbone rather than keeping it fully frozen typically improves in-domain performance. DistilMOS is motivated by the observation that this same fine-tuning regime also introduces two recurrent failure modes on small MOS datasets: catastrophic forgetting of the pretrained SSL knowledge and overfitting with poor cross-domain generalization (Yang et al., 20 Jan 2026).

The paper characterizes catastrophic forgetting in explicitly layer-wise terms. Lower SSL layers capture acoustic and speaker/prosodic properties, while higher layers encode more linguistic or semantic information. When the backbone is optimized only for scalar MOS regression on a relatively small corpus such as BVCC, the learned representation can drift away from this pretrained structure. The reported canonical correlation analysis (CCA) shows that vanilla MOS fine-tuning produces features that are poorly correlated with the original SSL representations, almost like a random network. This is presented as a representation-level account of why zero-shot transfer to datasets such as SOMOS degrades sharply (Yang et al., 20 Jan 2026).

2. Layer-wise self-distillation via discrete token prediction

The defining operation in DistilMOS is the addition of a parallel auxiliary task: alongside MOS regression, the model predicts, for each SSL layer, a sequence of discrete token IDs obtained by clustering that layer’s frame-level hidden representations. DistilMOS is therefore a multi-task MOS predictor in which the auxiliary objectives are token classification tasks tied to the internal structure of the pretrained SSL encoder (Yang et al., 20 Jan 2026).

For each SSL layer $n$, the pretrained encoder is first run on the training utterances to produce frame-level outputs

$$\mathbf{h}^{(n)}_{1:L} \in \mathbb{R}^{L \times d},$$

where $L$ is sequence length and $d$ is hidden dimension. A separate $k$-means model is then fit for each layer, producing centroids $\{\mu^{(n)}_1,\dots,\mu^{(n)}_k\}$. Each frame vector is assigned to its nearest centroid,

$s$0

This yields a discrete token sequence for each layer and utterance. The token labels are computed once offline on the training set and reused during DistilMOS training. The clustering is done separately for each layer, so each layer has its own token space (Yang et al., 20 Jan 2026).

The method uses all Transformer layers of the SSL backbone as distillation sources. In the main experiments, the number of clusters is $s$1 per layer. Because the targets are discrete cluster assignments rather than continuous embeddings, the method constrains the fine-tuned network to preserve the coarse latent structure of the pretrained SSL model without requiring exact reproduction of every hidden vector (Yang et al., 20 Jan 2026).

The paper explicitly frames this as self-distillation. The “teacher” is not a separate frozen model but the original pretrained SSL model’s own internal representations, converted into pseudo labels through $s$2-means. The “student” is the MOS prediction model, including the fine-tuned SSL backbone and downstream modules, trained to predict those token IDs. A common misconception is therefore ruled out by design: DistilMOS does not depend on an external teacher network (Yang et al., 20 Jan 2026).

3. Model architecture and training objective

DistilMOS is built on a fully fine-tuned pretrained SSL encoder. The reported backbones are wav2vec 2.0 Base, WavLM Base, WavLM Base+, and WavLM Large. The encoder produces frame-level hidden states from all Transformer layers, which are aggregated by a learnable layer-wise weighted sum,

$s$3

followed by a linear projector to 256 dimensions (Yang et al., 20 Jan 2026).

The projected sequence is processed by a Feature Processor consisting of three stacked blocks, each with the sequence Linear $s$4 Conv1D $s$5 BatchNorm $s$6 GELU. A subsequent CNN-BLSTM module applies Conv1D, BLSTM, and Linear + GELU, together with a residual connection and LayerNorm, to capture longer-range temporal dependencies. MOS prediction is then obtained by mean pooling over time and a final linear layer that outputs the scalar $s$7 (Yang et al., 20 Jan 2026).

For the self-distillation task, DistilMOS attaches $s$8 parallel token predictors, one per selected SSL layer. Each predictor is a 3-layer MLP with GELU. All predictors share the same input sequence, namely the output of the Feature Processor, and each produces a per-frame distribution over the $s$9 cluster IDs of its corresponding SSL layer (Yang et al., 20 Jan 2026).

Training minimizes a joint objective composed of MOS regression and layer-wise token classification: $\hat{s}$0

$\hat{s}$1

$\hat{s}$2

with $\hat{s}$3. All parameters, including the SSL encoder, aggregation weights, Feature Processor, CNN-BLSTM, MOS head, and token predictors, are trained jointly by backpropagation. At inference time, the token predictors are discarded, so runtime cost is essentially the same as for a standard SSL-MOS model (Yang et al., 20 Jan 2026).

The implementation details are specific. All linear, Conv1D, and BLSTM layers in the Feature Processor and CNN-BLSTM use 256 dimensions. Training is run for 10,000 steps with batch size 32 on one NVIDIA GH200 (120GB), using AdamW with learning rate $\hat{s}$4, betas $\hat{s}$5, weight decay $\hat{s}$6, gradient clipping 10.0, and a OneCycleLR scheduler. Checkpoints are saved every 1,000 steps, and the best model is selected by utterance-level SRCC on the BVCC validation set. Clustering uses MiniBatchKMeans with partial fit, batch size 64, and BVCC training data only (Yang et al., 20 Jan 2026).

4. Evaluation protocol, baselines, and empirical results

The in-domain dataset is BVCC, consisting of 7,106 English synthetic utterances from 187 systems, with each utterance rated by 8 listeners and MOS in $\hat{s}$7. The split is 4,974 train, 1,066 validation, and 1,066 test. Zero-shot evaluation is performed on SOMOS, which contains 20,000 synthetic utterances plus 100 natural utterances from 200 TTS acoustic models, with at least 17 listeners per utterance and MOS in $\hat{s}$8. DistilMOS is evaluated on SOMOS without any fine-tuning (Yang et al., 20 Jan 2026).

For BVCC, evaluation is reported at both utterance level and system level using LCC, SRCC, KTAU, and MSE. For SOMOS, utterance-level LCC, SRCC, KTAU, and MSE are reported. The comparison set includes four variants: SSL-MOS, DistilMOS, a matched-architecture variant without token prediction, and an MSE distillation baseline that regresses original SSL embeddings for each layer instead of predicting discrete tokens (Yang et al., 20 Jan 2026).

The principal quantitative finding is that DistilMOS improves over SSL-MOS across all four SSL backbones in both in-domain and out-of-domain settings. On BVCC, gains are reported for LCC, SRCC, and KTAU at both utterance and system level, with especially strong gains at system level. For WavLM Base, the reported utterance-level SRCC improves from 0.870 to 0.884, and system-level SRCC improves from 0.916 to 0.928. DistilMOS also usually improves MSE, although the paper emphasizes correlation metrics as the primary criterion (Yang et al., 20 Jan 2026).

On SOMOS, the zero-shot gains are larger. For WavLM Base+, the utterance-level SRCC improves from 0.287 to 0.370, and LCC improves from 0.315 to 0.396. The paper notes that MSE on SOMOS is high in absolute terms and varies, and argues that cross-dataset calibration makes correlation metrics more meaningful indicators in this setting (Yang et al., 20 Jan 2026).

The ablations sharpen the causal claim. Removing token prediction reduces performance relative to DistilMOS in both in-domain and zero-shot evaluations, indicating that the gain does not arise from architecture alone. Replacing discrete token prediction with embedding-level MSE distillation is also consistently worse in correlation metrics; for WavLM Base on SOMOS, SRCC is 0.375 for DistilMOS and 0.362 for MSE distillation. The reported conclusion is that discrete token prediction is a better way to distill multi-layer SSL knowledge for MOS prediction than $\hat{s}$9 matching of continuous embeddings (Yang et al., 20 Jan 2026).

The paper also studies the number of clusters $\mathcal{L}_{\text{MOS}} = (\hat{s} - s)^2.$0 using WavLM Base with $\mathcal{L}_{\text{MOS}} = (\hat{s} - s)^2.$1. The reported SRCC curves show relatively stable performance across this range, with all tested values outperforming both baseline and ablation models, including in the SOMOS zero-shot scenario. This suggests that the presence of the self-distillation signal matters more than precise token granularity (Yang et al., 20 Jan 2026).

5. Representation preservation and catastrophic forgetting

A central analytical result in DistilMOS is the CCA-based comparison between the average-pooled output of the MOS model’s Feature Processor and the average-pooled representations of each Transformer layer in the pretrained SSL model. The comparison spans DistilMOS, MSE distillation, the matched architecture without token prediction, SSL-MOS, and a randomly initialized MOS predictor used to measure architectural bias (Yang et al., 20 Jan 2026).

The reported pattern is stark. DistilMOS and MSE distillation maintain high CCA across most layers, implying that their internal representations remain strongly aligned with the pretrained SSL backbone’s layer-wise features. By contrast, the models trained without self-distillation have CCA curves close to random initialization, indicating that their features are almost uncorrelated with the original SSL layers. The paper interprets this as evidence of severe catastrophic forgetting under MOS-only fine-tuning (Yang et al., 20 Jan 2026).

This analysis gives a mechanistic account of why DistilMOS transfers better across domains. MOS regression alone supplies only one scalar supervision signal per utterance. The token prediction tasks supply much denser supervision: one label per frame and per layer. This suggests that DistilMOS should be understood not merely as a multi-task predictor but as a regularized fine-tuning scheme that constrains optimization to remain close to the pretrained SSL manifold while still adapting to MOS labels. The reported association between persistent SSL alignment and improved SOMOS zero-shot performance is consistent with that interpretation (Yang et al., 20 Jan 2026).

The same section also clarifies why discrete token targets may outperform continuous embedding regression. The paper argues that token prediction captures coarse but task-relevant structure, whereas direct embedding matching may overfit idiosyncratic SSL features rather than focusing on abstractions that matter for speech quality assessment. This is presented as the main reason DistilMOS exceeds the embedding-MSE variant in the reported correlation metrics (Yang et al., 20 Jan 2026).

6. Limitations, scope conditions, and future directions

The paper identifies several limitations. First, clustering is performed on the BVCC training set only, so the resulting centroids and token IDs may not optimally reflect representations from other datasets. Second, the method incurs additional offline computation for per-layer $\mathcal{L}_{\text{MOS}} = (\hat{s} - s)^2.$2-means clustering and adds one token predictor per layer during training. Third, the approach is designed around Transformer-based SSL speech encoders with accessible layer-wise hidden states; adaptation to substantially different architectures would require decisions about which internal representations to cluster and distill (Yang et al., 20 Jan 2026).

These limitations bound the scope of the current evidence. The method is demonstrated on BVCC for training and in-domain testing, and on SOMOS for true zero-shot evaluation, but not on a broader multi-corpus training setup. The backbone set is varied, but it remains within the family of pretrained SSL speech encoders already common in MOS prediction. A plausible implication is that the strongest generalization gains may depend on the interaction between tokenization quality and the domain coverage of the corpus used to construct the token targets.

The future directions stated in the paper are correspondingly targeted. They include applying DistilMOS on larger or combined datasets to reduce clustering mismatch across domains, developing a more universal MOS predictor by jointly clustering and training on multiple corpora such as BVCC and SOMOS, extending the framework to other subjective quality tasks including non-speech audio quality, music, and audio-visual MOS, and exploring alternative SSL backbones, multimodal backbones, or more sophisticated discretization strategies beyond $\mathcal{L}_{\text{MOS}} = (\hat{s} - s)^2.$3-means, such as vector quantization and product quantization (Yang et al., 20 Jan 2026).

In sum, DistilMOS defines a specific response to a precise failure mode in SSL-based MOS prediction: when full fine-tuning erodes the pretrained representation hierarchy, supplement scalar MOS supervision with dense, layer-wise pseudo labels derived from that hierarchy itself. Within the reported experiments, this strategy improves in-domain correlation, strengthens system-level ranking, and substantially improves zero-shot transfer, while preserving close representational alignment with the original SSL backbone (Yang et al., 20 Jan 2026).