UTMOS Score: Neural MOS Evaluation

• UTMOS Score is a neural model–based metric for predicting perceptual speech quality, combining ensemble learning and SSL features to mirror human judgments.
• It integrates strong learners (deep networks with contrastive loss and phoneme encoding) and weak learners (traditional regression models) to achieve robust utterance-level MOS predictions.
• It serves as a benchmark in speech evaluation across TTS, voice conversion, and enhancement, providing objective system-level scores that correlate highly with human listener ratings.

The UTMOS score is an objective, neural model–based metric for predicting the Mean Opinion Score (MOS) of speech, widely adopted for non-intrusive evaluation of speech quality and naturalness in fields such as text-to-speech (TTS), voice conversion, and speech enhancement. Built on ensemble learning over self-supervised learning (SSL) speech representations and traditional regressors, UTMOS delivers utterance-level and system-level predictions highly correlated with human listener judgments. It has become a de facto standard reference for automatic MOS estimation in large-scale speech evaluation studies and benchmarking challenges.

1. System Architecture and Methodology

UTMOS is designed as an ensemble framework integrating two categories of models:

• Strong learners: Deep networks constructed by fine-tuning large-scale SSL models (e.g., wav2vec2.0, HuBERT, WavLM)—these ingest raw speech and generate frame-level SSL features, which are passed to bidirectional LSTM layers and subsequent linear modules. UTMOS departs from conventional approaches by computing frame-level scores, averaging them to yield utterance-level MOS.
• Weak learners: Lighter regression models (ridge regression, SVR, decision trees, LightGBM, Gaussian process regression), each operating on utterance-level mean embeddings extracted from various SSL models.

The outputs from both learners are aggregated using a multi-stage stacking ensemble: first, the predictions from individual learners are cross-validated and collected; second, several levels of meta-learners (regression/MLP layers) combine these outputs to construct the final MOS prediction.

Strong learners introduce several enhancements:

• Contrastive loss: For any pair of utterances $x_1, x_2$ with true and predicted MOS difference $d_{x_1,x_2}, \hat{d}_{x_1,x_2}$, the contrastive loss $$\mathcal{L}_{x_1,x_2}^{con} = \max(0, |d_{x_1,x_2} - \hat{d}_{x_1,x_2}| - \alpha)$$ encourages the network to preserve correct relative rankings, improving metrics like SRCC and KTAU.
• Listener identity and data domain embeddings: To capture variations in scoring due to specific listeners or evaluation conditions, embeddings are injected; training is conducted on listener-dependent MOS labels, with mean aggregation used for evaluation.
• Phoneme encoding: ASR outputs are clustered (DBSCAN over normalized Levenshtein distance) to yield a reference phoneme sequence. An auxiliary BLSTM encodes both the reference and the detected sequence, concatenated with SSL features along the time dimension.
• Data augmentation: Small random changes in speaking rate and pitch (suitably constrained so as not to change perceptual MOS) are applied, improving generalization, especially in overfitting-prone regimes.

The overall strong learner training loss is

$$\mathcal{L} = \beta \cdot \mathcal{L}^{reg} + \gamma \cdot \mathcal{L}^{con}$$

where

$$\mathcal{L}^{reg}(y, \hat{y}) = \mathbb{I}(|y-\hat{y}| > \tau) \cdot (y - \hat{y})^2$$

and $\mathcal{L}^{con}$ is as above.

2. UTMOS Score Calculation and Metric Properties

The UTMOS score is the ensemble's final prediction: at inference, the trained stacking meta-learner outputs an utterance-level MOS estimate for each sample. For system-level evaluation, UTMOS scores are averaged over the set of utterances generated by a system. Formally, the prediction may be written as

$UTMOS = \frac{1}{N} \sum_{i=1}^N MOS_{predicted,i}$

where $N$ is the set of evaluated utterances. During training, integration of both regression and ranking losses encourages not only accurate MOS magnitude prediction but also preservation of the correct relative quality order.

Ablation studies confirm:

• Removal of contrastive loss component negatively affects ranking accuracy (SRCC, KTAU).
• Eliminating listener/dependence embeddings degrades out-of-domain prediction.
• Excluding phoneme encoding reduces performance, especially where linguistics influence perceived quality.
• Disabling data augmentation reduces robustness in low-data settings.

UTMOS is validated on the VoiceMOS Challenge, achieving top system-level metrics: for example, a main track system-level SRCC of 0.936 and OOD system-level SRCC up to 0.988.

3. Extensions: Use as Benchmark and Model-Free Evaluation

UTMOS scores have become a standard reference in speech system benchmarking. For example, in the Interspeech 2024 Discrete Speech Unit Challenge (Guo et al., 9 Apr 2024), UTMOS was employed to score both resynthesis and text-to-speech outputs, providing a direct measure of naturalness across submissions employing different token sets (wav2vec2.0 and FunCodec). Here, UTMOS was purely used as a model-based perceptual metric: a higher score indicated synthesis closer to natural speech, regardless of underlying bitrate or architecture. The best systems in the challenge achieved UTMOS scores rivaling those of natural speech, e.g., reports such as 4.43 ± 0.07 for ground truth.

Although the technical computation—i.e., model code and normalization strategies—is specified in the original UTMOS publication (Saeki et al., 2022), the deployment of UTMOS in major challenges follows the same principles: input speech is preprocessed to extract SSL features, batched through a deep regression ensemble, and the resulting scores are aggregated for system ranking.

4. Incorporating Score Distributions and Subjective Variability

A significant insight is that conventional MOS prediction, including UTMOS, often disregards the variance of opinion scores from human listeners. Integrating distributional statistics—variance, median, and histograms of ratings—into the loss allows for not just improved point estimate accuracy (lower RMSE, higher SRCC) but also uncertainty quantification.

Key strategies, which can be applied to UTMOS or similar models, include:

• Variance weighting of loss: Each training sample contributes to the loss weighted inversely by rating variance, prioritizing confident judgments.
• Multi-task regression: Joint prediction of MOS, variance, and median improves model calibration.
• Opinion score histogram prediction: The network outputs a discrete probability distribution over possible scores, aligning with the empirical distribution from listeners.

Empirical results show up to a 0.016 improvement in RMSE and ≥1% increase in SRCC over baseline MOS predictors when adopting such strategies (Faridee et al., 2022). This suggests that future UTMOS variants could be extended to multi-target learning or histogram prediction to better capture subjective variability and listener uncertainty.

5. Comparative Analysis: UTMOS vs. Alternative Metrics

UTMOS is a neural MOS-prediction model, predicting continuous-valued scores as proxies for subjective judgments. Recent research has introduced alternatives emphasizing distributional or ranking-based evaluation:

• TTSDS (Minixhofer et al., 17 Jul 2024): Rather than rely solely on MOS predictions, TTSDS evaluates multiple factors (prosody, intelligibility, speaker characteristics, using specialized feature distances) and computes similarity to real-speech and distractor datasets via 2-Wasserstein distance. The final score, an unweighted average over factors, achieves Pearson and Spearman correlations with human MOS over 0.6–0.83—sometimes outperforming UTMOS especially in modern/LLM-based TTS, by providing factor-level diagnostics.
• URGENT-PK (Wang et al., 30 Jun 2025): Focuses on system ranking via pairwise comparisons rather than absolute MOS prediction, motivated by evidence that human raters are more reliable in A/B tests. By defining each instance as a pair of enhanced speech outputs from two systems, its pairwise model boosts data efficiency and robustness. ECS (Enumerating-Comparing-Scoring) algorithm aggregates these fine-grained pair judgments into system rankings, leading to improved SRCC/KRCC over UTMOS even with limited annotations.

A comparison of properties:

Metric	Target	Features Leveraged	Primary Output	Robustness to Subjectivity
UTMOS	MOS	SSL embeddings, phonemes, listener embeddings	Utterance/system MOS	Moderate; can be extended via score distribution integration
TTSDS	Human Quality	Prosody, speaker, ASR, SSL feat. dists	Multi-factor score	High; per-factor and multi-dataset calibration
URGENT-PK	System Rank	Mel/SSL features, pairwise comp.	Relative ranking	High; robust to scale inconsistencies

6. Limitations, Considerations, and Future Directions

UTMOS delivers strong performance under typical MOS prediction paradigms but exhibits some limitations:

• Domain generalization: Out-of-distribution cases require explicit design (listener/domain embeddings, data augmentation). Alternative methods like TTSDS show greater robustness due to multi-factor calibration.
• Reliance on rating availability: UTMOS may struggle in domains with limited or unbalanced human annotations. URGENT-PK's pairwise data multiplication is a promising solution.
• Interpretability: UTMOS yields a scalar score; compositional approaches (e.g., TTSDS) illuminate which specific factors (prosody, intelligibility) drive system weakness.
• Subjective uncertainty: Models that predict a full rating distribution rather than just the mean (as explored in (Faridee et al., 2022)) would improve transparency and reliability, especially for deployment in perceptually sensitive applications.

A plausible implication is that next-generation UTMOS-like systems will incorporate second-order supervision, factorized diagnostics, and possibly pairwise ranking modules within the neural architecture in order to address these challenges and deliver more holistic, trustworthy speech quality assessment.

7. Practical Impact and Benchmarking Role

The adoption of UTMOS as an evaluation standard in speech challenges and system benchmarks underscores its utility:

• Enables objective, reproducible comparison across synthesized and processed speech without human-in-the-loop evaluation.
• Facilitates rapid, large-scale benchmarking, especially for low-bitrate or non-standard TTS/voice codecs, where subjective testing is impractical (Guo et al., 9 Apr 2024).
• Provides a tightly correlated proxy for listener perception, as documented in large-scale challenge results and empirical studies.

UTMOS—and its multi-factor, distributional, or ranking-augmented extensions—has established a robust methodological foundation for perceptual assessment in computational speech research, shaping both the design and the comparative evaluation of modern neural speech systems.