---
title: Mean Opinion Score (MOS)
url: https://www.emergentmind.com/topics/mean-opinion-score-mos
type: topic
---

# Mean Opinion Score (MOS)

The Mean Opinion Score (MOS) is a foundational metric in evaluation science, providing a scalar summary of human perceptual judgments over a range of signals, most prominently in speech and audio quality assessment but also with growing relevance in domains such as image quality and explainable AI. Originally introduced in telecommunication standards, MOS has become the “gold standard” for benchmarking generative, enhancement, and transmission systems due to its ability to capture human-centric quality attributes that are elusive to objective algorithms.

## 1. Formal Definition, Computation, and Statistical Properties

MOS is computed as the arithmetic mean of K discrete ratings assigned by a group of human judges to a given stimulus, typically on an ordinal scale (e.g., 1 = “bad” to 5 = “excellent”). For a stimulus $u$ and observed scores $\{s_{u,1},…,s_{u,K}\}$,

\[
\mathrm{MOS}(u) = \frac{1}{K}\sum_{k=1}^{K} s_{u,k}
\]

At the system level, MOS scores across all generated outputs are further averaged. In practice, MOS experiments employ Absolute Category Rating (ACR, ITU-T P.800), and results are often reported with confidence intervals based on t-statistics. For $N$ ratings:

\[
\mathrm{CI} = \bar{X}\;\pm\;t_{\alpha/2,\,N-1}\;\frac{S}{\sqrt{N}}
\]

where $S$ is the sample standard deviation of the ratings.

In image and XAI quality, the protocol is unchanged: a set of annotators rates each stimulus, and the MOS is the arithmetic mean over all observers after filtering outliers [2407.20427].

## 2. Limitations, Biases, and Score Interpretation

MOS, despite its ubiquity, suffers critical limitations relating to data quantization, annotator bias, and context dependency:

- **Quantization Effects:** MOS relies on discretized labels, implicitly assuming annotators’ internal judgments are perfectly captured by rounding to the nearest integer. This ignores the true continuous nature of perceptual quality [2506.18307].
- **Range-Equalizing Bias:** Listeners stretch their use of the scale to fill the available range. When evaluated in narrow quality contexts (e.g., only high-quality systems), even minor degradations are rated "bad," possibly resulting in MOS shifts of up to one point for the worst system in the narrowest subset [2305.10608]. This bias causes MOS to be fundamentally relative: the absolute placement of systems on the 1–5 scale can change with the composition of the test set.
- **Annotator Bias and Inconsistency:** Different raters use the scale differently, and each utterance/system is rated by a small random subset. The average can be skewed by individual tendencies, making direct comparison across utterances problematic [2103.00110, 2110.09103].
- **Information Collapse:** By reducing K ratings to a mean, traditional workflows discard K – 1 data points per utterance, lose access to the variance or higher moment statistics, and can miss valuable distributional information [2103.00110, 2204.03219].
- **Interpretation in Univariate Setting:** A low MOS can be due to many different underlying artifacts (e.g., distortion, noise, prosodic errors) but offers no diagnostic decomposition [2506.04890].

Strategies such as fitting a quantized latent distribution to model the unobservable continuous quality and using its mode (rather than the raw mean) as the estimator address destructive effects of quantization and reduce estimation bias [2506.18307].

## 3. MOS Collection Protocols, Datasets, and Best Practices

MOS experiments should maximize both statistical reliability and annotation quality through carefully structured test designs and rigorous screening. Key elements include:

- **Stimulus Design:** Balanced test sets spanning domains, lengths, and attributes (e.g., 20–40 utterances/system) are standard [2204.03040].
- **Annotator Recruitment:** Stringent qualification filters (native fluency, approval rate) and mandatory use of calibrated equipment (headphones) are implemented in large-scale datasets such as SOMOS [2204.03040].
- **Quality Control:** Embedded ground-truth and validation stimuli, rejection of HITs with uniform answers, or "unnatural" labeling of trusted ground-truth clips are critical for reliable aggregation [2204.03040].
- **Aggregation and Reliability:** Bootstrap-based split-half reliability measures, expert validation, calculation of per-system and per-utterance correlations enable robust assessment of inter-rater agreement [2204.03040].
- **Dataset Design:** Use of a fixed vocoder across systems in TTS evaluation isolates acoustic-model effects and stabilizes ground-truth comparison [2204.03040].

These curated datasets underpin algorithmic advances in MOS prediction and support community benchmarking.

## 4. Automatic MOS Prediction: Model Architectures and Trends

With the cost and time of subjective MOS collection, machine learning models for automatic MOS regression have become prominent. Architectures have evolved rapidly:

- **Classic Approaches:** Early works (e.g., MOSNet) regressed the MOS directly from CNN or CNN-LSTM stacks over spectral features, ignoring per-rater variability.
- **Exploiting Full Opinion Distributions:** MBNet [2103.00110] and LDNet [2110.09103] explicitly model judge-specific ratings by introducing mean/bias subnets or listener-ID embeddings. This dual modeling leverages the full $K$-rating label vector, which improves both data efficiency and generalization by learning annotator-specific tendencies.
- **Distributional and Permissive Approaches:** DDOS [2204.03219] augments wav2vec architectures with distribution heads, learning to predict the full histogram of possible ratings. Some models regress both MOS and per-rater scores, or jointly learn mean and variance, thus capturing uncertainty and label spread.
- **Advanced Pooling and Temporal Modeling:** DRASP [2508.21407] combines global and segmental attentive pooling, and state-space sequence models such as MambaRate [2507.12090] employ self-supervised front ends and RBF encoded regression for cross-domain and cross-sampling-rate robustness.
- **Compact, Streaming Models:** SALF-MOS [2506.02082] achieves full-scale competitive MOS prediction with sub-2k parameter models through aggressive downsampling and multi-scale latent feature fusion.
- **Pairwise and Rank-Oriented Methods:** MOSPC [2306.10493] reframes MOS regression as a pairwise ranking problem, exploiting the practical importance of correct relative ordering rather than precise MOS regression. Techniques such as C-Mixup inject artificial examples for better out-of-distribution generalization.
- **Multitask Learning and Rater Bias Correction:** Recent advances co-train MOS with orthogonal perceptual or acoustic properties (e.g., clarity, reverberation time) and leverage multi-dataset/self-supervised semi-supervised approaches for robust modeling of missing labels or rater-specific bias [2212.01911].

Evaluation consistently relies on MSE, Pearson correlation (LCC), Spearman’s rank (SRCC), and Kendall’s tau. System-level metrics are generally more stable and reliable than utterance-level, but utterance-level performance is critical for model selection and rapid system iteration [2204.03040, 2506.02082].

## 5. Alternatives and Extensions to Univariate MOS

A single scalar MOS cannot disambiguate underlying factors that contribute to “poor” overall scores. Multivariate frameworks address this:

- **Joint Quality Modeling:** Probabilistic models predicting a vector of subjective scores (e.g., MOS, noisiness, coloration, discontinuity, loudness) via multivariate Gaussian posteriors with full covariance matrices enable quantification of both per-aspect uncertainty and inter-aspect dependencies [2506.04890].
- **Relative Quality and Single Opinion Calibration:** For problems where only single-opinion scores are available, methods such as Perceptual Constancy Constrained Calibration (PC3) combine learnable relative MOS-difference regressors and optimization with constraints to recover consistent MOS estimates across pairings [2404.19595].
- **Cross-Domain and OOD Applications:** Recent models demonstrate robust generalization across languages, speakers, codecs, and novel domains using few-shot or C-Mixup-based adaptation [2306.10493, 2507.12090].

In explainable AI, MOS is adopted as a metric for user-centric evaluation of explanation quality, and its (moderate) correlation with automatic metrics such as IAUC and DAUC is quantified, highlighting the complementarity and limitations of computational vs. subjective evaluation [2407.20427].

## 6. Mitigating Biases and Improving Validity

Rigorous experimental design is necessary to minimize the context dependency and range-equalizing bias of MOS:

- **Anchors and Z-Scoring:** Including poor and excellent anchors in the test set helps calibrate listener use of the scale, while post-hoc normalization (e.g., z-scoring MOS within each test context) reduces cross-context MOS drift [2305.10608].
- **Modeling Rater Bias:** Both statistical and machine learning frameworks can incorporate bias correction. Leave-one-out MOS, subtractive bias or affine calibration per rater, and explicit bias subnetworks improve label fidelity and model accuracy [2103.00110, 2212.01911].
- **Distribution-Fitting and Mode Estimation:** Fitting latent normal distributions to the observed rating histogram and using their mode as the regression target better captures the underlying perceptual quality by removing range and quantization bias compared to the simple average [2506.18307].

## 7. Applications, Impact, and Future Directions

Advances in MOS methodology have profound impact across domains:

- **Speech and Audio Synthesis:** MOS remains the reference benchmark for evaluating and comparing TTS, voice conversion, coding, and enhancement systems. Recent MOS predictors replace or accelerate listening tests, enable fast iteration, and support deployment at scale [2508.21407, 2103.00110].
- **Fake Audio Detection:** MOS prediction contributes directly to downstream tasks such as fake audio detection, both for training sample selection and for MOS-gated model fusion, improving detection efficacy in the presence of synthetic waveform manipulations [2401.13249].
- **Image and Explanation Assessment:** The migration of MOS methodology into image quality and explainability research underscores its versatility and ability to bridge user-centered and algorithmic evaluation [2407.20427].
- **Robust Evaluation and Benchmarking:** The development of large, open MOS datasets (e.g., SOMOS [2204.03040]) catalyzes progress by providing standardized, highly-controlled benchmarks spanning systems, speakers, and domains.

Continued research is focused on: increasing reliability and objectivity in MOS collection; modeling richer perceptual aspects and their joint distributions; automating bias correction; implementing scalable, low-resource deployable predictors; and further validating MOS as a cross-domain user-centered metric [2506.04890, 2508.21407, 2506.02082].

Source: https://www.emergentmind.com/topics/mean-opinion-score-mos